Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment, and relates to the technical field of networks. The method comprises the following steps: in response to an input instruction, creating a Pod unit; using the initial container to generate an initial operation environment based on the first data mirror image, and storing data corresponding to the initial operation environment to a shared space; and loading the second data mirror image by using the application container, reading data corresponding to the initial operation environment in the shared space, generating a target operation environment, and executing the target application. Compared with the prior art, the data mirror image is divided into the big data engine mirror image and the user-defined mirror image, then the shared space is set in the Pod unit, the data corresponding to the initial operation environment is stored in the shared module, and the second data mirror image can directly read the operation environment required by data derivation in the shared module. The problem that the second data mirror image needs to be updated when the first data mirror image is updated is avoided. Decoupling of the two is achieved, and flexibility is improved.

Description

Technical Field

[0001] This application relates to the field of network technology, and more specifically to a data processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] Currently, big data solutions running on Kubernetes (an open-source platform for automatically deploying, scaling, and managing containerized applications) only allow users to specify one image, typically coupling the big data engine image and the user image. Taking the Spark big data framework as an example, the framework provides a basic Spark image containing the minimum dependencies needed to run Spark, providing a basic runtime environment. However, if a user wants to run more complex programs (such as TensorFlow, PyTorch, etc.) within the Spark framework, they need to create a new user image based on their needs, starting with the Spark engine image and adding specific configurations, libraries, application code, or other dependencies. This results in a new coupling between the big data engine image and the user image.

[0003] In this coupled configuration of big data engine images and user images, the strong dependency between them necessitates frequent updates to both the big data engine and user images. This increases maintenance costs, time consumption, and the likelihood of errors in resource allocation, coordination, and verification, and also leads to fragmentation of version resources. Therefore, current Kubernetes big data solutions suffer from a lack of flexibility in updating and maintaining both big data engine and user images due to their coupled dependency. Summary of the Invention

[0004] In view of this, this application provides a data processing method, apparatus, storage medium, and electronic device, which can improve the current big data solutions on K8S, which suffer from the problem of insufficient flexibility in updating and maintaining the big data engine image and user image due to the strong dependency between them.

[0005] In a first aspect, this application provides a data processing method, including:

[0006] In response to an input command, a Pod unit is created; the Pod unit includes an initial container, at least one application container, and a shared space.

[0007] The initial container is used to generate an initial runtime environment based on the first data image, and the data corresponding to the initial runtime environment is stored in the shared space;

[0008] The application container loads the second data image, reads the data corresponding to the initial runtime environment in the shared space, generates the target runtime environment, and executes the target application.

[0009] The first data image includes a pre-built big data engine image, used to generate the initial runtime environment; the second data image includes a custom image with dependencies for executing the target application, used to generate the target runtime environment.

[0010] Optionally, before generating the initial runtime environment based on the first data image using the initial container, the method further includes: initiating an authentication request when a request to update the big data engine image is received; and, if the authentication result is a level-one permission, building and updating the big data engine image in response to an update instruction input by the first data interface; wherein the level-one permission is an administrator permission, and the first data interface is used to receive the update instruction input by the administrator.

[0011] Optionally, before loading the second data image using the application container, the method further includes: upon receiving a request to modify the custom image, initiating an authentication request; if the authentication result is a level-two permission, in response to the modification instruction input by the second data interface, adding configuration information to establish a dependency relationship with the target application; and generating the custom image with the dependency relationship to execute the target application based on the configuration information; wherein the level-two permission is a user permission, and the second data interface is used to receive the modification instruction input by the user.

[0012] Optionally, the input instruction includes the target application to be executed and the required resources; the step of creating a Pod unit in response to the input instruction includes: determining the number of application containers based on the number of target applications to be executed; and creating the shared space in the Pod unit based on the required resources.

[0013] Optionally, creating the shared space in the Pod unit includes: creating a shared space of temporary storage volume type, wherein the capacity of the shared space meets the required resource requirements.

[0014] Optionally, after executing the target application, the method further includes: destroying the Pod unit and reclaiming the resources occupied by the destroyed Pod unit.

[0015] Secondly, this application provides a data processing apparatus, comprising:

[0016] A Pod unit is created in response to an input command; the Pod unit includes an initial container, at least one application container, and a shared space.

[0017] The first generation unit is configured to use the initial container to generate an initial running environment based on the first data image, and to store the data corresponding to the initial running environment in the shared space;

[0018] The second generation unit is configured to load the second data image using the application container, read the data corresponding to the initial running environment in the shared space, generate the target running environment, and execute the target application.

[0019] The first data image includes a pre-built big data engine image, used to generate the initial runtime environment; the second data image includes a custom image with dependencies for executing the target application, used to generate the target runtime environment.

[0020] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method described in the first aspect.

[0021] Fourthly, this application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the data processing method of the first aspect;

[0022] Fifthly, this application provides a computer program product having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data processing method described in the first aspect.

[0023] By employing the above technical solutions, this application provides a data processing method, apparatus, storage medium, and electronic device. First, in response to an input instruction, a Pod unit is created. The Pod unit includes an initial container, at least one application container, and a shared space. The initial container, based on a first data image, generates an initial runtime environment and stores the data corresponding to the initial runtime environment in the shared space. The application container loads a second data image and reads the data corresponding to the initial runtime environment from the shared space, generating a target runtime environment and executing the target application. The first data image includes a pre-built big data engine image used to generate the initial runtime environment; the second data image includes a custom image with dependencies for executing the target application, used to generate the target runtime environment. Compared to related technologies, this method first distinguishes between a big data engine image and a user-defined image, then sets up a shared space in the Pod unit, storing the data corresponding to the initial runtime environment in a shared module. This allows the second data image to directly read the data in the shared module and derive the required runtime environment. This avoids the problem that the second data image needs to be updated when the first data image is updated. It achieves decoupling between the two and improves flexibility.

[0024] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application is shown;

[0028] Figure 2 This illustration shows a schematic diagram of a K8S big data solution architecture in a related technology provided by an embodiment of this application;

[0029] Figure 3 This illustration shows a schematic diagram of a K8S big data solution architecture provided in an embodiment of this application;

[0030] Figure 4 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application is shown. Detailed Implementation

[0031] To better understand the above-mentioned objectives, features, and advantages of this application, the solutions of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other. Furthermore, to gain a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be simplified in their depiction to simplify the drawings.

[0032] To address the inflexibility in updating and maintaining current big data solutions on Kubernetes due to the coupling and dependency between the big data engine image and the user image, this embodiment proposes a data processing method. For example... Figure 1 As shown, the method includes:

[0033] S101, Create a Pod unit.

[0034] Create a Pod object in a Kubernetes (K8S) environment. K8S is an open-source container orchestration system used to automate the deployment, scaling, and management of containerized applications. A Pod is the most basic unit of execution in K8S, representing a process running on the cluster. Each Pod can contain one or more containers, as well as shared storage, network, and other resources. By creating Pod units, users can define and manage application instances running on the K8S cluster.

[0035] S102, using the initial container based on the first data image, generates the initial application environment and stores the data corresponding to the initial application environment in the shared space.

[0036] In Kubernetes (K8S), a Pod can contain one or more application containers. The initial container primarily provides the environment needed for other containers to run, such as initializing data volumes and configuring networks. The first data image refers to the Docker image used to create the initial container; it may contain scripts, tools, or data needed to initialize the application environment. After starting, the initial container executes tasks defined in its image. These tasks may include copying configuration files, downloading dependencies, and setting environment variables to generate the basic environment required by the application containers.

[0037] Shared spaces primarily use the Empty Dir volume type. An Empty Dir is a simple, temporary directory used to store data shared by containers within a Pod. Its lifecycle is the same as the Pod; when the Pod is deleted, the data in the Empty Dir is also permanently deleted. Due to its temporary nature, Empty Dir is suitable for storing data that doesn't require persistence, such as cached files and temporary files. When creating an Empty Dir shared space, its capacity can be specified to meet the required resource needs. This ensures that even if multiple containers write data simultaneously, the shared space has sufficient capacity to store this data. It's important to note that because Empty Dir is temporary, it is not suitable for storing data that needs to be stored long-term.

[0038] In this embodiment, for data that needs to be persisted, the shared space can also refer to a persistent volume (PV) in Kubernetes, which can be connected to physical storage devices, distributed storage systems, or cloud storage services. Through PVs, the storage type, capacity, storage class, and other related attributes can be defined. Initial containers may write some initialization data, configuration files, or other important information to the PV, so that even if the Pod is deleted or rebuilt, this data can still be accessed and used by subsequent application containers.

[0039] S103, load the second data image using the application container, read the data corresponding to the initial application environment in the shared space, generate the target application environment, and execute the target application.

[0040] In a Kubernetes (K8S) Pod, the application container is the component responsible for executing the actual application logic. The second data image refers to the Docker image used by the application container. This image contains the application's binaries, dependencies, configuration files, etc., and is the foundation for the application to run in a containerized environment. When a Pod is created, K8S starts the application container based on the configuration in the Pod definition and loads the specified second data image. The application container then starts the application based on this image. After the application container starts, persistent storage volumes read data from these shared spaces. This data may include initialization data, configuration files, or other important information written by the initial container during the early stages of Pod startup. The target application environment refers to the environment generated by the application container based on the read data and the configuration in the loaded second data image. This configuration includes the runtime environment required by the application, loading necessary resource files, setting environment variables, etc. The target application can be any type of service or application, such as a web server, database service, API gateway, etc.

[0041] By employing the above technical solutions, this application provides a data processing method, apparatus, storage medium, and electronic device. First, in response to an input instruction, a Pod unit is created. The Pod unit includes an initial container, at least one application container, and a shared space. An initial runtime environment is generated based on a first data image using the initial container, and the data corresponding to the initial runtime environment is stored in the shared space. A second data image is loaded using the application container, and the data corresponding to the initial runtime environment in the shared space is read to generate a target runtime environment and execute the target application. The first data image includes a pre-built big data engine image used to generate the initial runtime environment; the second data image includes a custom image with dependencies for executing the target application, used to generate the target runtime environment. Compared with related technologies, this method first distinguishes between a big data engine image and a user-defined image, then sets up a shared space in the Pod unit, storing the data corresponding to the initial runtime environment in a shared module, enabling the second data image to directly read the data in the shared module to derive the required runtime environment. This avoids the problem that the second data image needs to be updated when the first data image is updated.

[0042] Currently, big data solutions running on Kubernetes, whether submitted directly using the framework or via an Operator, only allow users to specify one image. This means the big data engine image and the user image are coupled. For example, in the Spark big data framework, such as... Figure 2 As shown, the engine maintainer typically only provides a Spark image, which contains the minimum dependencies needed to run Spark. If users need to run complex programs such as TensorFlow or PyTorch on Spark, they need to package the Spark image and their own dependencies into a new image. This new image is then submitted to Kubernetes for execution. The two images are designed to be interdependent and cannot be deployed and updated separately.

[0043] This coupling has several drawbacks: **Difficulty in updating:** When a Spark image (such as a big data engine image) needs updating, all user-defined images need to be rebuilt, even if the user's application code and dependencies remain unchanged. This increases maintenance costs and time consumption. **Coupling between engine and business development:** Image coupling typically means that the engine's development process is coupled with the business's development process, potentially reducing developer autonomy and efficiency, and requiring constant communication and coordination between the engine and business teams. **Build and storage overhead:** Every time a Spark image is updated, all user-defined images that depend on it need to be rebuilt and stored, potentially wasting computing resources and storage space. **Deployment complexity:** If the Spark image and all its derived user-defined images need frequent updates, the deployment process becomes more complex, requiring more coordination and verification work, which may increase the chance of errors. **Version tracking challenges:** Tracking different Spark image versions and their dependencies with user-defined images can be a complex and error-prone process, especially in large environments, easily leading to version fragmentation. The key to the method proposed in this embodiment lies in distinguishing between the big data engine image and the user-defined image, and then setting up a shared module in the Pod. The initial container data is stored in the shared module, and the application container only needs to read from the shared module. Thus, when the first image data is updated, the second image is prevented from needing to be updated along with the first image. This achieves decoupling between the first and second images, removes their dependency, and improves flexibility.

[0044] Optionally, before generating the initial runtime environment based on the first data image using the initial container, the method further includes: initiating an authentication request when a request to update the big data engine image is received; and, if the authentication result is level one permission, building and updating the big data engine image in response to the update instruction input by the first data interface; wherein, level one permission is administrator permission, and the first data interface is used to receive the update instruction input by the administrator.

[0045] In this embodiment, the first data mirror communicates with a type of user with certain permissions, namely administrators, through a first data interface. The construction steps of the big data engine mirror include, in response to an update command input through the first data interface, constructing and updating the big data engine mirror if the verification result indicates type one permission.

[0046] The first data image includes a pre-built big data engine image used to generate the initial runtime environment. A "first-class permission user" refers to a group of users with specific permission levels, specifically "administrators." Administrators typically have the highest privileges on the system or services, allowing them to modify and update operations. The first data interface is used to communicate with first-class permission users (i.e., administrators). Through this interface, administrators can send commands or data to the first data image. When the big data engine image needs to be built or updated, a series of steps are initiated. Before starting the construction or update of the big data engine image, the system initiates a permission verification request. This is typically to ensure that only users with the appropriate permissions can perform the operation. When the administrator enters update commands through the first data interface, the system responds to these commands. These commands may include specifying new image configurations, adding or deleting components, updating versions, etc. Based on the update commands entered by the administrator through the first data interface, the system begins building or updating the big data engine image. This may involve operations such as pulling new components from the repository, modifying configuration files, and setting environment variables. In this process, only verified administrators can send update commands through a specific data interface, thus ensuring that the image building and update operations are performed by administrators with sufficient privileges.

[0047] Optionally, before loading the second data image using the application container, the method further includes: initiating an authentication request when a request to modify the custom image is received; if the authentication result is a level 2 permission, adding configuration information that establishes a dependency relationship with the target application in response to the modification instruction input by the second data interface; and generating a custom image with a dependency relationship to execute the target application based on the configuration information; wherein, level 2 permission is a user permission, and the second data interface is used to receive the modification instruction input by the user.

[0048] In this embodiment, the second data mirror communicates with users with second-class permissions through a second data interface. The second-class permissions refer to users.

[0049] The second data image includes a custom image containing the dependencies required to execute the target application, used to generate the target runtime environment. Second-class privileged users refer to a group of users with second-class privileges, specifically "users." These users typically have limited permissions to the system or services, capable of performing basic operations but potentially unable to perform advanced or sensitive ones. The second data interface is used to communicate with second-class privileged users (i.e., ordinary users). Through this interface, users can send commands or data to the second data image. In some scenarios, it's necessary to build a custom image containing all the dependencies required to execute the target application. Before building or modifying the custom image, the system initiates a permission verification request. This ensures that only users with the appropriate permissions can perform this operation. If the permission verification shows that the user is a second-class privileged user and has the permission to perform the relevant operations, the system allows the user to continue. When ordinary users input commands through the second data interface, the system responds to these commands. These commands may include adding dependencies, configuring environment variables, setting network parameters, etc., to establish the correct dependencies for the target application. Based on the user's input via the second data interface, the system adds the configuration information required to establish dependencies with the target application to the custom image. This may include specifying the versions of dependency libraries, setting environment variables, configuring network connections, etc., to ensure that the target application can find and correctly use these dependencies at runtime. This process helps ensure that only verified users can modify the configuration of the second data image, thereby maintaining the security and stability of the system.

[0050] Optionally, the input instructions include the target application to be executed and the required resources; in response to the input instructions, a Pod unit is created, including: determining the number of application containers based on the number of target applications to be executed; and creating a shared space within the Pod unit based on the required resources.

[0051] In this embodiment, the user-input instructions contain two key pieces of information: first, the number of target applications to be executed, i.e., how many application instances the user wants to run; and second, the resource requirements of these applications, such as the quantity and type of resources like CPU, memory, and storage. The Kubernetes (K8S) system creates Pod units based on the user's input instructions. A Pod is the smallest unit for running applications in K8S and can contain one or more containers, as well as shared network and storage resources. Based on the number of target applications specified by the user, K8S determines how many application containers to create in the Pod. For example, if the user wants to run 5 application instances, the Pod will contain 5 corresponding application containers. In addition to determining the number of application containers, K8S also configures the shared space in the Pod according to the user-specified resource requirements. Shared space is typically implemented using persistent storage volumes to store data that application containers need to share. Based on the resource requirements, K8S selects an appropriate storage type, capacity, and configuration, creates a persistent storage volume, and mounts it to the Pod so that application containers can access and share this data. Kubernetes (K8S) can dynamically create Pod units based on users' actual needs and configure them with an appropriate number of application containers and shared space, thereby ensuring that applications can run in the cluster with the correct configuration and sufficient resources. This flexibility allows K8S to adapt to various application scenarios and resource requirements.

[0052] Optionally, a shared space can be created in the Pod unit, including: creating a shared space of the temporary storage volume type, wherein the capacity of the shared space meets the required resource requirements.

[0053] In this embodiment, creating a shared space of temporary storage volume type to destroy Pod units is a crucial step in Kubernetes resource management. This primarily utilizes the Empty Dir storage volume type. An Empty Dir is a simple, temporary directory used to store shared data within a Pod. Its lifecycle is the same as the Pod; when the Pod is deleted, the data in the Empty Dir is permanently deleted as well. Due to its temporary nature, Empty Dir is suitable for storing data that does not require persistence, such as cached files and temporary files. When creating an Empty Dir shared space, its capacity can be specified to meet the required resource needs. This ensures that even if multiple containers write data simultaneously, the shared space has sufficient capacity to store this data. Through Kubernetes' storage resource quota mechanism, administrators can limit the maximum storage capacity that a Pod or container can use to prevent a Pod or container from consuming excessive storage resources.

[0054] Optionally, after executing the target application, the method further includes: destroying the Pod unit and reclaiming the resources occupied by the destroyed Pod unit.

[0055] In this embodiment, destroying Pod units is a crucial step in Kubernetes resource management. This step ensures efficient resource utilization and flexible management, contributing to effective resource management and reliable application delivery.

[0056] The typical steps for destroying a Pod unit are as follows:

[0057] Step 1: Confirm Destruction Needs: Before deciding to destroy a Pod, you first need to confirm whether it is truly necessary. This is usually based on business needs, resource utilization, or because the Pod has encountered a problem and needs to be replaced or repaired.

[0058] Step 2: Obtain Pod Information: Using the Kubernetes command-line tool kubectl, you can obtain detailed information about the Pod, including its name, namespace, and tags. This information is the basis for subsequent destruction operations.

[0059] Step 3: Check Pod Status: Before destroying a Pod, it's usually necessary to check its status to ensure it's no longer performing important tasks or processing critical data. If the Pod is running important tasks, it may be necessary to wait for the tasks to complete or migrate it to another Pod before destruction.

[0060] Step 4: Perform the destruction operation: Once you've confirmed that the Pod needs to be destroyed, you can use the `kubectl delete` command to perform the destruction operation. This command sends a request to the Kubernetes API, instructing you to delete the specified Pod.

[0061] Step 5: Wait for the Pod to be destroyed: Kubernetes will begin terminating the containers in the Pod and releasing the resources they occupy. This process may take some time, depending on the termination policy of the containers in the Pod and whether a graceful termination grace period is configured.

[0062] Step Six: Verify the Destruction Result: After the destruction operation is complete, you need to verify whether the Pod has been successfully deleted. You can use the `kubectl get pods` command to view the current Pod list and confirm that the target Pod no longer appears in the list.

[0063] Step 7: Clean up related resources: If the Pod uses storage volumes or other persistent resources, you may also need to manually clean up these resources to avoid resource waste or potential security risks.

[0064] Optionally, after destroying the Pod unit, the method further includes: reclaiming the resources occupied by the Pod unit.

[0065] In this embodiment, the resource recycling process typically includes the following steps:

[0066] Step 1: Resource Cleanup: First, the system cleans up all resource objects associated with the Pod unit, such as storage volumes and network connections. These resource objects are no longer needed after the Pod is destroyed, so they must be cleaned up to release resources.

[0067] Step 2: Resource Release: After cleaning up resource objects, the system releases the actual resources occupied by the Pod unit, such as CPU and memory. These resources are marked as available and can be used by other Pods or workloads.

[0068] Step 3: Update Resource Status: After resource reclamation is complete, the system updates the resource status information to reflect the current amount of available resources. This way, when a new Pod or workload requires resources, the scheduler can make decisions based on the latest resource status information.

[0069] When a Pod is destroyed, the resources it occupied (such as CPU, memory, and storage) are not automatically released back to the system. Without resource reclamation, these resources remain occupied, preventing other Pods or workloads from obtaining the necessary resources, thus impacting the performance and stability of the entire cluster. Through resource reclamation, Kubernetes clusters can maintain high resource utilization, ensuring that more workloads can still run even with limited resources. At the same time, resource reclamation also helps reduce resource waste and lower operating costs.

[0070] Furthermore, such as Figure 3 The diagram shows a schematic of a K8S big data solution architecture according to this embodiment.

[0071] Big data administrators build big data engine images, which are container images pre-installed with big data processing tools and frameworks (such as Flink and Spark). They provide a basic runtime environment for big data applications and services, enabling developers to build applications involving data processing and analysis. Big data engine images pre-install big data processing tools and frameworks, which are binary files or compressed packages (such as JAR files) that can be called at runtime.

[0072] User-built custom images are images created by users based on their application needs, starting with a big data engine image and adding specific configurations, libraries, application code, or other dependencies. These images typically reflect the user's specific business logic, data processing methods, or software dependencies, enabling the user's application to run effectively in a containerized environment. User-built custom images are images created by users based on their application needs, including user code and other dependencies besides big data processing (such as TensorFlow, NumPy, etc.).

[0073] A user creates a Pod object in a Kubernetes (K8S) environment. K8S is an open-source platform for automatically deploying, scaling, and managing containerized applications. It provides a framework for container orchestration, enabling applications to run and be monitored in physical or virtual machine clusters, ensuring high availability and scalability. This Pod object consists of three parts: an initialization container, a main container, and shared storage. A container is a lightweight software virtualization technology used to encapsulate and run applications and their dependencies. Containers are isolated from other parts of the system, ensuring they have their own file system, CPU, memory, process space, and other resources, allowing applications to run in isolation from other applications on any computer that supports a container runtime environment.

[0074] Initialize container settings: In the Kubernetes Pod definition, an initialization container is introduced. This initialization container uses the Big Data Engine image. Its task is to download and prepare the binary files or compressed packages required by the big data environment and store them in shared storage.

[0075] The main container runs: The actual application container then starts, using a user-defined image. It locates the large data dependencies placed in shared storage by the initialization container. When the application starts, it reads the already loaded large data dependencies and runs the user code.

[0076] Shared storage: Data from the big data engine image is stored in a shared module, thus preventing the second image from needing to be updated along with the first image when the first image's data is updated. This decouples the first and second images, removes their dependency, and improves flexibility. Specifically, an Empty Dir is configured, which will be used to share data between the InitContainer and the application container.

[0077] When updating big data dependencies or user dependencies, they can be done independently, reducing the complexity of the update. User images no longer need to include big data dependencies, making them more lightweight, and big data engine image updates do not require repeated builds of user images, saving build resources.

[0078] Based on the solutions proposed in the above embodiments, the solution proposed in this embodiment has the following technical advantages: Simplified image updates: Updates to big data dependencies or user dependencies can be performed independently, reducing update complexity. Simplified collaboration between the engine team and the business team: Both parties only need to define the user interface of the big data component without modification, and can independently develop and deploy. Reduced image size: User images no longer need to include big data dependencies, making them more lightweight, and updates to the big data engine image do not require repeated building of user images, saving build resources. Improved flexibility and maintainability: Users can update their dependencies more flexibly, and system administrators can update the big data engine image more easily. Improved deployment efficiency: Because there is no need to worry about building dependent user images, the entire system is simpler. Simplified version tracking: Users and the engine release versions independently, without interdependence, and the overall cluster can ensure unified engine version updates and upgrades without fragmentation issues.

[0079] Furthermore, as Figure 1 and Figure 3 To provide a specific implementation of the method shown, this embodiment offers a data processing device, such as... Figure 4 As shown, the device includes: a creation unit 401, a first generation unit 402, and a second generation unit 403.

[0080] Create unit 401, configured to create Pod units in response to input commands;

[0081] The first generation unit 402 is configured to use the initial container to generate an initial running environment based on the first data image, and to store the data corresponding to the initial running environment in the shared space;

[0082] The second generation unit 403 is configured to load the second data image using the application container, read the data corresponding to the initial running environment in the shared space, generate the target running environment, and execute the target application.

[0083] The first data image includes a pre-built big data engine image, used to generate the initial runtime environment; the second data image includes a custom image with dependencies for executing the target application, used to generate the target runtime environment.

[0084] In a specific application scenario, the first generation unit 402 is further configured to initiate a permission verification request, and if the verification result is a type of permission, it responds to the update instruction input by the first data interface to build and update the big data engine image.

[0085] In specific application scenarios, the second generation unit 403 is further configured to initiate a permission verification request, and if the verification result is a Class II permission, in response to the instruction input from the second data interface, add configuration information that establishes a dependency relationship with the target application.

[0086] In a specific application scenario, the creation unit 401 is specifically configured to determine the number of application containers based on the number of target applications to be executed; and to create the shared space in the Pod unit based on the required resource requirements.

[0087] In a specific application scenario, the creation unit 401 is specifically configured to create a shared space of the temporary storage volume type, and the capacity of the shared space meets the required resource requirements.

[0088] In a specific application scenario, creation unit 401 is specifically configured to destroy the Pod unit.

[0089] In a specific application scenario, creation unit 401 is specifically configured to reclaim the resources occupied by the Pod unit.

[0090] It should be noted that other corresponding descriptions of the functional units involved in the data processing apparatus provided in this embodiment can be found in [reference needed]. Figure 1 and Figure 3 The corresponding descriptions in [the document] will not be repeated here.

[0091] Based on the above, Figure 1 and Figure 3 Accordingly, this embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. Figure 1 and Figure 3 The method shown.

[0092] Based on the above, Figure 1 and Figure 3 Accordingly, this embodiment also provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the above-described method. Figure 1 and Figure 3 The method shown.

[0093] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0094] Based on the above, Figure 1 and Figure 3 The method shown, and Figure 4 To achieve the above objectives, this application also provides an electronic device, which can be configured on a computer side, etc. The device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-described objectives. Figure 1 and Figure 3 The method shown.

[0095] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0096] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0097] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented using hardware. Applying the scheme of this embodiment, the data image is first divided into a big data engine image and a user-defined image. Then, a shared space is set up in the Pod unit, and the data corresponding to the initial running environment is stored in the shared module, enabling the second data image to directly read the data in the shared module to derive the required running environment. This avoids the problem that the second data image also needs to be updated when the first data image is updated. It achieves decoupling between the two and improves flexibility.

[0099] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0100] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features of this application.

[0101] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. As used herein, the term “and / or” means including one or more of the associated listed any and all possible combinations. Additionally, when used in this application, the terms “comprise” and its variations “comprises” and / or “comprising” refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase “comprising a…” does not exclude the presence of additional identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment focuses on describing the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, apparatuses, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0102] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0103] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A data processing method, characterized in that, include: In response to input commands, create a Pod unit; The Pod unit includes an initial container, at least one application container, and a shared space; The initial container is used to generate an initial runtime environment based on the first data image, and the data corresponding to the initial runtime environment is stored in the shared space; The application container loads the second data image, reads the data corresponding to the initial runtime environment in the shared space, generates the target runtime environment, and executes the target application. The first data image includes a pre-built big data engine image, used to generate the initial runtime environment; the second data image includes a custom image with dependencies for executing the target application, used to generate the target runtime environment.

2. The method according to claim 1, characterized in that, Before generating the initial runtime environment based on the first data image using the initial container, the method further includes: Upon receiving a request to update the big data engine image, an authentication request is initiated; If the authentication result indicates Level 1 access, the big data engine image is built and updated in response to the update command input from the first data interface. The first-level permission is the administrator permission, and the first data interface is used to receive the update command input by the administrator.

3. The method according to claim 1, characterized in that, Before loading the second data image using the application container, the method further includes: Upon receiving a request to modify the custom image, an authentication request is initiated; If the authentication result is a level 2 permission, in response to the modification instruction input from the second data interface, add configuration information that establishes a dependency relationship with the target application; Based on the configuration information, a custom image with the dependencies of the target application is generated; The second-level permission is a user permission, and the second data interface is used to receive the modification instructions input by the user.

4. The method according to claim 1, characterized in that, The input instructions include the target application to be executed and the required resources; The process of creating a Pod unit in response to an input command includes: The number of application containers is determined based on the number of target applications that need to be executed; Based on the required resources, the shared space is created in the Pod unit.

5. The method according to claim 4, characterized in that, Creating the shared space within the Pod unit includes: Create a shared space of type temporary storage volume, and the capacity of the shared space meets the required resource requirements.

6. The method according to claim 1, characterized in that, After executing the target application, the method further includes: The Pod unit is destroyed, and the resources occupied by the destroyed Pod unit are reclaimed.

7. A data processing apparatus, characterized in that, include: Create a unit, configured to create a Pod unit in response to an input command; The Pod unit includes an initial container, at least one application container, and a shared space; The first generation unit is configured to use the initial container to generate an initial running environment based on the first data image, and to store the data corresponding to the initial running environment in the shared space; The second generation unit is configured to load the second data image using the application container, read the data corresponding to the initial running environment in the shared space, generate the target running environment, and execute the target application. The first data image includes a pre-built big data engine image, used to generate the initial runtime environment; the second data image includes a custom image with dependencies for executing the target application, used to generate the target runtime environment.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

9. A computer program product having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

Images