Computing component containerization deployment method, device, computer equipment and storage medium

Abstract

This application relates to a method, apparatus, computer device, and storage medium for containerized deployment of computing components. The method includes: responding to a request from a user node for a Flink computing task, obtaining Flink computing task information; performing containerized orchestration on the Flink computing task information to generate a containerized orchestration file; obtaining a Flink object instance corresponding to the Flink computing component; and deploying the Flink object instance to the corresponding target container node based on the containerized orchestration file. This method can reduce the probability of problems such as inability to query normally, service interruption, and crashes in Flink computing components, ensuring the normal operation of Flink computing components.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, and storage medium for containerized deployment of computing components. Background Technology

[0002] Flink is a distributed streaming computing component that can efficiently process unbounded or bounded data streams. However, when performing large-scale data computations based on Flink, if users submit overloaded tasks, Flink may struggle to automatically allocate these tasks, leading to issues such as inability to query normally, service interruptions, and crashes, thus affecting the normal operation of Flink. Summary of the Invention

[0003] Therefore, it is necessary to provide a containerized deployment method, apparatus, computer equipment, and storage medium for computing components that can ensure the normal operation of Flink computing components when users submit overloaded tasks, in order to address the above-mentioned technical problems.

[0004] Firstly, this application provides a method for containerized deployment of computing components. The method includes:

[0005] In response to a user node's request for a Flink computing task, obtain Flink computing task information;

[0006] The Flink computing task information is containerized and orchestrated to generate a containerized orchestration file;

[0007] Obtain the Flink object instance corresponding to the Flink computing component, and deploy the Flink object instance to the corresponding target container node based on the containerized orchestration file.

[0008] In one embodiment, the containerized orchestration file includes Flink instance information and deployment container information;

[0009] The step of deploying the Flink object instance to the corresponding target container node based on the containerized orchestration file includes:

[0010] Based on the Flink instance information, start the Flink task management module instance and the Flink task execution module instance;

[0011] Based on the deployment container information, the Flink task management module instance and the Flink task execution module instance are deployed to the corresponding target container nodes.

[0012] In one embodiment, after deploying the Flink task management module instance and the Flink task execution module instance to the corresponding target container nodes based on the deployment container information, the method further includes:

[0013] Determine the raw computation data and metadata of the Flink computation task, and cache the raw computation data and metadata;

[0014] Obtain intermediate computation result data generated by each operator when executing the Flink computation task, and cache the intermediate computation result data.

[0015] In one embodiment, the method further includes:

[0016] In response to the workspace configuration command, establish the resource usage spaces corresponding to the Flink computing task;

[0017] In response to the identification command, each container node is assigned a corresponding node identifier and each user node is assigned an anti-affinity taint, wherein the node identifier and the anti-affinity taint together characterize the resource scheduling relationship between the container node and the user node.

[0018] In one embodiment, the method further includes at least one of the following:

[0019] In response to instance adjustment instructions, adjust the Flink object instances in the preset workspace;

[0020] Obtain the resource usage information corresponding to the Flink computing task, and adjust the Flink object instances in the preset workspace based on the resource usage information.

[0021] In one embodiment, the method further includes:

[0022] In response to the configuration adjustment command, the application configuration information and workspace configuration information are adjusted, wherein the application configuration information is used to manage the preset workspace and the application configuration information is used to manage the Flink object instance.

[0023] Secondly, this application also provides a containerized deployment apparatus for computing components. The apparatus includes:

[0024] The configuration management module is used to respond to requests from user nodes for Flink computing tasks and obtain Flink computing task information.

[0025] The service management module is used to containerize and orchestrate the Flink computing task information and generate containerized orchestration files.

[0026] The containerization base module is used to obtain the Flink object instance corresponding to the Flink computing component, and deploy the Flink object instance to the corresponding target container node based on the containerization orchestration file.

[0027] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0028] In response to a user node's request for a Flink computing task, obtain Flink computing task information;

[0029] The Flink computing task information is containerized and orchestrated to generate a containerized orchestration file;

[0030] Obtain the Flink object instance corresponding to the Flink computing component, and deploy the Flink object instance to the corresponding target container node based on the containerized orchestration file.

[0031] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0032] In response to a user node's request for a Flink computing task, obtain Flink computing task information;

[0033] The Flink computing task information is containerized and orchestrated to generate a containerized orchestration file;

[0034] Obtain the Flink object instance corresponding to the Flink computing component, and deploy the Flink object instance to the corresponding target container node based on the containerized orchestration file.

[0035] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0036] In response to a user node's request for a Flink computing task, obtain Flink computing task information;

[0037] The Flink computing task information is containerized and orchestrated to generate a containerized orchestration file;

[0038] Obtain the Flink object instance corresponding to the Flink computing component, and deploy the Flink object instance to the corresponding target container node based on the containerized orchestration file.

[0039] The aforementioned containerized deployment method, apparatus, computer device, and storage medium for computing components, in response to Flink computing task requests, obtain Flink computing task information, and perform containerized orchestration on the Flink computing task information to generate a containerized orchestration file. This transforms the Flink computing task information into an orchestration file for containerization. Then, after obtaining the Flink object instance corresponding to the Flink computing component, based on the containerized orchestration file, the Flink object instance can be deployed to the corresponding target container node, achieving containerization of the Flink computing component. This involves rationally allocating Flink object instances, which serve as Flink computing node resources, to different container nodes. Therefore, this application achieves the distribution and isolation of Flink computing node resources, enabling more rational task and resource scheduling when receiving overloaded tasks. This reduces the probability of problems such as Flink computing components failing to query normally, service stopping, and crashing, ensuring the normal operation of the Flink computing component. Attached Figure Description

[0040] Figure 1 This is an application environment diagram of a containerized deployment method for computing components in one embodiment;

[0041] Figure 2 This is a flowchart illustrating a method for containerizing and deploying computing components in one embodiment.

[0042] Figure 3 This is a flowchart illustrating a method for containerizing and deploying computing components in another embodiment;

[0043] Figure 4 This is a schematic diagram of the composition architecture of a multi-tenant architecture in one embodiment;

[0044] Figure 5 This is a system architecture diagram of a containerized computing component system in one embodiment;

[0045] Figure 6 This is a schematic diagram of the module structure for containerized deployment of computing components in one embodiment;

[0046] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] The containerized deployment method for computing components provided in this application can be applied to, for example... Figure 1 In the application environment shown, the first terminal 102 communicates with the second terminal 104 via a network. The first terminal 102 can serve as a terminal for deploying user nodes, and the second terminal 104 can serve as a terminal for container nodes. The first terminal 102 can send a request for a Flink computing task to the second terminal 104. In response to the request, the second terminal 104 will obtain Flink computing task information; perform containerized orchestration on the Flink computing task information to generate a containerized orchestration file; obtain the Flink object instance corresponding to the Flink computing component; and deploy the Flink object instance to the corresponding target container node based on the containerized orchestration file. The first terminal 102 and the second terminal 104 can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, portable wearable devices, and servers. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The second terminal 104 can be a single terminal or a cluster; similarly, the first terminal 102 can be a single terminal or a cluster.

[0049] In one embodiment, such as Figure 2 As shown, a containerized deployment method for computing components is provided, including the following steps:

[0050] Step S10: In response to the request from the user node for the Flink computing task, obtain the Flink computing task information.

[0051] It's important to note that Flink computing tasks can be big data computing tasks based on Flink. Flink is a distributed stream processing framework, primarily composed of an API & Libraries layer, a Runtime core layer, and a physical deployment layer. The API & Libraries layer provides programming interfaces and top-level libraries; the Runtime core layer is the core implementation layer of the Flink distributed computing framework, used for job transformation, task scheduling, resource allocation, and task execution; and the physical deployment layer supports deploying and running Flink applications on different platforms.

[0052] The containerized deployment method for computing components in this application can be applied to a containerized deployment system for computing components, which consists of a configuration management module, a service management module, and a containerization base module.

[0053] As an example, step S10 includes: responding to a request from a user node for a Flink computing task, obtaining Flink computing task information; encapsulating the Flink computing task information into Flink parameter information according to a preset format through the configuration management module; and sending the Flink parameter information to the service management module through the interface between the configuration management module and the service management module. In this way, the service management module can obtain the Flink computing task information by decapsulating the Flink parameter information.

[0054] As an example, Flink computation task information can be user-submitted job data, such as JAR files, data flow graphs, and job graphs.

[0055] Step S20: Containerize and orchestrate the Flink computing task information to generate a containerized orchestration file.

[0056] As an example, a containerized orchestration file can be a containerized scheduling orchestration YAML file, which is used to orchestrate resource objects in a Kubernetes cluster. A containerized scheduling orchestration YAML file can be considered a resource manifest file.

[0057] As an example, current applications are typically composed of multiple containerized components that work together to enable the application to function as designed. Containerization orchestration refers to the process of organizing the work of individual containerized components and the application layer, and containerization orchestration documents can be considered as guidelines for organizing the work of individual containerized components and the application layer.

[0058] As an example, step S20 includes: decapsulating the Flink parameter information through the service management module to obtain the task scheduling information, task execution information, and user information of the user node for the Flink computing task; establishing a containerized system scheduling and orchestration file based on the task scheduling information, task execution information, and user information to obtain a containerized orchestration file; and sending the containerized orchestration file to the containerized base module through the service management module.

[0059] Step S30: Obtain the Flink object instance corresponding to the Flink computing component, and deploy the Flink object instance to the corresponding target container node based on the containerized orchestration file.

[0060] It should be noted that the containerization base module includes a containerization platform, which can be the Kubernetes platform.

[0061] As an example, step S30 includes: starting each Flink object instance through the containerization platform based on the containerization orchestration file; obtaining user permission information passed by the service management module; and fixing each Flink object instance to the corresponding target container node for operation.

[0062] As an example, user permission information is used to identify the deployment priority of user nodes for each container node. User permission information can be a label or an anti-affinity taint.

[0063] The aforementioned containerized deployment method, apparatus, computer device, and storage medium for computing components, in response to Flink computing task requests, obtain Flink computing task information, and perform containerized orchestration on the Flink computing task information to generate a containerized orchestration file. This transforms the Flink computing task information into an orchestration file for containerization. Then, after obtaining the Flink object instance corresponding to the Flink computing component, based on the containerized orchestration file, the Flink object instance can be deployed to the corresponding target container node, achieving containerization of the Flink computing component. This involves rationally allocating Flink object instances, which serve as Flink computing node resources, to different container nodes. Therefore, this application achieves the distribution and isolation of Flink computing node resources, enabling more rational task and resource scheduling when receiving overloaded tasks. This reduces the probability of problems such as Flink computing components failing to query normally, service stopping, and crashing, ensuring the normal operation of the Flink computing component.

[0064] In one embodiment, based on the above embodiments, referring to Figure 3 The containerized orchestration file includes Flink instance information and deployment container information. The containerized foundation module includes a containerized platform. Deploying the Flink object instance to the corresponding target container node based on the containerized orchestration file includes:

[0065] Step S31: Based on the Flink instance information, start the Flink task management module instance and the Flink task execution module instance.

[0066] It should be noted that Flink instance information is used to determine which Flink object instances need to be launched for this Flink computing task. Flink instance information can be instance name or identifier, etc. Deployment container information is used to determine which container nodes need to be scheduled for this Flink computing task. Container nodes can be running nodes in a Kubernetes cluster, and each container node includes one or more containers.

[0067] As an example, step S31 includes: generating corresponding instance startup information through a containerization platform based on the Flink instance information; feeding back the instance startup information to the service management module through the containerization platform; and starting the Flink task management module instance and the Flink task execution module instance through the service management module in response to the instance startup information.

[0068] As an example, an instance of the Flink task management module can be an instance of the Flink core component JobManager, and an instance of the Flink task execution module can be an instance of the Flink core component TaskManager.

[0069] Step S32: Based on the deployment container information, deploy the Flink task management module instance and the Flink task execution module instance to the corresponding target container nodes.

[0070] As an example, container deployment information can be one or more of the following: node tags of container nodes or anti-affinity taints of user nodes. Node tags are used to identify whether a container node can be used as the running node for a Flink computing task submitted by a user node, while anti-affinity taints are used to identify the priority of a user node in selecting a container node, or to identify whether a user node has selected the corresponding container node.

[0071] As an example, step S32 includes: obtaining the node tags of each container node and the anti-affinity taints of user nodes in the container deployment information; selecting target container nodes among the container nodes based on the node tags and anti-affinity taints; and deploying the Flink task management module instance and the Flink task execution module instance to the corresponding target container nodes through the containerization platform. This embodiment of the application, by setting node tags for container nodes and anti-affinity taints for user nodes, enables different user nodes to use different computing node resources. This allows for effective management of the computing resources of the containerization platform, and different user nodes can be reasonably allocated corresponding computing node resources. Therefore, when receiving overloaded tasks, task and resource scheduling can be performed more rationally, reducing the probability of problems such as Flink computing components failing to query normally, service stopping, and crashing, thus ensuring the normal operation of Flink computing components.

[0072] In one embodiment, based on the above embodiments, the containerization base module further includes a data caching system. After deploying the Flink task management module instance and the Flink task execution module instance to the corresponding target container nodes based on the deployment container information, the method further includes:

[0073] Determine the original computation data and metadata of the Flink computation task, and cache the original computation data and metadata; obtain the intermediate computation result data generated by each operator when executing the Flink computation task, and cache the intermediate computation result data.

[0074] It should be noted that the containerization base module can consist of a containerization platform and a data caching system, which is used to cache the data of the containerization platform.

[0075] As an example, if the raw data and metadata of a Flink computing task are received through a containerized platform, the raw computing data and metadata are cached through a data caching system to speed up the reading of data files during the execution of the Flink computing task. During the execution of the Flink computing task, intermediate computing results generated by each operator are obtained and cached through the data caching system to reduce the time spent running intermediate operators during the execution of the Flink computing task. In this way, the embodiments of this application can solve the problems of data loading time and intermediate operator running time during the execution of Flink computing tasks, and can improve the computing efficiency of the Flink computing component. Therefore, when receiving overloaded tasks, there is more room for reasonable scheduling of tasks and resources, reducing the probability of problems such as Flink computing components failing to query normally, service stopping, and crashing, and ensuring the normal operation of the Flink computing component.

[0076] In one embodiment, based on the above embodiments, the configuration management module includes a workspace management module, and the method further includes:

[0077] In response to the workspace configuration command, establish the resource usage spaces corresponding to the Flink computing task; in response to the identification command, assign a corresponding node identifier to each container node and assign an anti-affinity taint to each user node, wherein the node identifier and the anti-affinity taint together characterize the resource scheduling relationship between the container node and the user node.

[0078] As an example, refer to Figure 4 , Figure 4 This is a schematic diagram of a multi-tenant architecture in an embodiment of this application. Node 1, Node 2, and Node 3 are all container nodes, and Tenant 1, Tenant 2, and Tenant 3 are all user nodes. Label is a node tag, anti-affinity refers to an anti-affinity taint, and Flink pod is a container for Flink computing tasks. For example, if the node tag of Node 1 is Tenant 2, it indicates that user node "Tenant 2" can select Node 1 as its own container node. If the anti-affinity taint of user node "Tenant 1" is "Node 2", it indicates that user node "Tenant 1" cannot select Node 2 as its own container node.

[0079] As an example, in response to the workspace configuration command, the resource usage spaces corresponding to the Flink computing task are established through the workspace management module; in response to the identification command, each container node is labeled with a corresponding node identifier through each of the first type interfaces in the workspace management module; and each user node is labeled with an anti-affinity taint through each of the second type interfaces in the workspace management module. The node identifier and the anti-affinity taint are used together to identify the resource scheduling relationship between the container node and the user node. The node label is used to identify whether the container node can be used as the running node for the Flink computing task submitted by the user node, and the anti-affinity taint is used to identify the priority of the user node in selecting the container node, or to identify whether the user node has selected the corresponding container node. Thus, this embodiment of the application implements setting node labels for container nodes and setting anti-affinity taints for user nodes. Subsequently, when executing Flink computing tasks, different user nodes can use different computing node resources, thereby achieving a reasonable allocation of computing node resources.

[0080] In one embodiment, based on the above embodiments, the configuration management module further includes an application configuration module, and the method further includes:

[0081] In response to instance adjustment instructions, adjust the Flink object instances in the preset workspace.

[0082] As an example, in response to instance adjustment instructions, the Flink object instances in the preset workspace are adjusted through the application configuration module. The adjustment methods can include adding instances, deleting instances, querying instances, and modifying instances.

[0083] In one embodiment, based on the above embodiments, the configuration management module further includes an application configuration module, and the method further includes:

[0084] Obtain the resource usage information corresponding to the Flink computing task, and adjust the Flink object instances in the preset workspace based on the resource usage information.

[0085] As an example, resource usage information corresponding to Flink computing tasks is obtained, where the resource usage information can be the amount of resources occupied by each Flink object instance. Based on the resource usage information, Flink object instances with resource usage exceeding a preset resource threshold are identified in a preset workspace as Flink object instances to be reduced. The reduction is then performed on the Flink object instances to be reduced through the application configuration module. In this way, the embodiments of this application can achieve the adjustment and optimization of Flink instance objects, allowing for selective reuse of these Flink instance objects in the next deployment of Flink computing components, thereby improving the efficiency of the next deployment of Flink computing components.

[0086] In one embodiment, based on the above embodiments, the configuration management module further includes a template configuration module, and the method further includes:

[0087] In response to the configuration adjustment command, the application configuration information and workspace configuration information are adjusted, wherein the application configuration information is used to manage the preset workspace and the application configuration information is used to manage the Flink object instance.

[0088] As an example, the workspace configuration information is the module configuration information of the workspace management module, which is used to adjust the module configuration of the workspace management module to manage the preset workspace; the application configuration information is the module configuration information of the application configuration module, which is used to adjust the module configuration of the application configuration module. In this way, by adjusting the module configuration of the application configuration module, the Flink object instance can be deployed to another container node, thereby managing the Flink object instance.

[0089] As an example, in response to a configuration adjustment command, the template configuration module adjusts the module configurations in the application configuration module and the workspace management module. Module configurations can include creating or deleting resource usage spaces, assigning or deleting node tags and anti-affinity taints, etc. Thus, this embodiment of the application, by setting a configuration management module, can adjust the module configurations of the application configuration module and the workspace management module. This allows for selective reuse of the application configuration module and the workspace management module during subsequent deployments of Flink computing components, improving the efficiency of subsequent deployments.

[0090] Reference Figure 5 , Figure 5This is a system architecture diagram of the containerization system for computing components in this embodiment of the application. The containerization system for computing components includes a configuration management module, a service management module, and a containerization base module. The configuration management module is used to obtain Flink computing task information and send it to the service management module. The service management module is used to perform containerization orchestration based on the Flink computing task information, generate a containerization orchestration file, and send the containerization orchestration file to the containerization base module. The containerization base module is used to deploy Flink object instances to the corresponding target container nodes based on the containerization orchestration file.

[0091] As an example, the configuration management module includes a workspace management module, an application configuration module, and a template configuration module. The workspace management module is used to establish the resource usage spaces corresponding to each Flink computing task, and to assign corresponding node identifiers to each container node and anti-affinity taints to each user node; the application configuration module is used to adjust Flink object instances in the preset workspace, and / or to adjust Flink object instances in the preset workspace based on resource usage information; the template configuration module is used to adjust the module configurations of the application configuration module and the workspace management module.

[0092] As an example, the service management module also provides features such as a deployment engine, a service engine, and an instance controller.

[0093] As an example, the containerization foundation module includes a containerization platform and a data caching system. The containerization platform provides container management functionality, and the data caching system provides caching acceleration functionality.

[0094] As an example, the containerization platform is used to launch Flink task management module instances and Flink task execution module instances based on Flink instance information, and to deploy Flink task management module instances and Flink task execution module instances to the corresponding target container nodes based on deployment container information; the data caching system is used to cache the original computation data and metadata of Flink computation tasks, as well as cache the intermediate computation result data generated by each operator when executing Flink computation tasks.

[0095] As an example, the containerization foundation module also includes a log module and an operation and maintenance monitoring module. The log module is used to collect the running logs of Flink computing components, and the operation and maintenance monitoring module is used to detect the status of Flink computing components.

[0096] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0097] Based on the same inventive concept, this application also provides a computing component containerization deployment apparatus for implementing the above-described computing component containerization deployment method. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more computing component containerization deployment apparatus embodiments provided below can be found in the limitations of the computing component containerization deployment method described above, and will not be repeated here.

[0098] In one embodiment, such as Figure 6 As shown, a containerized deployment apparatus for computing components is provided, comprising:

[0099] The configuration management module 310 is used to obtain Flink computing task information in response to requests from user nodes for Flink computing tasks.

[0100] Service management module 320 is used to containerize and orchestrate the Flink computing task information and generate containerized orchestration files;

[0101] The containerization base module 330 is used to obtain the Flink object instance corresponding to the Flink computing component and deploy the Flink object instance to the corresponding target container node based on the containerization orchestration file.

[0102] In one embodiment, the containerization orchestration file includes Flink instance information and deployment container information, and the containerization base module 330 is further used for:

[0103] Based on the Flink instance information, start the Flink task management module instance and the Flink task execution module instance; based on the deployment container information, deploy the Flink task management module instance and the Flink task execution module instance to the corresponding target container nodes respectively.

[0104] In one embodiment, the device further includes:

[0105] The data caching module (not shown in the figure) is used to determine the original computation data and metadata of the Flink computation task and cache the original computation data and metadata; and to obtain the intermediate computation result data generated by each operator when executing the Flink computation task and cache the intermediate computation result data.

[0106] In one embodiment, the device further includes:

[0107] The workspace configuration module (not shown) is used to establish the resource usage space corresponding to the Flink computing task in response to the workspace configuration command; and to assign a corresponding node identifier to each container node and an anti-affinity taint to each user node in response to the identifier command, wherein the node identifier and the anti-affinity taint together characterize the resource scheduling relationship between the container node and the user node.

[0108] In one embodiment, the device further includes:

[0109] The first instance adjustment module (not shown in the figure) is used to adjust the Flink object instances in the preset workspace in response to instance adjustment commands.

[0110] In one embodiment, the device further includes:

[0111] The second instance adjustment module (not shown in the figure) is used to obtain the resource usage information corresponding to the Flink computing task, and adjust the Flink object instances in the preset workspace based on the resource usage information.

[0112] In one embodiment, the device further includes:

[0113] The template configuration adjustment module (not shown in the figure) is used to adjust the application configuration information and workspace configuration information in response to the configuration adjustment command. The application configuration information is used to manage the preset workspace and the application configuration information is used to manage the Flink object instance.

[0114] Each module in the aforementioned containerized deployment device for computing components can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can invoke and execute the operations corresponding to each module.

[0115] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as strings. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a containerized deployment method for computing components.

[0116] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0117] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0118] In response to a request from a user node for a Flink computing task, obtain Flink computing task information; perform containerized orchestration on the Flink computing task information to generate a containerized orchestration file; obtain the Flink object instance corresponding to the Flink computing component, and deploy the Flink object instance to the corresponding target container node based on the containerized orchestration file.

[0119] In one embodiment, the containerized orchestration file includes Flink instance information and deployment container information, and the processor, when executing the computer program, further implements the following steps:

[0120] Based on the Flink instance information, start the Flink task management module instance and the Flink task execution module instance; based on the deployment container information, deploy the Flink task management module instance and the Flink task execution module instance to the corresponding target container nodes respectively.

[0121] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0122] Determine the original computation data and metadata of the Flink computation task, and cache the original computation data and metadata; obtain the intermediate computation result data generated by each operator when executing the Flink computation task, and cache the intermediate computation result data.

[0123] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0124] In response to the workspace configuration command, establish the resource usage spaces corresponding to the Flink computing task; in response to the identification command, assign a corresponding node identifier to each container node and assign an anti-affinity taint to each user node, wherein the node identifier and the anti-affinity taint together characterize the resource scheduling relationship between the container node and the user node.

[0125] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0126] In response to instance adjustment instructions, adjust the Flink object instances in the preset workspace.

[0127] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0128] Obtain the resource usage information corresponding to the Flink computing task, and adjust the Flink object instances in the preset workspace based on the resource usage information.

[0129] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0130] In response to the configuration adjustment command, the application configuration information and workspace configuration information are adjusted, wherein the application configuration information is used to manage the preset workspace and the application configuration information is used to manage the Flink object instance.

[0131] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0132] In response to a request from a user node for a Flink computing task, obtain Flink computing task information; perform containerized orchestration on the Flink computing task information to generate a containerized orchestration file; obtain the Flink object instance corresponding to the Flink computing component, and deploy the Flink object instance to the corresponding target container node based on the containerized orchestration file.

[0133] In one embodiment, the containerized orchestration file includes Flink instance information and deployment container information, and the computer program, when executed by a processor, further implements the following steps:

[0134] Based on the Flink instance information, start the Flink task management module instance and the Flink task execution module instance; based on the deployment container information, deploy the Flink task management module instance and the Flink task execution module instance to the corresponding target container nodes respectively.

[0135] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0136] Determine the original computation data and metadata of the Flink computation task, and cache the original computation data and metadata; obtain the intermediate computation result data generated by each operator when executing the Flink computation task, and cache the intermediate computation result data.

[0137] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0138] In response to the workspace configuration command, establish the resource usage spaces corresponding to the Flink computing task; in response to the identification command, assign a corresponding node identifier to each container node and assign an anti-affinity taint to each user node, wherein the node identifier and the anti-affinity taint together characterize the resource scheduling relationship between the container node and the user node.

[0139] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0140] In response to instance adjustment instructions, adjust the Flink object instances in the preset workspace.

[0141] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0142] Obtain the resource usage information corresponding to the Flink computing task, and adjust the Flink object instances in the preset workspace based on the resource usage information.

[0143] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0144] In response to the configuration adjustment command, the application configuration information and workspace configuration information are adjusted, wherein the application configuration information is used to manage the preset workspace and the application configuration information is used to manage the Flink object instance.

[0145] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0146] In response to a request from a user node for a Flink computing task, obtain Flink computing task information; perform containerized orchestration on the Flink computing task information to generate a containerized orchestration file; obtain the Flink object instance corresponding to the Flink computing component, and deploy the Flink object instance to the corresponding target container node based on the containerized orchestration file.

[0147] In one embodiment, the containerized orchestration file includes Flink instance information and deployment container information, and the computer program, when executed by a processor, further implements the following steps:

[0148] Based on the Flink instance information, start the Flink task management module instance and the Flink task execution module instance; based on the deployment container information, deploy the Flink task management module instance and the Flink task execution module instance to the corresponding target container nodes respectively.

[0149] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0150] Determine the original computation data and metadata of the Flink computation task, and cache the original computation data and metadata; obtain the intermediate computation result data generated by each operator when executing the Flink computation task, and cache the intermediate computation result data.

[0151] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0152] In response to the workspace configuration command, establish the resource usage spaces corresponding to the Flink computing task; in response to the identification command, assign a corresponding node identifier to each container node and assign an anti-affinity taint to each user node, wherein the node identifier and the anti-affinity taint together characterize the resource scheduling relationship between the container node and the user node.

[0153] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0154] In response to instance adjustment instructions, adjust the Flink object instances in the preset workspace.

[0155] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0156] Obtain the resource usage information corresponding to the Flink computing task, and adjust the Flink object instances in the preset workspace based on the resource usage information.

[0157] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0158] In response to the configuration adjustment command, the application configuration information and workspace configuration information are adjusted, wherein the application configuration information is used to manage the preset workspace and the application configuration information is used to manage the Flink object instance.

[0159] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0160] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0161] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A computing component containerization deployment method, characterized in that, The method includes: In response to a user node's request for a Flink computing task, obtain Flink computing task information; The Flink computing task information is containerized and orchestrated to generate a containerized orchestration file; Obtain the Flink object instance corresponding to the Flink computing component. Based on the Flink instance information, start the Flink task management module instance and the Flink task execution module instance. Based on the deployment container information, deploy the Flink task management module instance and the Flink task execution module instance to the corresponding target container nodes, respectively. Determine the original computation data and metadata of the Flink computation task, and cache the original computation data and metadata; obtain the intermediate computation result data generated by each operator when executing the Flink computation task, and cache the intermediate computation result data.

2. The compute component containerization deployment method of claim 1, wherein, The method further includes: In response to the workspace configuration command, establish the resource usage spaces corresponding to the Flink computing task; In response to the identification command, each container node is assigned a corresponding node identifier and each user node is assigned an anti-affinity taint, wherein the node identifier and the anti-affinity taint together characterize the resource scheduling relationship between the container node and the user node.

3. The compute component containerization deployment method of claim 2, wherein, The method further includes at least one of the following: In response to instance adjustment instructions, adjust the Flink object instances in the preset workspace; Obtain the resource usage information corresponding to the Flink computing task, and adjust the Flink object instances in the preset workspace based on the resource usage information.

4. The containerized deployment method for computing components according to claim 3, characterized in that, The method further includes: In response to the configuration adjustment command, the application configuration information and workspace configuration information are adjusted, wherein the application configuration information is used to manage the preset workspace and the application configuration information is used to manage the Flink object instance.

5. A containerized deployment device for computing components, characterized in that, The device includes: The configuration management module is used to respond to requests from user nodes for Flink computing tasks and obtain Flink computing task information. The service management module is used to containerize and orchestrate the Flink computing task information and generate containerized orchestration files. The containerization base module is used to obtain the Flink object instance corresponding to the Flink computing component, start the Flink task management module instance and the Flink task execution module instance according to the Flink instance information, and deploy the Flink task management module instance and the Flink task execution module instance to the corresponding target container node according to the deployment container information. The data caching module is used to determine the original computation data and metadata of the Flink computation task and cache the original computation data and metadata; and to obtain the intermediate computation result data generated by each operator when executing the Flink computation task and cache the intermediate computation result data.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images