A data processing method, device, medium and terminal of a heterogeneous multi-cluster

Abstract

The application discloses a data processing method and device of a heterogeneous multi-cluster, a medium and a terminal, and relates to the technical field of data processing. The method comprises the following steps: adding new Kubernetes virtual nodes in a Kubernetes data processing system, and corresponding each Kubernetes virtual node to each heterogeneous cluster in the heterogeneous multi-cluster; creating an adapter for each heterogeneous cluster based on the Kubernetes virtual node; deploying a node agent corresponding to the adapter in the Kubernetes data processing system, wherein the adapter can communicate with an application programming interface server of the Kubernetes through the node agent; and converting the operation of the application programming interface server on kubelet into a job operation on the heterogeneous cluster based on the node agent. The application solves the problem that the heterogeneous multi-cluster scheduling is difficult, and realizes that the heterogeneous multi-cluster can be processed like a single cluster through the use of the Kubernetes cluster software.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-cluster data processing, and in particular to a heterogeneous multi-cluster data processing method and device, medium and terminal. BACKGROUND

[0002] A plurality of node servers are organized into a cluster through a local area network to collectively implement business services. To ensure comprehensive data processing of large-scale businesses, multiple clusters are usually required. In the case of multiple clusters in a user production environment, cluster data processing personnel need to process the running state, resource and configuration state, etc. of all service components that have an impact on business reliability in all clusters.

[0003] The existing multi-cluster or multi-cloud job scheduling data processing faces the problem of heterogeneity between clusters. Not only are the cluster hardware inconsistent between heterogeneous clusters, but the cluster scheduling data processing software used in each cluster can also be inconsistent. When using heterogeneous clusters, the different cluster scheduling data processing software can cause inconsistencies in the external interfaces of each cluster, making it difficult to schedule existing heterogeneous multi-clusters, resulting in submitted jobs not running normally and causing a great waste of cluster resources. SUMMARY

[0004] In view of the above deficiencies of the prior art, the present application aims to provide a heterogeneous multi-cluster data processing method, device, medium and terminal, which aims to solve the problem of difficult scheduling of heterogeneous multi-clusters.

[0005] To solve the above technical problems, the first aspect of the present application provides a heterogeneous multi-cluster data processing method, which comprises:

[0006] A heterogeneous multi-cluster data processing method comprises:

[0007] A new Kubernetes virtual node is added to the Kubernetes data processing system, and each Kubernetes virtual node corresponds to each heterogeneous cluster in the heterogeneous multi-cluster;

[0008] An adapter is created for each heterogeneous cluster based on the Kubernetes virtual node;

[0009] A node agent program corresponding to the adapter is deployed in the Kubernetes data processing system, wherein the adapter can communicate with the application programming interface server of Kubernetes through the node agent program;

[0010] Based on the node agent program, the operation of the application programming interface server on kubelet is converted into a job operation on the heterogeneous cluster.

[0011] As a further improved technical solution, the new Kubernetes virtual node in the Kubernetes data processing system is added, and each of the Kubernetes virtual nodes corresponds to each of the heterogeneous clusters in the heterogeneous multi-cluster includes:

[0012] By writing the Ping function that implements the provision node interface in virtual-kubelet;

[0013] By writing the notify node state function that implements the provision node interface in virtual-kubelet;

[0014] Start the virtual node service of the virtual-kubelet, add a new Kubernetes virtual node in the Kubernetes data processing system, and correspond each of the Kubernetes virtual nodes to each of the heterogeneous clusters in the heterogeneous multi-cluster.

[0015] As a further improved technical solution, the adapter is created for each of the heterogeneous clusters based on the Kubernetes virtual node includes:

[0016] Based on the Kubernetes virtual node, an adapter is created for each of the heterogeneous clusters, and a uniform interface is written for the adapter using Golang language, the uniform interface includes creating a job, stopping a job, obtaining job information and obtaining job resource specification functions, wherein the writing of the interface needs to call the native API of the adapter corresponding heterogeneous cluster.

[0017] As a further improved technical solution, a node agent program corresponding to the adapter is deployed in the Kubernetes data processing system, wherein the adapter can communicate with the application programming interface server of Kubernetes through the node agent program includes:

[0018] Based on the virtual-kubelet open source framework, a node agent program is created;

[0019] Write the interface standard of the node agent program to realize the create pod, delete pod, get pod information, notify pod and notify node state functions of the PodLifecycleHandler interface in the virtual-kubelet open source framework;

[0020] Based on the creating Pod, deleting Pod, obtaining Pod information, notifying Pod and notifying node state functions, the node agent program realizes communication with an application programming interface server of Kubernetes.

[0021] As a further improved technical solution, the node agent program converts operations of the application programming interface server on the kubelet into job operations on the heterogeneous cluster, and the creating Pod function of the node agent program calls a creating job function of the adapter uniform interface.

[0022] Based on the creating Pod function of the node agent program, a creating job function of the adapter uniform interface is called.

[0023] Based on the deleting Pod function of the node agent program, a stopping job function of the adapter uniform interface is called.

[0024] Based on the obtaining Pod information function of the node agent program, an obtaining job information function of the adapter uniform interface is called.

[0025] Based on the notifying Pod function of the node agent program, a job state is obtained by calling the obtaining job information function of the adapter uniform interface through a timing task, and a state of a Pod is updated.

[0026] Based on the notifying node state function of the node agent program, a resource state of a virtual node is updated by calling an obtaining job resource specification function of the adapter uniform interface through a timing task.

[0027] As a further improved technical solution, the creating job function is used to submit a request for creating a job to a corresponding heterogeneous cluster, and the stopping job function is used to stop a job running in the heterogeneous cluster.

[0028] As a further improved technical solution, the obtaining job information function is used to obtain information of a job from the heterogeneous cluster, and the obtaining job resource specification function is used to obtain information of a resource specification from the heterogeneous cluster.

[0029] As a further improved technical solution, the creating Pod function converts a single-cluster Pod created by Kubernetes into a job of the heterogeneous cluster by calling the creating job function of the adapter uniform interface, and the deleting Pod function converts a single-cluster Pod to be deleted by Kubernetes into deletion of a corresponding job of the heterogeneous cluster by calling the stopping job function of the adapter uniform interface.

[0030] As a further improved technical solution, the obtaining Pod information function converts Pod information obtained by the application programming interface server of Kubernetes into information of a corresponding job of the heterogeneous cluster by calling the obtaining job information function of the adapter uniform interface.

[0031] As a further improved technical solution, the notification Pod function notifies the Pod state information of the Kubernetes through a timing call of the adapter unified interface of the acquisition job information function, and the notification node state function acquires the heterogeneous cluster resource information through a timing call of the adapter unified interface of the acquisition job resource specification function, and notifies the node resource information of the Kubernetes.

[0032] The second aspect of the embodiment of the application provides a data processing device of a heterogeneous multi-cluster, comprising:

[0033] a virtual node module, configured to add new Kubernetes virtual nodes in a Kubernetes data processing system, and correspond each of the Kubernetes virtual nodes to each of the heterogeneous clusters in the heterogeneous multi-cluster;

[0034] a creation adapter module, configured to create an adapter for each of the heterogeneous clusters based on the Kubernetes virtual nodes;

[0035] a software creation module, configured to deploy a node agent corresponding to the adapter in the Kubernetes data processing system, wherein the adapter can communicate with an application programming interface server of the Kubernetes through the node agent;

[0036] an operation conversion module, configured to convert operations of the application programming interface server on a kubelet to job operations on the heterogeneous cluster based on the node agent.

[0037] The third aspect of the embodiment of the application provides a computer readable storage medium, the computer readable storage medium stores one or more programs, the one or more programs can be executed by one or more processors to implement the steps in the data processing method of the heterogeneous multi-cluster as any one of the above.

[0038] The fourth aspect of the embodiment of the application provides a terminal device, comprising: a processor, a memory and a communication bus; the memory stores a computer readable program which can be executed by the processor;

[0039] The communication bus realizes the connection and communication between the processor and the memory;

[0040] The processor realizes the steps in the data processing method of the heterogeneous multi-cluster as any one of the above when executing the computer readable program.

[0041] Beneficial effects: Compared with the prior art, the heterogeneous multi-cluster data processing method of the application comprises adding new Kubernetes virtual nodes in the Kubernetes data processing system, each of the Kubernetes virtual nodes corresponding to each heterogeneous cluster in the heterogeneous multi-cluster; an adapter is individually created for each of the heterogeneous clusters based on the Kubernetes virtual nodes; a node agent program corresponding to the adapter is deployed in the Kubernetes data processing system, wherein the adapter can communicate with the application programming interface server of Kubernetes through the node agent program; based on the node agent program, the operation of the application programming interface server on kubelet is converted into a job operation on the heterogeneous cluster; the application solves the problem of difficult scheduling of heterogeneous multi-cluster by using the above method, and realizes that the heterogeneous multi-cluster can be data-processed like a single cluster by using Kubernetes cluster software. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a flowchart of the heterogeneous multi-cluster data processing method of the application.

[0043] Figure 2 is a structure principle diagram of the terminal device provided by the application.

[0044] Figure 3 is a structure principle diagram of the terminal device provided by the application.

[0045] Figure 4 is a principle diagram of the heterogeneous multi-cluster data processing method of the application.

[0046] The implementation of the object of the application, functional features and advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0047] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the related drawings. The preferred embodiments of the present application are shown in the drawings. However, the present application can be implemented in many different forms, and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments of the present application, and are not intended to limit the present application.

[0049] The inventors have found that the prior art has the following problems:

[0050] (1)Kubernetes (k8s, an abbreviation of "uber" with 8 instead of 8 characters) is an open source, used for containerized applications on multiple hosts in a data processing cloud platform.

[0051] The existing multi-cluster or multi-cloud job scheduling data processing will face the problem of heterogeneity between clusters. The cluster hardware is inconsistent between the heterogeneous clusters, and the cluster scheduling data processing software used in each cluster may also be inconsistent. For example, the A cluster data processing is a supercomputer cluster, and the cluster scheduling data processing software used is Slurm; the B cluster uses the modelArt developed by Huawei, and we can only operate through the API provided by it; the C cluster is a private cluster based on Kubernetes; when using heterogeneous clusters, due to the difference in cluster scheduling data processing software, the external interface of each cluster is inconsistent, making it difficult to schedule the existing heterogeneous multi-cluster; and the existing job running needs to occupy a certain resource, and the system needs to determine which cluster resource can meet the job requirements and allocate the job to the corresponding cluster. When a data processing employee submits a job to a heterogeneous cluster for running, the system cannot allocate the job according to whether the cluster resource can meet the job requirements, resulting in that the job cannot run normally or causing great waste of cluster resources.

[0052] (2) The naive method for job scheduling of heterogeneous multi-cluster is to create a scheduler that uses the API and cluster communication of each cluster to determine the scheduling operation of the job according to the obtained information. However, this implementation method needs to implement all scheduling strategies itself and cannot use the current open source scheduler scheme, and the scalability and functionality are poor; and the cloud native open source ecosystem around the Kubernetes cluster data processing software is developing rapidly, and there are many new developments of single-cluster scheduler solutions based on Kubernetes cluster data processing software, such as the Kubernetes native scheduler kube-sheduler, the Huawei Volcano scheduler, etc. How to convert the job scheduling problem between heterogeneous multi-clusters into a scheduling problem within a single cluster based on Kubernetes, so as to fully utilize the capabilities of the existing open source projects in the Kubernetes cloud native community, and use the powerful functions of the existing Kubernetes-oriented schedulers, is a technical problem to be solved in the prior art.

[0053] In order to solve the above problems, various non-limiting embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

[0054] As Figure 1 shown, the data processing method of the heterogeneous multi-cluster provided by the embodiments of the present application comprises the following steps:

[0055] S1, adding new Kubernetes virtual nodes in the Kubernetes data processing system, each of the Kubernetes virtual nodes corresponding to each of the heterogeneous clusters in the heterogeneous multi-cluster;

[0056] Specifically, one heterogeneous cluster is represented as a virtual Kubernetes virtual node, which is logically consistent with the ordinary Kubernetes worker node in the perspective of the Kubernetes single-cluster data processing system. All capabilities under the Kubernetes single-cluster data processing system, including the capabilities of the scheduler, the job controller and all other components, can be reused, and on this basis, the open source ecological capabilities around the Kubernetes single-cluster data processing system can be used, including the open source kube-scheduler and its function-rich plug-ins, the volcano scheduler, various jobs and custom resource controllers.

[0057] The adding of the new Kubernetes virtual nodes in the Kubernetes data processing system, each of the Kubernetes virtual nodes corresponding to each of the heterogeneous clusters in the heterogeneous multi-cluster includes the following steps:

[0058] S101, implementing the Ping function of the virtual-kubelet node supply interface by writing;

[0059] S102, implementing the node state notification function of the virtual-kubelet node supply interface by writing;

[0060] S103, starting the virtual node service of the virtual-kubelet, adding a new Kubernetes virtual node in the Kubernetes data processing system, each of the Kubernetes virtual nodes corresponding to each of the heterogeneous clusters in the heterogeneous multi-cluster.

[0061] Specifically, a virtual node is implemented using the virtual-kubelet open source framework, and specifically, a Ping function implementing a NodeProvider interface in the virtual-kubelet and a NotifyNodeStatus function implementing a NodeProvider interface in the virtual-kubelet need to be written. After starting the virtual node service of the virtual-kubelet open source framework, a new Kubernetes virtual node is added in the Kubernetes data processing system, and the new Kubernetes virtual node represents a heterogeneous cluster. The principle of implementation is that the virtual-kubelet can create a Kubernetes Node resource object, but the Node resource object does not correspond to a real physical node, but a logical node.

[0062] S2, creating an adapter for each of the heterogeneous clusters based on the Kubernetes virtual node;

[0063] The adapter of each heterogeneous cluster shields the API differences of the heterogeneous cluster software. The adapter communicates downwardly with the API in each heterogeneous cluster, including submitting a job to the cluster, obtaining the running status of the cluster job, and obtaining the cluster resource information. The adapter communicates upwardly with the virtual node agent. The virtual node agent can indirectly communicate with each heterogeneous cluster using the consistent API provided by the adapter.

[0064] The adapter of each heterogeneous cluster shields the API differences of the heterogeneous cluster software. The adapter communicates downwardly with the API in each heterogeneous cluster, including submitting a job to the cluster, obtaining the running status of the cluster job, and obtaining the cluster resource information. The adapter communicates upwardly with the virtual node agent. The virtual node agent can indirectly communicate with each heterogeneous cluster using the consistent API provided by the adapter.

[0065] Based on the Kubernetes virtual node, an adapter is created for each of the heterogeneous clusters. A uniform interface is written for the adapter using the Golang language. The uniform interface includes creating a job, stopping a job, obtaining job information, and obtaining job resource specification functions. The writing of the interface needs to call the native API of the adapter corresponding to the heterogeneous cluster.

[0066] Specifically, the adapter is written in Golang language, and the adapter is mainly implemented by developers or maintainers of each heterogeneous cluster according to the interface standard defined by us. A set of unified interfaces are defined, including the functions of creating a job (CreateTrainJob), stopping a job (StopTrainJob), obtaining job information (GetTrainJob), and obtaining resource specification (GetResourceSpec). The functions of these unified interfaces are respectively called the native API of the corresponding heterogeneous cluster to realize the functions, and the communication between the adapter and the heterogeneous cluster adopts the http protocol.

[0067] The creating job function is configured to submit a request for creating a job to the corresponding heterogeneous cluster, and the stopping job function is configured to stop a job running in the heterogeneous cluster.

[0068] The obtaining job information function is configured to obtain job information from the heterogeneous cluster, and the obtaining resource specification function is configured to obtain resource specification information from the heterogeneous cluster.

[0069] Specifically, the creating job function is configured to submit a request for creating a job to the corresponding heterogeneous cluster, and the stopping job function is configured to stop a job running in the heterogeneous cluster.

[0070] The stopping job function is configured to stop a job, and the stopping job function is specifically configured to stop a job running in the heterogeneous cluster. The input parameter of the stopping job is a job ID.

[0071] The obtaining job information function is configured to obtain job information, and the obtaining job information function is specifically configured to obtain information of a certain job from the heterogeneous cluster. The input parameter of the obtaining job information is a job ID, and the return parameter of the obtaining job information includes the name of the job, the running state of the job, the starting running time of the job, and the ending running time of the job.

[0072] The obtaining resource specification function is configured to obtain resource specification information, and the obtaining resource specification function is specifically configured to obtain resource specification information from the heterogeneous cluster. The input parameter of the obtaining resource specification is a resource specification type, and the return parameter of the obtaining resource specification is whether the resource specification can be met by the heterogeneous cluster.

[0073] S3, deploying a node agent corresponding to the adapter in the Kubernetes data processing system, wherein the adapter can communicate with the application programming interface server of Kubernetes through the node agent.

[0074] Specifically, each node agent is responsible for communicating with the corresponding adapter downward, calling the unified API interface to indirectly communicate with the heterogeneous cluster, and communicating with the application programming interface server in Kubernetes upward by implementing the general interface of the Kubernetes general node agent and the application programming interface server in Kubernetes. These functions are implemented by calling the functions of the unified interface in the adapter.

[0075] The method comprises the following steps:

[0076] S301, creating a node agent based on the virtual-kubelet open source framework;

[0077] S302, writing the interface standard of the node agent, and implementing the CreatePod, DeletePod, GetPod, NotifyPods, and NotifyNodeStatus functions of the PodLifecycleHandler interface in the virtual-kubelet open source framework;

[0078] S303, based on the CreatePod, DeletePod, GetPod, NotifyPods, and NotifyNodeStatus functions, implementing the communication between the node agent and the application programming interface server in Kubernetes.

[0079] Specifically, a node agent that can communicate with the application programming interface server in Kubernetes is created for each adapter, and the implementation of the node agent uses the virtual-kubelet open source framework, thereby implementing the CreatePod, DeletePod, GetPod, NotifyPods, and NotifyNodeStatus functions of the PodLifecycleHandler interface in the virtual-kubelet open source framework. After implementing these functions, the node agent can receive various operation instructions of the application programming interface server in Kubernetes.

[0080] S4, based on the node agent, converting the operations of the application programming interface server on kubelet into job operations on the heterogeneous cluster.

[0081] Specifically, the node agent program can convert the operation of the Kubernetes application programming interface server on the kubelet to a job operation on the heterogeneous cluster.

[0082] The operation of the application programming interface server on the kubelet to a job operation on the heterogeneous cluster based on the node agent program includes the following steps:

[0083] S401, based on the creation of the Pod function of the node agent program, the adapter uniform interface is called to create a job function;

[0084] S402, based on the deletion of the Pod function of the node agent program, the adapter uniform interface is called to stop the job function;

[0085] S403, based on the Pod information function of the node agent program, the adapter uniform interface is called to get the job information function;

[0086] S404, based on the Pod function of the node agent program, the adapter uniform interface is called to get the job information function to get the job state and update the state of the Pod through a timing task;

[0087] S405, based on the node state function of the node agent program, the adapter uniform interface is called to get the job resource specification to get the resource situation of the heterogeneous cluster and update the resource state of the virtual node through a timing task.

[0088] Specifically, the creation of the Pod function converts the single-cluster Pod created by Kubernetes to create a job of the heterogeneous cluster by calling the create job function of the adapter uniform interface, and the deletion of the Pod function converts the single-cluster Pod to be deleted by Kubernetes to delete the corresponding heterogeneous cluster job by calling the stop job function of the adapter uniform interface.

[0089] Specifically, the Pod information function converts the Pod information obtained by the Kubernetes application programming interface server to the corresponding heterogeneous cluster job information by calling the get job information function of the adapter uniform interface.

[0090] Specifically, the Pod function notifies the Kubernetes of the Pod state information by calling the get job information function of the adapter uniform interface at a timing, and the node state function gets the heterogeneous cluster resource information by calling the get job resource specification function of the adapter uniform interface at a timing, and notifies the Kubernetes of the node resource information.

[0091] Specifically, the creation Pod of the node agent program calls the creation job of the adapter; the deletion Pod of the node agent program calls the stop job of the adapter; the acquisition Pod information of the node agent program calls the acquisition job information of the adapter; the notification Pod interface of the node agent program updates the state of the Pod by calling the acquisition job information to acquire the job state through a timing task; the notification node state interface of the node agent program updates the resource state of the virtual node by calling the acquisition job resource specification to acquire the resource condition of the heterogeneous cluster through a timing task.

[0092] Creation Pod: the creation Pod function implementation; when Kubernetes wants to create a single-cluster Pod, the operation is converted into creating a heterogeneous cluster job in this implementation, and the creation job function of the adapter is called to realize.

[0093] Delete Pod: the deletion Pod function implementation; when Kubernetes wants to delete a single-cluster Pod, the operation is converted into deleting the corresponding heterogeneous cluster job in this implementation, and the stop job function of the adapter is called to realize.

[0094] Acquisition Pod information: acquisition Pod information; when the application programming interface server of Kubernetes is used to acquire Pod information, the operation is converted into acquiring the corresponding heterogeneous cluster job information in this implementation, and the acquisition job information function of the adapter is called to realize.

[0095] Notification Pod: active notification Pod information change implementation; the acquisition job information function of the adapter is called in time to notify Kubernetes about the state and other information changes of the Pod.

[0096] Notification node state: active notification node information change implementation; the acquisition job resource specification of the adapter is called to acquire the resource-related information of the heterogeneous cluster to timely notify Kubernetes about the resource and other information changes of the node.

[0097] In order to process data in a heterogeneous multi-cluster as a common Kubernetes single-cluster, each cluster is represented as a virtual node, and a node agent and an adapter are created for each virtual node. Each cluster adapter hides the API differences of the heterogeneous cluster software and communicates with the API inside each cluster, including submitting jobs to the cluster, obtaining cluster job running status, and obtaining cluster resource information. The adapter communicates with the virtual node agent, which can use the consistent API provided by the adapter to communicate with each cluster indirectly. The virtual node agent communicates with the Kubernetes application programming interface server and pretends to be a common Kubernetes virtual node agent. Therefore, the virtual node agent needs to implement the functional interface of the Kubernetes virtual node agent to be able to accept Kubernetes data processing and operations.

[0098] The method has the following advantages:

[0099] The job data processing and scheduling problem of the heterogeneous multi-cluster is converted into single-cluster Kubernetes job data processing and scheduling, so that the rich and powerful capabilities around the Kubernetes open source community can be fully utilized;

[0100] The design of the adapter makes the architecture decoupled, and the virtual node agent only needs to call the unified API without calling the native API of the heterogeneous cluster. The adapter is implemented by the developers and maintainers of the heterogeneous cluster, and only needs to be implemented according to the interface standard we proposed;

[0101] The design of the virtual node agent logically represents the heterogeneous cluster as a common node, which can be directly controlled by Kubernetes. The node agent hides the lower-level heterogeneous differences and maintains the complete functions of Kubernetes.

[0102] For example:

[0103] For example, the modelArts cluster is accessed:

[0104] Step 1: Create an adapter for the modelArts cluster using the golang language, use the http protocol to communicate with modelArts, and implement the following unified interface:

[0105] Create a job: through the Post method of http, the request url is / training-jobs, and the body of the Post method is transmitted into parameters including: task name, image id, resource specification id, and start command; this function is used to create a job.

[0106] Stop job: through the Post method of http, request the url of modelArts / training-jobs / {jobid} / versions / {versionId} / stop to stop the job of jobid;

[0107] Get job information: through the Get method of http, request the url of modelArts / training-jobs / {jobId} / versions to get job information;

[0108] Get job resource specification: through the Get method of http, request the url of modelArts / flavors to get resource specification information.

[0109] Second step: create a Provider project using golang language, import the library of virtual-kubelet, and realize all functions of virtual-kubelet's supply node (NodeProvider) and notification node state interface, which need to call the unified interface of corresponding adapter, as follows:

[0110] Notify node state: through the timer, call the adapter's get job resource specification function to get the resource usage of modelArts cluster, so as to notify the resource usage of virtual node.

[0111] Create Pod: call the adapter's create job function to realize the conversion from kubernetes creating Pod to adapter creating heterogeneous cluster job.

[0112] Delete Pod: call the adapter's stop job function to realize the conversion from kubernetes deleting Pod to adapter stopping heterogeneous cluster job.

[0113] Get Pod information: call the adapter's get job information function to realize the conversion from kubernetes getting Pod information to adapter getting heterogeneous cluster job information.

[0114] Notify Pod: through the timer, call the adapter's get job information function to get the real-time state information of job, so as to notify the state information of Pod.

[0115] Third step: running the adapter service and the Provider service. The user triggers the creation of a Pod function in the Provider by creating a Pod resource in Kubernetes, which is scheduled by the Kubernetes scheduler to the virtual node represented by the Provider. The creation of the Pod function calls the creation of the job function of the adapter, which sends a creation job request to the modelArts cluster through a Post request, thereby successfully creating a heterogeneous cluster job.

[0116] Fourth step: when the state of modelArts changes to running, the Pod notifies the state information through a timer and obtains the state information through the adapter's job information function, and updates the state of the Pod to running. The running state of the Pod can be obtained by querying the command line data processing tool of Kubernetes.

[0117] Fifth step: the user initiates the deletion of the Pod resource through the command line data processing tool or application programming interface server d interface of Kubernetes, triggering the deletion of the Pod function in the Provider. The deletion of the Pod function calls the stop job function of the adapter, which sends a stop job request to the modelArts cluster through a Post request, thereby successfully stopping the heterogeneous cluster job.

[0118] Based on the above-mentioned heterogeneous multi-cluster data processing method, the embodiment provides a heterogeneous multi-cluster data processing device, comprising:

[0119] A virtual node module 1 is used to add new Kubernetes virtual nodes in the Kubernetes data processing system, and each of the Kubernetes virtual nodes corresponds to each of the heterogeneous clusters in the heterogeneous multi-cluster;

[0120] A creation adapter module 2 is used to create an adapter for each of the heterogeneous clusters based on the Kubernetes virtual nodes;

[0121] A software creation module 3 is used to deploy a node agent corresponding to the adapter in the Kubernetes data processing system, wherein the adapter can communicate with the application programming interface server of Kubernetes through the node agent;

[0122] An operation conversion module 4 is used to convert the operation of the application programming interface server on kubelet to a job operation on the heterogeneous cluster based on the node agent.

[0123] Furthermore, it is worth noting that the working process of the data processing device based on heterogeneous multi-cluster provided in this embodiment is the same as the working process of the data processing method based on heterogeneous multi-cluster described above. For details, please refer to the working process of the data processing method based on heterogeneous multi-cluster, which will not be repeated here.

[0124] Based on the above-described heterogeneous multi-cluster data processing method, this embodiment provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps in the heterogeneous multi-cluster data processing method as described in the above embodiment.

[0125] like Figure 2 As shown, based on the above-described heterogeneous multi-cluster data processing method, this application also provides a terminal device, which includes at least one processor 20; a display screen 21; and a memory 22, and may further include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can call logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0126] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0127] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0128] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0129] The heterogeneous multi-cluster data processing method of the present application compared with the prior art comprises: adding new Kubernetes virtual nodes in the Kubernetes data processing system, each of the Kubernetes virtual nodes corresponding to each of the heterogeneous clusters in the heterogeneous multi-cluster; creating an adapter for each of the heterogeneous clusters based on the Kubernetes virtual nodes; deploying a node agent program corresponding to the adapter in the Kubernetes data processing system, wherein the adapter can communicate with the application programming interface server of Kubernetes through the node agent program; converting the operation of the application programming interface server on kubelet into a job operation on the heterogeneous cluster based on the node agent program; the above method is used to solve the problem of difficult scheduling of heterogeneous multi-cluster, and the heterogeneous multi-cluster can be data-processed like a single cluster through the use of Kubernetes cluster software.

[0130] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes shall belong to the protection scope of the appended claims of the present application.

Claims

1. A data processing method for heterogeneous multi-cluster systems, characterized in that, include: Adding a new Kubernetes virtual node to the Kubernetes data processing system, and assigning each Kubernetes virtual node to each heterogeneous cluster in the heterogeneous multi-cluster system, includes: implementing the Ping function of the provisioning node interface in the virtual-kubelet; implementing the node status notification function of the provisioning node interface in the virtual-kubelet; starting the virtual node service of the virtual-kubelet, and adding a new Kubernetes virtual node to the Kubernetes data processing system, with each Kubernetes virtual node corresponding to each heterogeneous cluster in the heterogeneous multi-cluster system. Based on the Kubernetes virtual node, an adapter is created separately for each heterogeneous cluster, including: creating an adapter separately for each heterogeneous cluster based on the Kubernetes virtual node, and writing a unified interface for the adapter using the Golang language. The unified interface includes functions for creating jobs, stopping jobs, obtaining job information, and obtaining job resource specifications. The writing of the interface requires calling the native API of the heterogeneous cluster corresponding to the adapter. Deploying a node agent program corresponding to the adapter in the Kubernetes data processing system, wherein the adapter can communicate with the Kubernetes application programming interface server through the node agent program, including: creating a node agent program based on the virtual-kubelet open source framework; writing the interface standard of the node agent program to implement the functions of creating Pods, deleting Pods, obtaining Pod information, notifying Pods, and notifying node status of the PodLifecycleHandler interface in the virtual-kubelet open source framework; and realizing the communication between the node agent program and the Kubernetes application programming interface server based on the functions of creating Pods, deleting Pods, obtaining Pod information, notifying Pods, and notifying node status of the node; wherein each node agent program is responsible for communicating downward with the corresponding adapter, calling a unified API interface to indirectly communicate with heterogeneous clusters, and each node agent program communicates upward with the Kubernetes application programming interface server through the general interface of the Kubernetes ordinary node agent; Based on the node agent program, the operations of the application programming interface server on kubelet are converted into job operations on the heterogeneous cluster.

2. The data processing method for heterogeneous multi-cluster systems according to claim 1, characterized in that, The step of converting the application programming interface server's operations on kubelet into job operations on the heterogeneous cluster based on the node agent program includes: The Pod creation function of the node agent program calls the job creation function of the adapter unified interface; The delete Pod function of the node agent program calls the stop job function of the adapter unified interface; The node agent program uses the function of obtaining Pod information to call the function of obtaining job information in the unified interface of the adapter. Based on the Pod notification function of the node agent program, the job status is obtained and the Pod status is updated by calling the job information acquisition function of the adapter unified interface through a scheduled task. The node agent program's notification node status function obtains the resource status of the heterogeneous cluster by calling the adapter's unified interface through a scheduled task to retrieve job resource specifications and update the resource status of virtual nodes.

3. The data processing method for heterogeneous multi-cluster systems according to claim 2, characterized in that, The job creation function is used to submit a request to create a job to the corresponding heterogeneous cluster, and the job stop function is used to stop a job that is running in the heterogeneous cluster.

4. The data processing method for heterogeneous multi-cluster systems according to claim 3, characterized in that, The function to obtain job information is used to obtain job information from the heterogeneous cluster, and the function to obtain job resource specifications is used to obtain resource specification information from the heterogeneous cluster.

5. The data processing method for heterogeneous multi-cluster systems according to claim 4, characterized in that, The Pod creation function converts a single-cluster Pod created by Kubernetes into a job that creates a heterogeneous cluster by calling the job creation function of the unified interface of the adapter. The Pod deletion function converts a single-cluster Pod to be deleted by Kubernetes into a job that deletes the corresponding heterogeneous cluster by calling the job stop function of the unified interface of the adapter.

6. The data processing method for heterogeneous multi-cluster systems according to claim 5, characterized in that, The function to obtain Pod information converts the Pod information obtained from the Kubernetes application programming interface server into the function to obtain the corresponding heterogeneous cluster job information by calling the job information obtain function of the unified interface of the adapter.

7. A heterogeneous multi-cluster data processing apparatus, used to implement the heterogeneous multi-cluster data processing method as described in any one of claims 1-6, characterized in that, include: The virtual node module is used to add new Kubernetes virtual nodes to the Kubernetes data processing system, and to associate each of the Kubernetes virtual nodes with each heterogeneous cluster in the heterogeneous multi-cluster system. Create an adapter module to create a separate adapter for each heterogeneous cluster based on the Kubernetes virtual node; A software creation module is used to deploy a node agent program corresponding to the adapter in the Kubernetes data processing system, wherein the adapter can communicate with the Kubernetes application programming interface server through the node agent program; The operation conversion module is used to convert the operation of the application programming interface server on kubelet into job operation on the heterogeneous cluster based on the node agent program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the heterogeneous multi-cluster data processing method as described in any one of claims 1-6.

9. A terminal device, characterized in that, include: Processor, memory, and communication bus; The memory stores a computer-readable program that can be executed by the processor; The communication bus enables communication between the processor and the memory; When the processor executes the computer-readable program, it implements the steps of the heterogeneous multi-cluster data processing method as described in any one of claims 1-6.

Images