Offline task processing method, device, system and equipment and storage medium

Abstract

The invention provides an off-line task processing method, device, system and equipment and a storage medium, which are used for solving the problem that the processing limitation is relatively large when an off-line task is processed. The method at least comprises the following steps: acquiring a mapping relationship between a plurality of first scheduling nodes and each second scheduling node; receiving a task processing request; the task processing request carries resource configuration data of a target offline task; based on the mapping relationship, selecting idle resources in node resources of corresponding second scheduling nodes and at least one target scheduling node matched with the resource configuration data from the plurality of first scheduling nodes; and adopting the at least one target scheduling node to respectively schedule idle resources of the corresponding second scheduling node to process the target offline task, and obtaining a task processing result. The idle resources in the plurality of second resource scheduling clusters are integrated together, so that the processing limitation when the offline task is processed is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an offline task processing method, device, system, equipment and storage medium. Background Art

[0002] With the continuous development of technology, more and more client or server devices can deploy resource scheduling systems (such as container orchestration (Kubernetes) systems) to form resource scheduling clusters, so that application programs can be automatically deployed, expanded, and managed based on resource scheduling clusters.

[0003] Generally, in the process of automatically deploying, expanding and managing applications, in response to a submission operation for an online task, the resource scheduling cluster can process the online task in a timely manner. Among them, the online task is a task with high processing timeliness requirements, for example, the online task may include an application task of a Web application or a service task of a database service.

[0004] Since processing online tasks generally does not occupy all resources of the resource scheduling cluster, the resource scheduling cluster has low resource utilization. In this case, you can usually use the idle resources in the resource scheduling cluster to process offline tasks with lower processing timeliness requirements to improve resource utilization. Offline tasks can be batch processing tasks, data analysis tasks, model training tasks, etc.

[0005] In the related art, the method for processing offline tasks is usually to select a resource scheduling cluster with a resource amount that is suitable for idle resources from a plurality of resource scheduling clusters used to process online tasks, such as selecting a resource scheduling cluster with a larger amount of idle resources; using this resource scheduling cluster to receive resource configuration data of the offline task, and processing the offline task based on the resource configuration data.

[0006] However, since the idle resources in each resource scheduling cluster have small specifications and are distributed on the resource scheduling nodes contained in the resource scheduling cluster, the idle resources are fragmented. Therefore, the method of selecting a resource scheduling cluster to process an offline task each time can neither guarantee the processing efficiency of the offline task nor effectively improve the resource utilization of the resource scheduling cluster.

[0007] It can be seen that the related technologies have great limitations when processing offline tasks. Summary of the invention

[0008] Embodiments of the present application provide an offline task processing method, apparatus, system, device, and storage medium for solving the problem of large processing limitations when processing offline tasks.

[0009] In a first aspect, an offline task processing method is provided, which is applied to a first resource scheduling cluster including a plurality of first scheduling nodes, comprising:

[0010] Acquire a mapping relationship between the plurality of first scheduling nodes and each second scheduling node in a plurality of second resource scheduling clusters;

[0011] Receive a task processing request; the task processing request carries resource configuration data of a target offline task;

[0012] Based on the mapping relationship, selecting, from the plurality of first scheduling nodes, idle resources in the node resources of the corresponding second scheduling node and at least one target scheduling node matching the resource configuration data;

[0013] The at least one target scheduling node is used to schedule the idle resources of the corresponding second scheduling node to process the target offline task and obtain the task processing result.

[0014] In a second aspect, an offline task processing device is provided, which is applied to a first resource scheduling cluster including a plurality of first scheduling nodes, including:

[0015] An acquisition module: used to acquire a mapping relationship between the plurality of first scheduling nodes and each second scheduling node in a plurality of second resource scheduling clusters;

[0016] Processing module: used to receive a task processing request; the task processing request carries resource configuration data of the target offline task;

[0017] The processing module is further used to: based on the mapping relationship, select, from the plurality of first scheduling nodes, idle resources in the node resources of the corresponding second scheduling node and at least one target scheduling node that matches the resource configuration data;

[0018] The processing module is also used to: use the at least one target scheduling node to schedule the idle resources of the corresponding second scheduling nodes to process the target offline task and obtain the task processing result.

[0019] Optionally, the processing module is specifically used for:

[0020] Based on the mapping relationship, the scheduling resource amount corresponding to each of the multiple first scheduling nodes is obtained; wherein the scheduling resource amount corresponding to the first scheduling node includes: the idle resource amount of the idle resources in the node resources of the second scheduling node corresponding to the first scheduling node;

[0021] Based on the resource configuration data, create configuration sub-data of each of the multiple offline sub-tasks; wherein the configuration sub-data is used to: configure the running environment of the corresponding offline sub-task; each of the offline sub-tasks is split from the target offline task;

[0022] For the multiple offline subtasks, respectively perform: based on the obtained scheduling resource amounts, select a first scheduling node whose scheduling resource amount matches the configuration sub-data of an offline subtask from the multiple first scheduling nodes as a target scheduling node.

[0023] Optionally, the processing module is specifically used for:

[0024] For the multiple first scheduling nodes, perform the following operations respectively:

[0025] Based on the mapping relationship, receiving an amount of idle resources reported by a second scheduling node corresponding to a first scheduling node;

[0026] When it is determined that the idle resource amount is received, the idle resource amount is stored in the node attribute of the one first scheduling node, and the scheduling resource amount corresponding to the one first scheduling node is obtained.

[0027] Optionally, when the first resource scheduling cluster includes a management node, the processing module is specifically used to:

[0028] The management node is used to respectively perform the following operations on the multiple first scheduling nodes:

[0029] Based on the mapping relationship, read the idle resource amount from the node attribute of the second scheduling node corresponding to the first scheduling node;

[0030] The obtained idle resource amount is written into the node attribute of the one first scheduling node to obtain the scheduling resource amount corresponding to the one first scheduling node.

[0031] Optionally, when the resource configuration data is converted into configuration sub-data of respective offline sub-tasks, each of the target scheduling nodes is used to process an offline sub-task, and the offline sub-task is split from the target offline task;

[0032] Then the processing module is specifically used for:

[0033] For the at least one target scheduling node, perform the following operations respectively:

[0034] According to the resource scheduling strategy of the second scheduling node corresponding to the target scheduling node, the resource scheduling field in the configuration sub-data of the offline sub-task processed by the target scheduling node is updated to obtain configuration update data;

[0035] Sending the configuration update data to a second scheduling node corresponding to the one target scheduling node, so that the second scheduling node processes the corresponding offline subtask based on the received configuration update data, and generates and returns a subtask result;

[0036] Based on the received subtask results, a task processing result of the target offline task is generated.

[0037] Optionally, when the first resource scheduling cluster includes a management node, the processing module is specifically configured to perform any of the following operation modes:

[0038] Using the management node, adding the resource scheduling field and its field value in the configuration sub-data of an offline sub-task processed by a target scheduling node to the environment variable of the configuration sub-data, and deleting the resource scheduling field and its field value in the configuration sub-data to obtain configuration update data;

[0039] The management node is used to update the field value of the resource scheduling field in the configuration sub-data of an offline sub-task processed by a target scheduling node to a preset value, thereby obtaining configuration update data.

[0040] Optionally, in the mapping relationship, there is a one-to-one correspondence between the multiple first scheduling nodes and each second scheduling node.

[0041] In a third aspect, an offline task processing system is provided, including a first resource scheduling cluster and multiple second resource scheduling clusters; a mapping relationship exists between multiple first scheduling nodes included in the first resource scheduling cluster and each second scheduling node in the multiple second resource scheduling clusters; wherein:

[0042] The first resource scheduling cluster receives a task processing request; wherein the task processing request carries resource configuration data of a target offline task;

[0043] The first resource scheduling cluster selects, from the plurality of first scheduling nodes, idle resources in the node resources of the corresponding second scheduling node and at least one target scheduling node that matches the resource configuration data based on the mapping relationship;

[0044] The first resource scheduling cluster schedules idle resources of the second scheduling nodes corresponding to each of the at least one target scheduling node in the plurality of second resource scheduling clusters, processes the target offline task, and obtains a task processing result.

[0045] Optionally, the system further includes a management node, and the management node performs the following operations respectively for the multiple first scheduling nodes:

[0046] Based on the mapping relationship, read the idle resource amount from the node attribute of a second scheduling node corresponding to a first scheduling node in the plurality of second resource scheduling clusters;

[0047] The obtained idle resource amount is written into the node attribute of the one first scheduling node, so that the first resource scheduling cluster obtains the scheduling resource amount corresponding to the one first scheduling node.

[0048] Optionally, the system further comprises a management node, and when the resource configuration data is converted into configuration sub-data of respective multiple offline sub-tasks, each of the target scheduling nodes is used to process an offline sub-task, and the offline sub-task is split from the target offline task;

[0049] Then, for the at least one target scheduling node, the management node respectively performs: adding the resource scheduling field and its field value in the configuration sub-data of the offline sub-task processed by the target scheduling node to the environment variable of the configuration sub-data, deleting the resource scheduling field and its field value in the configuration sub-data, obtaining configuration update data, and sending the configuration update data to the second scheduling node corresponding to the target scheduling node;

[0050] The second scheduling nodes corresponding to the at least one target scheduling node respectively perform: obtaining resource scheduling fields and field values ​​thereof from the environment variables of the received configuration update data, and updating the resource scheduling configuration of the processing process assigned to the configuration update data based on the obtained resource scheduling fields and field values ​​thereof, and using the processing process to process the corresponding offline subtask based on the configuration update data, and generating and returning the subtask result;

[0051] The management node generates a task processing result of the target offline task based on the received subtask results.

[0052] According to a fourth aspect, a computer program product is provided, comprising a computer program, which implements the method described in the first aspect when executed by a processor.

[0053] A fifth aspect provides a computer device, including:

[0054] A memory for storing program instructions;

[0055] A processor, configured to call the program instructions stored in the memory, and execute the method according to the first aspect according to the obtained program instructions, including:

[0056] The processor obtains a mapping relationship between a plurality of first scheduling nodes included in the first resource scheduling cluster and each second scheduling node in a plurality of second resource scheduling clusters, and stores the mapping relationship in the memory.

[0057] The processor receives a task processing request, which carries resource configuration data of a target offline task. The processor obtains a mapping relationship from a memory, and selects, from a plurality of first scheduling nodes, idle resources in the node resources of a corresponding second scheduling node, and at least one target scheduling node that matches the resource configuration data based on the mapping relationship. The processor uses at least one target scheduling node to schedule the idle resources of the corresponding second scheduling node to process the target offline task, and obtains a task processing result.

[0058] In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method described in the first aspect.

[0059] In an embodiment of the present application, the first resource scheduling cluster spans the physical boundaries between multiple second resource scheduling clusters, and the idle resources of each second scheduling node in the multiple second resource scheduling clusters are integrated together, so that the first resource scheduling cluster has a larger resource quantity specification of idle resources compared to a single second resource scheduling cluster, increases the optional nodes when processing offline tasks, and reduces the processing limitations when processing offline tasks.

[0060] Furthermore, by increasing the number of optional nodes when processing offline tasks, at least one target scheduling node that is more suitable for the target offline task can be selected to process the target offline task. There is no need to select a second scheduling node for processing the target offline task from a second resource scheduling cluster where idle resources are fragmented. While ensuring the processing efficiency of offline tasks, the resource utilization of the resource scheduling cluster is effectively improved, thereby achieving the purpose of reducing the processing limitations when processing offline tasks.

[0061] Furthermore, the first resource scheduling cluster separates the process of receiving the task processing request from the process of processing the target offline task, thereby avoiding the problem that when the second resource scheduling cluster receives the task processing request, it directly calls the node resources to process the target offline task, which results in occupying the node resources of the second resource scheduling cluster to process online tasks, affecting the processing efficiency and processing success rate of online tasks in the second resource scheduling cluster, and further reducing the processing limitations when processing offline tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1A Schematic diagram 1 of an application field of the offline task processing method provided in an embodiment of the present application;

[0063] Figure 1B Schematic diagram 2 of an application field of the offline task processing method provided in an embodiment of the present application;

[0064] Figure 1C An example of an application field of the offline task processing method provided in the embodiment of the present application is shown in FIG. Figure 3 ;

[0065] Figure 2A A flowchart of an offline task processing method in the related art;

[0066] Figure 2B An application scenario of the offline task processing method provided in the embodiment of the present application;

[0067] Figure 3 A schematic diagram of a flow chart of an offline task processing method provided in an embodiment of the present application;

[0068] Figure 4A A schematic diagram of a principle of an offline task processing method provided in an embodiment of the present application;

[0069] Figure 4B A second schematic diagram of a principle of an offline task processing method provided in an embodiment of the present application;

[0070] Figure 4C A schematic diagram of the principle of the offline task processing method provided in the embodiment of the present application Figure 3 ;

[0071] Figure 5A A fourth schematic diagram of a principle of an offline task processing method provided in an embodiment of the present application;

[0072] Figure 5B A fifth schematic diagram of a principle of an offline task processing method provided in an embodiment of the present application;

[0073] Fig. 6A A sixth schematic diagram of a principle of an offline task processing method provided in an embodiment of the present application;

[0074] Figure 6B A schematic diagram of a principle of an offline task processing method provided in an embodiment of the present application is shown in FIG7;

[0075] Fig. 7A A schematic diagram of the principle of the offline task processing method provided in the embodiment of the present application Figure 8 ;

[0076] Figure 7B A schematic diagram of the principle of an offline task processing system provided in an embodiment of the present application;

[0077] Figure 8 A structural schematic diagram 1 of an offline task processing device provided in an embodiment of the present application;

[0078] Fig. 9A second structural diagram of an offline task processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0080] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0081] (1) Online tasks:

[0082] Online tasks are tasks that have strict requirements on response time and availability. They are usually services that can directly interact with the task submission platform, need to respond quickly to platform requests, and need to maintain high availability and stability. Online tasks can include but are not limited to Web applications, API services, database services, etc.

[0083] (2) Offline tasks:

[0084] Offline tasks are tasks that do not require high response time and latency and can tolerate a certain amount of latency, including but not limited to batch processing jobs, data analysis, machine learning model training, etc. Unlike online services, offline tasks focus more on completing the task itself rather than responding to platform requests in real time.

[0085] The embodiments of the present application relate to artificial intelligence (AI) technology and cloud computing. Artificial intelligence technology and cloud computing can be applied to multiple fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, smart transportation, smart maps, assisted driving, vehicle terminals, aircraft, digital twins, virtual humans, robots, artificial intelligence generated content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology and cloud computing will be applied in more fields and play an increasingly important role.

[0086] It should be noted that in the embodiments of the present application, operations involving obtaining data such as task processing requests are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0087] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0088] The following is a brief introduction to the application fields of the offline task processing method provided in the embodiment of the present application.

[0089] With the continuous development of technology, more and more client or server devices can deploy resource scheduling systems (such as container orchestration (Kubernetes) systems) to form resource scheduling clusters, so that application programs can be automatically deployed, expanded, and managed based on resource scheduling clusters.

[0090] Please refer to Figure 1A , is a schematic diagram of the architecture of a Kubernetes cluster. The Kubernetes cluster consists of a master node and multiple worker nodes. The master node is the control center of the Kubernetes cluster, which can receive task requests from clients, create at least one workload (Pod) based on the resource requirements indicated by the received task requests, and allocate the Pod to the appropriate worker node to run based on the resource scheduling strategy of the Kubernetes cluster.

[0091] Please refer to Figure 1B , is a schematic diagram of the architecture of the master node in the Kubernetes cluster. The master node includes the Application Programming Interface (API) service component (API Server), the scheduling component (Scheduler), the main control component (Controllers) and the storage component (etcd).

[0092] The API service component provides interface services for the Kubernetes cluster and can receive requests from clients or other internal components. Requests from other internal components include creating Pods, updating Pods, and deleting Pods for the Kubernetes cluster. The API service component can also verify and authorize received requests. After verification and authorization, the obtained data can be written to the storage component.

[0093] The storage component can be in the form of distributed key-value storage to store relevant data of the Kubernetes cluster. The main control component can include multiple controllers such as ReplicaSet controller, Deployment controller, Node controller, etc., which are used to handle automated tasks such as replica management or fault recovery.

[0094] The scheduling component is used to create at least one workload (Pod) based on the resource requirements indicated by the received task request, and to allocate the Pod to a suitable worker node based on the resource scheduling policy of the Kubernetes cluster. Resource requirements can be processor requirements, memory requirements, etc., without specific restrictions. Resource scheduling policies can include affinity policies, etc., without specific restrictions.

[0095] Please refer to Figure 1C , which is an architectural diagram of a worker node in a Kubernetes cluster. A worker node contains a worker control component (Kubelet), a container runtime component (Container Runtime), and a network proxy component (Kube-proxy). The worker control component is used to manage and maintain the Pods on the worker nodes. The container runtime component is used to start and manage containers and provide a containerized operating environment. The network proxy component is used to configure network rules and route traffic load balancing to multiple Pods.

[0096] Generally, in the process of automatically deploying, expanding and managing applications, in response to a submission operation for an online task, the resource scheduling cluster can process the online task in a timely manner. Among them, the online task is a task with high processing timeliness requirements, for example, the online task may include an application task of a Web application or a service task of a database service.

[0097] Since processing online tasks generally does not occupy all resources of the resource scheduling cluster, the resource scheduling cluster has low resource utilization. In this case, you can usually use the idle resources in the resource scheduling cluster to process offline tasks with lower processing timeliness requirements to improve resource utilization. Offline tasks can be batch processing tasks, data analysis tasks, model training tasks, etc.

[0098] In the related art, the method for processing offline tasks is usually to select a resource scheduling cluster whose resource amount of idle resources matches that of the many resource scheduling clusters used to process online tasks, such as selecting a resource scheduling cluster with a larger amount of idle resources; using this resource scheduling cluster to receive the resource configuration data of the offline task, and processing the offline task based on the resource configuration data.

[0099] Please refer to Figure 2AIn response to the presentation operation for multiple Kubernetes clusters, multiple Kubernetes clusters are presented on the offline task submission platform; in response to the idle resource statistics operation triggered for multiple Kubernetes clusters, the total amount of idle resources of each of the multiple Kubernetes clusters is presented on the offline task submission platform; in response to the selection operation triggered for one of the multiple Kubernetes clusters, a task submission entrance is presented on the offline task submission platform; in response to the task upload operation triggered for the task submission entrance, a task processing request is sent to the selected Kubernetes cluster, and the Kubernetes cluster receives the task processing request; the task processing request carries the resource configuration data of the offline task, which is used to instruct the processing of the offline task; in response to the resource configuration data carried by the task processing request, the Kubernetes cluster creates multiple workload Pods, and schedules the working nodes in the Kubernetes cluster to run the multiple Pods to obtain the task execution result.

[0100] However, since the idle resources in each resource scheduling cluster have small specifications and are distributed on the resource scheduling nodes contained in the resource scheduling cluster, the idle resources are fragmented. Therefore, the method of selecting a resource scheduling cluster to process an offline task each time can neither guarantee the processing efficiency of the offline task nor effectively improve the resource utilization of the resource scheduling cluster.

[0101] It can be seen that the related technologies have great limitations when processing offline tasks.

[0102] In order to solve the problem of large processing limitations when processing offline tasks, the present application proposes an offline task processing method. The method is applied to a first resource scheduling cluster comprising multiple first scheduling nodes. In the method, a mapping relationship between multiple first scheduling nodes and each second scheduling node in multiple second resource scheduling clusters is obtained. A task processing request is received, and the task processing request carries the resource configuration data of the target offline task. Based on the mapping relationship, from the multiple first scheduling nodes, idle resources in the node resources of the corresponding second scheduling node and at least one target scheduling node that matches the resource configuration data are selected. At least one target scheduling node is used to schedule the idle resources of the corresponding second scheduling node to process the target offline task, and obtain the task processing result.

[0103] In an embodiment of the present application, the first resource scheduling cluster spans the physical boundaries between multiple second resource scheduling clusters, and the idle resources of each second scheduling node in the multiple second resource scheduling clusters are integrated together, so that the first resource scheduling cluster has a larger resource quantity specification of idle resources compared to a single second resource scheduling cluster, increases the optional nodes when processing offline tasks, and reduces the processing limitations when processing offline tasks.

[0104] Furthermore, by increasing the number of optional nodes when processing offline tasks, at least one target scheduling node that is more suitable for the target offline task can be selected to process the target offline task. There is no need to select a second scheduling node for processing the target offline task from a second resource scheduling cluster where idle resources are fragmented. While ensuring the processing efficiency of offline tasks, the resource utilization of the resource scheduling cluster is effectively improved, thereby achieving the purpose of reducing the processing limitations when processing offline tasks.

[0105] Furthermore, the first resource scheduling cluster separates the process of receiving the task processing request from the process of processing the target offline task, thereby avoiding the problem that when the second resource scheduling cluster receives the task processing request, it directly calls the node resources to process the target offline task, thereby occupying the node resources for processing online tasks, affecting the processing efficiency and success rate of online tasks, and further reducing the processing limitations when processing offline tasks.

[0106] The following describes the application scenarios of the offline task processing method provided by this application.

[0107] Please refer to Figure 2B , which is a schematic diagram of an application scenario of the offline task processing method provided in this application. The application scenario includes a first resource scheduling cluster 101, multiple second resource scheduling clusters 102 and a client 103; the first resource scheduling cluster 101 and the second resource scheduling cluster 102 can communicate with each other, and the first resource scheduling cluster 101 and the client 103 can communicate with each other; the communication method can be to communicate using wired communication technology, for example, by connecting a network cable or a serial port cable; or to communicate using wireless communication technology, for example, by communicating through Bluetooth or wireless fidelity (WIFI) and other technologies, without specific limitation.

[0108] The client 103 generally refers to a device that can submit offline tasks, etc., including, for example, a terminal device, a third-party application that the terminal device can access, or a web page that the terminal device can access. The first resource scheduling cluster 101 generally refers to a device cluster that does not need to process online tasks, etc., including, for example, a terminal device or a server. The second resource scheduling cluster 102 generally refers to a device cluster that needs to process online tasks, etc., including, for example, a terminal device or a server. The first resource scheduling cluster 101 and each second resource scheduling cluster 102 are different device clusters.

[0109] Terminal devices include but are not limited to mobile phones, computers, smart medical devices, smart home appliances, vehicle-mounted terminals or aircraft, etc. Servers include but are not limited to cloud servers, local servers or associated third-party servers, etc. The first resource scheduling cluster 101, the plurality of second resource scheduling clusters 102 and the client 103 can all adopt cloud computing to reduce the occupation of local computing resources; cloud storage can also be adopted to reduce the occupation of local storage resources.

[0110] In response to a submission operation triggered for a target offline task, a client 103 sends a task processing request to the first resource scheduling cluster 101. The first resource scheduling cluster 101 receives the task processing request sent by the client 103 and obtains resource configuration data of the target offline task carried in the task processing request.

[0111] The first resource scheduling cluster 101 schedules multiple second resource scheduling clusters 102 to process target offline tasks based on resource configuration data to obtain task processing results. The first resource scheduling cluster 101 returns the task processing results to the client 103, and the client 103 receives the task processing results returned by the first resource scheduling cluster 101 and presents the task processing results.

[0112] The following is based on Figure 2B , from the perspective of a first resource scheduling cluster including multiple first scheduling nodes, the offline task processing method provided by the embodiment of the present application is specifically introduced. Please refer to Figure 3 , which is a flow chart of the offline task processing method provided in an embodiment of the present application.

[0113] S301: Obtain a mapping relationship between a plurality of first scheduling nodes and each second scheduling node.

[0114] When multiple second resource scheduling clusters are running, the first resource scheduling cluster can obtain the mapping relationship between multiple first scheduling nodes contained in the first resource scheduling cluster and each second scheduling node from the multiple second resource scheduling clusters. Each second scheduling node comes from any second resource scheduling cluster in the multiple second resource scheduling clusters. Each second scheduling node can be used to schedule node resources to process online tasks or offline tasks, etc. In the embodiment of the present application, each second scheduling node is used to schedule node resources to process online tasks as an example for introduction.

[0115] The first resource scheduling cluster is configured to process offline tasks, and the second resource scheduling cluster is configured to process online tasks. Since processing offline tasks requires the deployment of additional components in the resource scheduling cluster, the additional components that the offline tasks depend on can be deployed only in the first resource scheduling cluster, and do not need to be deployed in each second resource scheduling cluster, so that the processing process of offline tasks is decoupled from each second resource scheduling cluster, avoiding the occupation of storage resources of each second resource scheduling cluster, and avoiding the misleading influence of the additional components that the offline tasks depend on on the processing of online tasks by each second resource scheduling cluster, thereby reducing the processing limitations when processing offline tasks.

[0116] Node resources may include CPU resources, storage resources, etc., without specific restrictions.

[0117] For example, additional components that offline tasks depend on include subtask controllers (such as Horizontal PodAutoscaler, HPA), which are used to automatically adjust the number of subtasks created. For example, for batch offline tasks, the number of subtasks can be dynamically adjusted according to the load; for model training offline tasks, the number of copies of each round of training subtasks can be adjusted according to CPU utilization or other hardware indicators.

[0118] For another example, the additional components that the offline task depends on include a node controller (such as Cluster Autoscaler), which is used to automatically adjust the number of first scheduling nodes in the first resource scheduling cluster to ensure that there are sufficient computing resources to meet the current workload requirements.

[0119] For another example, the additional components that offline tasks depend on include subtask creation controllers, such as a one-time subtask creation controller (such as a Job Controller) or a periodic subtask creation controller (such as a CronJob Controller). A one-time subtask creation controller can be used to create a subtask that is executed only once, and a periodic subtask creation controller can be used to create a subtask that is executed periodically.

[0120] For another example, the additional components that offline tasks rely on include tag and attribute configuration components (such as Taints and Tolerations), which are used to add tags or configure attributes for subtasks. By adding tags to subtasks, the first scheduling node can configure whether to process subtasks with such tags, thereby achieving a richer offline task processing process. By configuring attributes for subtasks, subtasks configured with a certain attribute can be assigned to the corresponding first scheduling node configured with a certain attribute for processing, thereby achieving a richer offline task processing process.

[0121] For another example, the additional components that offline tasks depend on include resource usage configuration components (such as Resource Quotas and Limit Ranges), which are used to constrain resource usage within the namespace to prevent an offline task from over-consuming resources and causing other tasks processed in the first resource scheduling cluster to experience processing exceptions and other unstable situations, thereby improving the stability of the offline task processing environment.

[0122] For another example, the additional components that offline tasks depend on include persistent storage components (such as Persistent Volumes or Persistent Volume Claims), which can provide persistent storage functions for the first resource scheduling cluster. For example, offline model training tasks require access to a large number of data sets, and the processing results must be saved and output after the offline tasks are completed. Therefore, the processing results of each data set and offline tasks can be stored in the persistent storage component.

[0123] Since the processing logic and dependent additional components of online tasks and offline tasks are different, the task processing architecture that processes offline tasks in the first resource scheduling cluster and processes online tasks in each second resource scheduling cluster can avoid interference with the processing process of online tasks when processing offline tasks, which can not only improve the processing accuracy of offline tasks, but also ensure the processing accuracy and timeliness of online tasks.

[0124] The first scheduling node is a node in the first resource scheduling cluster for processing tasks, such as the first scheduling node is a working node in the first Kubernetes cluster. The second scheduling node is a node in the corresponding second resource scheduling cluster for processing tasks, such as the second scheduling node is a working node in a second Kubernetes cluster.

[0125] There can be multiple mapping relationships, two of which are introduced as examples below.

[0126] Mapping relationship 1:

[0127] In the mapping relationship, there is a one-to-one correspondence between multiple first scheduling nodes and each second scheduling node. Therefore, each first scheduling node has a unique corresponding second scheduling node, and each second scheduling node also has a unique corresponding first scheduling node.

[0128] Please refer to Figure 4A , including a first resource scheduling cluster and 10 second resource scheduling clusters, the first resource scheduling cluster includes 100 first scheduling nodes, respectively denoted as first scheduling node_1, first scheduling node_2, ... and first scheduling node_100. The 10 second resource scheduling clusters each include 10 second scheduling nodes, respectively denoted as second scheduling node 1_1, second scheduling node 1_2, ..., second scheduling node 2_1, second scheduling node 2_2, ..., ... second scheduling node 10_1, second scheduling node 10_2, ... second scheduling node 10_10. Then, each first scheduling node and each second scheduling node can correspond one to one according to the mapping relationship in Table 1.

[0129] Table 1

[0130] First resource scheduling cluster Each second resource scheduling cluster First scheduling node_1 Second scheduling node 1_1 First scheduling node_2 Second scheduling node 1_2 …… …… First scheduling node_11 Second scheduling node 2_1 First scheduling node_12 Second scheduling node 2_2 …… …… First scheduling node_100 The second scheduling node 10_10

[0131] Through a one-to-one correspondence, each first scheduling node can schedule a uniquely corresponding second scheduling node to process a corresponding subtask. The scheduling process is convenient and fast, and the scheduling efficiency can be improved.

[0132] Mapping relationship 2:

[0133] In the mapping relationship, each first scheduling node may correspond to multiple second scheduling nodes.

[0134] For example, each first scheduling node corresponds to two second scheduling nodes, and the difference in the amount of idle resources between the two second scheduling nodes corresponding to each first scheduling node is greater than a preset difference. Therefore, each time the second scheduling node corresponding to the first scheduling node is scheduled to process a subtask, the second scheduling node with the largest amount of idle resources among the two corresponding second scheduling nodes can be scheduled. This not only achieves the purpose of quickly scheduling a second scheduling node to process the corresponding subtask, but also reduces the number of first scheduling nodes, avoids resource waste such as setting up a first scheduling node specifically for some second scheduling nodes with very small amounts of idle resources, and improves the rationality of resource allocation.

[0135] Please refer to Figure 4B, including a first resource scheduling cluster and 10 second resource scheduling clusters, the first resource scheduling cluster includes 50 first scheduling nodes, respectively denoted as first scheduling node_1, first scheduling node_2, ... and first scheduling node_50. The 10 second resource scheduling clusters each include 10 second scheduling nodes, respectively denoted as second scheduling node 1_1, second scheduling node 1_2, ..., second scheduling node 2_1, second scheduling node 2_2, ..., ... second scheduling node 10_1, second scheduling node 10_2, ... second scheduling node 10_10. Each second scheduling node counts the amount of idle resources, and the preset difference is 10. Then, the mapping relationship between each first scheduling node and each second scheduling node is shown in the figure.

[0136] For another example, each first scheduling node corresponds to multiple second scheduling nodes in a second resource scheduling cluster, and there is no cross-cluster situation for the multiple second scheduling nodes corresponding to each first scheduling node. Therefore, each time the second scheduling node corresponding to the first scheduling node is scheduled to process a subtask, the second scheduling nodes with the largest amount of idle resources in the corresponding two second scheduling nodes can be scheduled. This not only achieves the purpose of quickly scheduling a second scheduling node to process the corresponding subtask, but also reduces the number of first scheduling nodes, avoids resource waste such as setting up a first scheduling node specifically for some second scheduling nodes with very small amounts of idle resources, and improves the rationality of resource allocation.

[0137] Please refer to Figure 4C , including a first resource scheduling cluster and 10 second resource scheduling clusters, the first resource scheduling cluster includes 10 first scheduling nodes, respectively denoted as first scheduling node_1, first scheduling node_2, ... and first scheduling node_10. The 10 second resource scheduling clusters each include 10 second scheduling nodes, respectively denoted as second scheduling node 1_1, second scheduling node 1_2, ..., second scheduling node 2_1, second scheduling node 2_2, ..., ... second scheduling node 10_1, second scheduling node 10_2, ... second scheduling node 10_10. Then, the mapping relationship between each first scheduling node and each second scheduling node is shown in the figure.

[0138] By scheduling the second scheduling node through the first scheduling node, the offline task processing process and the resource scheduling process do not need to reuse the resource scheduling process for processing online tasks in the second resource scheduling cluster, thereby avoiding the negative impact on the processing efficiency of online tasks when reusing the resource scheduling process for processing online tasks, and also avoiding the situation where offline tasks cannot be processed due to the higher priority of online tasks, thereby improving the stability of task processing.

[0139] The mapping relationship may be stored in a storage component in the first resource scheduling cluster, such as etcd in the kuberneters cluster.

[0140] S302: Receive a task processing request.

[0141] After obtaining the mapping relationship, the first resource scheduling cluster can connect to the offline task submission platform and receive the task processing request, which carries the resource configuration data of the target offline task. Taking the kubernetes cluster as an example, the resource configuration data can be a data serialization format (Yet Another Markup Language, YAML) file. The resource configuration data can include the task file of the target offline task, which is used to indicate the specific content of the target offline task; the resource configuration data can also include the resource request of the target offline task, such as CPU resources, storage resources, etc. The resource configuration data can also include container images, commands and other information. By configuring resources according to the resource configuration data, the operating environment of the target offline task can be obtained.

[0142] By connecting the first resource scheduling cluster to the offline task submission platform, the offline task submission platform cannot perceive the resource attributes of the idle resources in the second resource scheduling cluster, and can access the idle resources at zero cost without additional adaptation.

[0143] S303 , based on the mapping relationship, selecting, from the plurality of first scheduling nodes, idle resources in the node resources of the corresponding second scheduling node and at least one target scheduling node that matches the resource configuration data.

[0144] After receiving the task processing request, the task processing request can be parsed to obtain the resource configuration data of the target offline task carried in the task processing request, so that, based on the mapping relationship, the idle resources in the node resources of the corresponding second scheduling node can be selected from multiple first scheduling nodes, and at least one target scheduling node that matches the resource configuration data. The selection strategy for selecting at least one target scheduling node can be configured in the first resource scheduling cluster, so that the corresponding second scheduling node can be directly scheduled, and there is no need to configure the node selection strategy for processing offline tasks in the second resource scheduling cluster, so as to avoid the conflict between the node selection strategy of online tasks and the node selection strategy of offline tasks in the second resource scheduling cluster, thereby improving the operation stability of each cluster.

[0145] For example, the second scheduling nodes are sorted from large to small according to the amount of idle resources in the node resources, and according to the resource request indicated by the resource configuration data and the sorting, the first scheduling nodes corresponding to at least one second scheduling node whose sum of resource amounts meets the resource request are selected as at least one target scheduling node. For example, when the amount of resources indicated by the resource request is 6 and the maximum amount of idle resources of each second scheduling node is 3, starting from the second scheduling node with the largest amount of idle resources, when two second scheduling nodes are selected according to the sorting, the sum of the amount of idle resources is 5, which does not meet the amount of resources 6 indicated by the resource request, and three second scheduling nodes are selected according to the sorting. At this time, the sum of the amount of idle resources is 6, which meets the amount of resources 6 indicated by the resource request, then the first scheduling nodes corresponding to the three second scheduling nodes are selected as at least one target scheduling node.

[0146] For another example, each second resource scheduling cluster is sorted from large to small according to the maximum value of the amount of idle resources in each second scheduling node contained therein. At the same time, for each second resource scheduling cluster, each second scheduling node contained therein is sorted from large to small according to the amount of idle resources in the node resources. Thus, according to the resource request indicated by the resource configuration data and the above-mentioned sorting, a second scheduling node can be selected from at least one second resource scheduling cluster respectively, so that the sum of the resource amounts of each second scheduling node selected from different second resource scheduling clusters meets the resource request, and the first scheduling node corresponding to each of the at least one selected second scheduling node is used as at least one target scheduling node.

[0147] There are many methods for selecting the target scheduling node, which are not listed here one by one. The following is a specific introduction to one of the methods using the above-mentioned mapping relationship 1 as an example.

[0148] Based on the mapping relationship, the amount of scheduling resources corresponding to each of the multiple first scheduling nodes is obtained. The amount of scheduling resources corresponding to the first scheduling node includes: the amount of idle resources in the node resources of the second scheduling node corresponding to the first scheduling node. Based on the mapping relationship, the second scheduling nodes corresponding to each of the multiple first scheduling nodes can be determined first, and then the amount of idle resources respectively counted by the second scheduling nodes corresponding to each of the multiple first scheduling nodes can be obtained as the amount of scheduling resources corresponding to each of the multiple first scheduling nodes.

[0149] After obtaining the scheduling resource amounts corresponding to each of the multiple first scheduling nodes, the configuration sub-data of each of the multiple offline sub-tasks can be created based on the resource configuration data. The configuration sub-data is used to configure the operating environment of the corresponding offline sub-task. Each offline sub-task is split from the target offline task. Taking the first resource scheduling cluster as a kubernetes cluster as an example, the configuration sub-data can be a workload (Pod). The Job controller in the Kubernetes cluster will detect the resource configuration data and determine how many Pods need to be created for the resource configuration data to execute the target offline task based on the control policy configured by the Job controller.

[0150] After obtaining the configuration sub-data of each of the multiple offline sub-tasks, the following are performed respectively for the multiple offline sub-tasks: based on the obtained scheduling resource amounts, a first scheduling node whose scheduling resource amount matches the configuration sub-data of an offline sub-task is selected from multiple first scheduling nodes as the target scheduling node, so that the scheduling resource amount of the target scheduling node can configure the operating environment for the offline sub-task it processes according to the configuration sub-data.

[0151] There are many methods for selecting the first scheduling node whose scheduling resource quantity matches the configuration sub-data of an offline subtask. For example, the obtained scheduling resource quantities are sorted in order from large to small to obtain a scheduling resource sequence, and the configuration sub-data are sorted in order from large to small according to the resource request indicated by the configuration sub-data to obtain a sub-data sequence. Therefore, according to the ranking of each configuration sub-data in the sub-data sequence, the first scheduling node corresponding to the scheduling resource quantity with the same ranking can be determined in the scheduling resource sequence as the target scheduling node corresponding to the configuration sub-data.

[0152] For another example, a first scheduling node is selected whose scheduling resource amount is greater than or equal to the resource request indicated by the configuration sub-data of the offline subtask; for another example, a first scheduling node whose scheduling resource amount is greater than the resource request indicated by the configuration sub-data of the offline subtask, and the difference between the scheduling resource amount and the resource request is within a preset difference range, so that the scheduling resource amount of the selected first scheduling node will not cause resource waste due to far exceeding the resource request indicated by the configuration sub-data of the offline subtask, thereby achieving reasonable allocation of resources, and at the same time, a certain resource margin is reserved for the processing of the offline subtask through the difference, thereby avoiding the occurrence of offline subtask processing jams due to resource shortages, and improving the processing stability of offline subtasks.

[0153] By converting resource configuration data into multiple configuration sub-data, multiple second scheduling nodes can be scheduled to collaboratively process target offline tasks, thereby improving task processing efficiency and at the same time improving the utilization of idle resources, breaking the physical boundaries between different resource scheduling clusters, and effectively reducing the processing limitations of offline tasks.

[0154] As an embodiment, when obtaining the amount of scheduling resources, it can be achieved by sending and receiving data between a first scheduling node and its corresponding second scheduling node. For multiple first scheduling nodes, the following operations are performed respectively: based on the mapping relationship, the amount of idle resources reported by the second scheduling node corresponding to a first scheduling node is received. When it is determined that the amount of idle resources is received, the amount of idle resources is stored in the node attribute of a first scheduling node, and the amount of scheduling resources corresponding to the first scheduling node is obtained.

[0155] The node attributes are additional information that describes the corresponding first scheduling node in detail, including, for example, node priority, node name, node label or node resources, etc., without specific limitation.

[0156] The second scheduling node can periodically re-count the amount of idle resources and update the amount of idle resources stored in the node attributes of the first scheduling node; it can also update the amount of idle resources stored in the node attributes of the first scheduling node when there is a resource change in the amount of idle resources that is greater than a certain threshold; it can also re-count the amount of idle resources upon receiving an update indication from the first scheduling node, and update the amount of idle resources stored in the node attributes of the first scheduling node, etc., so that when processing offline tasks, the first scheduling node can directly obtain the amount of scheduling resources from the node attributes without the need for real-time interaction with the second scheduling node, thereby improving the efficiency of processing offline tasks.

[0157] As an embodiment, when the first resource scheduling cluster also includes a management node, the management node can be used to obtain the scheduling resource amount. For multiple first scheduling nodes, the following operations are performed respectively:

[0158] Based on the mapping relationship, the idle resource amount is read from the node attribute of the second scheduling node corresponding to the first scheduling node. The obtained idle resource amount is written into the node attribute of the first scheduling node to obtain the scheduling resource amount corresponding to the first scheduling node.

[0159] When reading the idle resource amount from the node attributes of the second scheduling node, the management node can be used to determine whether the node attributes of the second scheduling node contain a key string representing the idle resource amount, such as "offline", "offline-cpu" or "offline-memmory".

[0160] When it is determined that the key string exists, the amount of idle resources is read from the location of the key string; when it is determined that the key string does not exist, it means that there are no idle resources in the second scheduling node, or the second scheduling node does not accept scheduling, etc., then the amount of scheduling resources in the node attributes of the first scheduling node corresponding to the second scheduling node can be set to 0 or a null value.

[0161] When reading the idle resource amount from the node attribute of the second scheduling node, the management node may also be used to determine whether the auxiliary information field in the node attribute contains a key string, such as "offline", "offline-cpu" or "offline-memmory", etc. If the key string is contained, the idle resource amount may be read from the location where the key string is located; if the key string is not contained, the scheduling resource amount in the node attribute of the first scheduling node corresponding to the second scheduling node may be set to 0 or a null value.

[0162] The auxiliary information field may be a node label field (labels) or a node annotation field (annotation) in the node attributes, etc. By reusing the fields in the node attributes to store the idle resource amount of the second scheduling node, the function of storing the idle resource amount of the second scheduling node can be added to the node attributes without modifying the data format of the node attributes, so as to realize the reporting of the idle resource amount.

[0163] When writing the idle resource amount into the node attribute of a first scheduling node, a field for storing the scheduling resource amount can be added to the node attribute, or the idle resource amount can be directly written into the node resource field of the node attribute of the first scheduling node. The node resource field can be a field in the node attribute that indicates the resource amount of schedulable resources, such as allocatable, etc. By directly writing into the node resource field, the idle resources in the second scheduling node can be used as the schedulable resources in the corresponding first scheduling node. The offline task submission platform can intuitively view the idle resources contained in multiple second resource scheduling clusters through the first resource scheduling cluster, providing a global perspective of idle resources.

[0164] Please refer to Figure 5A , including node attributes of a second scheduling node corresponding to a first scheduling node, such as "node name: node-1", "API version: V2" and a node label field (labels), and the node label field includes "node role: worker".

[0165] The second scheduling node may count the amount of idle resources in the node resources. For example, if the amount of idle resources is 5, "idle resource amount: 5" may be written into the node label field in the node attributes of the second scheduling node.

[0166] The node attributes of the first scheduling node include "node name: node-2", "API version: V2" and node resource field (allocatable), etc. Then, the management node can assign "5" to the scheduling resource amount and add "scheduling resource amount: 5" to allocatable. By adding the key-value pair of the scheduling resource amount to the node resource field, the idle resource amount of the second scheduling node is used as the scheduling resource amount of the first scheduling node, which helps to achieve the purpose of the first scheduling node scheduling the idle resources in the second scheduling node.

[0167] By setting up a management node to specifically obtain the scheduling resource amount, it is avoided that the master node or the working node cannot obtain the scheduling resource amount in time due to too many other tasks, thereby improving the timeliness of obtaining the scheduling resource amount.

[0168] As an embodiment, a scheduling switch field can be configured in the node attributes of the second scheduling node to indicate whether the corresponding second scheduling node agrees to be scheduled by the first scheduling node. If the scheduling switch field indicates that scheduling is prohibited, the corresponding corresponding relationship can be deleted in the mapping relationship; if the scheduling switch field indicates that scheduling is allowed, the statistical amount of idle resources can be written into the node attributes.

[0169] Please refer to Figure 5B , contains the local node properties of a second scheduling node, including API version (apiVersion): V1, data object: Node, metadata and node status. Metadata includes node labels (labels), and node labels include mix.k8s.io / offline-cpu: 40, mix.k8s.io / offline-memory: 100Gi, mix.k8s.io / unschedulable: true, where mix.k8s.io / unschedulable is the scheduling switch field, and mix.k8s.io / unschedulable: true means that scheduling is allowed. The node status includes node resources (allocatable), and node resources include resources for processing online tasks, such as cpu: 80, memory: 200Gi. The node resources of the second scheduling node indicate the amount of resources that the second scheduling node can allocate to the Pod for online tasks.

[0170] Then the management node can obtain the reported node attributes of the second scheduling node for the corresponding first scheduling node. The reported node attributes include the node attributes in the local node attributes of the second scheduling node that can be read by the management node, including API version (apiVersion): V1, data object: Node, specification field and node status. The specification field includes unschedulable: true. The node status includes node resources (allocatable), and the node resources include cpu: 40, memory: 100Gi. The node resources in the reported node attributes include the idle resources in the second scheduling node, that is, the amount of resources that the second scheduling node can allocate to the Pod for offline tasks.

[0171] S304: Use at least one target scheduling node to schedule the idle resources of the corresponding second scheduling nodes to process the target offline tasks and obtain task processing results.

[0172] After selecting at least one target scheduling node, the idle resources of the corresponding second scheduling nodes are respectively scheduled to process the target offline tasks to obtain task processing results.

[0173] The process of processing the target offline task can be carried out according to the processing logic configured by the first resource scheduling cluster. For example, the first resource scheduling cluster is configured to convert the resource configuration data into the configuration sub-data of each of the multiple offline sub-tasks, and then use multiple second scheduling nodes to process one offline sub-task respectively. Then, each configuration sub-data can be sent to the corresponding second scheduling node for synchronous processing or sequential processing. If the target offline task is a model training task, the five sample data used in each round of training can be sent to five second scheduling nodes for training respectively, and then the model training loss of each round of training can be integrated in one of the second scheduling nodes to realize the processing of the target offline task.

[0174] For another example, the first resource scheduling cluster is configured so that whenever the target offline task reaches a specified stage, it will be handed over to the next second scheduling node for further processing. If the target offline task is a model training task, then one second scheduling node can be used for training for 100 rounds, and then the next second scheduling node can be used for further training for 100 rounds.

[0175] The following is a detailed introduction to a process of processing a target offline task. Other processes of processing a target offline task are similar and will not be described in detail here.

[0176] When the resource configuration data is converted into the configuration sub-data of multiple offline sub-tasks, each target scheduling node is used to process an offline sub-task, and the offline sub-task is split from the target offline task. For details, please refer to the previous introduction. Then, for at least one target scheduling node, perform the following operations respectively:

[0177] According to the resource scheduling strategy of the second scheduling node corresponding to a target scheduling node, the resource scheduling field in the configuration sub-data of an offline sub-task processed by the target scheduling node is updated to obtain configuration update data. The configuration update data is sent to the second scheduling node corresponding to the target scheduling node, so that the second scheduling node runs the corresponding offline sub-task based on the received configuration update data, and generates and returns the sub-task result.

[0178] Thus, the first resource scheduling cluster can generate the task processing result of the target offline task based on the received subtask results. The specific process of generating the task processing result of the target offline task based on the subtask results can be set according to the scenario. For example, in the model training scenario, each subtask result can be a model parameter after multiple rounds of training. Then the subtask result corresponding to the model parameter after the last round of training can be used as the task processing result of the target offline task, etc. There is no specific limitation.

[0179] Taking the Kubernetes cluster as an example, the resource scheduling field may include a request field and a limit field.

[0180] Since the second scheduling node has been selected for the configuration sub-data, it is no longer necessary to execute the resource scheduling policy on the configuration sub-data in the second resource scheduling cluster. Therefore, the resource scheduling field in the configuration sub-data can be updated according to the resource scheduling policy of the second scheduling node, so that the processing process of the offline sub-task will not conflict with the processing process of the online task, avoiding negative impact on the processing of the online task, and there is no need to configure idle resources as extended resources and other additional complex operations, thereby reducing the limitations of offline task processing.

[0181] As an embodiment, when the first resource scheduling cluster includes a management node, the management node can also be used to update data to avoid occupying processing resources of other nodes in the first resource scheduling cluster and improve the timeliness of data update. There are many methods for using a management node to update data, and two of them are introduced as examples below.

[0182] Method 1:

[0183] The management node is used to add the resource scheduling field and its field value in the configuration sub-data of an offline sub-task processed by a target scheduling node to the environment variable of the configuration sub-data, and delete the resource scheduling field and its field value in the configuration sub-data to obtain the configuration update data.

[0184] Please refer to Fig. 6A, contains the configuration sub-data of an offline subtask, including "API version (apiVersion): V1", "data object (kind): Pod", specification field (spec) and resource limit field (limits). The specification field includes the resource request field (request): "cpu: 20" and "storage (memory): 50Gi", and the resource limit field includes "cpu: 30" and "storage (memory): 60Gi".

[0185] Among them, the resource request field and the resource limitation field can be used as resource scheduling fields. Therefore, "cpu: 20" and "storage (memory): 50Gi" in the resource request field, and "cpu: 30" and "storage (memory): 60Gi" in the resource limitation field can be added to the environment variables of the configuration sub-data, and "cpu: 20" and "storage (memory): 50Gi" can be deleted, as well as "cpu: 30" and "storage (memory): 60Gi" to obtain the configuration update data.

[0186] Method 2:

[0187] The management node is used to update the field value of the resource scheduling field in the configuration sub-data of an offline sub-task processed by a target scheduling node to a preset value, so as to obtain configuration update data.

[0188] For example, the preset value may be 0 or any other value.

[0189] Please refer to Figure 6B , contains the configuration sub-data of an offline subtask, including "API version (apiVersion): V1", "data object (kind): Pod", specification field (spec) and resource limit field (limits). The specification field includes the resource request field (request): "cpu: 20" and "storage (memory): 50Gi", and the resource limit field includes "cpu: 30" and "storage (memory): 60Gi".

[0190] Among them, the resource request field can be used as a resource scheduling field. Therefore, the "cpu: 20" and "storage (memory): 50Gi" in the resource request field can be assigned to 0, and "cpu: 0" and "storage (memory): 0" are obtained to obtain the configuration update data; the "cpu: 20" and "storage (memory): 50Gi" in the resource request field can also be added to the environment variables of the configuration sub-data, and the "cpu: 20" and "storage (memory): 50Gi" in the resource request field can be assigned to 0, and "cpu: 0" and "storage (memory): 0" are obtained to obtain the configuration update data.

[0191] As an embodiment, when the second scheduling node receives the configuration update data, the resource scheduling configuration of the processing process assigned to the configuration update data can be updated based on the environment variables, and the processing process can be used to process the corresponding offline subtask based on the configuration update data. For example, based on the "cpu: 20" and "storage (memory): 50Gi" in the environment variables, according to the resource restriction policy of kubernetes, the resource scheduling configuration is performed in the cgroup, and the resource restriction is performed on the processing process assigned to the configuration update data, so that the processing process can perform resource scheduling according to the corresponding configuration subdata, and will not affect the processing process of the online task, thereby reducing the processing limitations of the offline task.

[0192] The following takes the first resource scheduling cluster as the first Kubernetes cluster and the multiple second resource scheduling clusters as the multiple second Kubernetes clusters as an example to introduce the offline task processing method provided in the embodiment of the present application. Please refer to Fig. 7A .

[0193] In response to the offline task submission operation triggered by the offline task submission platform, a task processing request for the target offline task is generated, the offline task submission platform sends the task processing request for the target offline task to the first Kubernetes cluster, and the first Kubernetes cluster receives the task processing request sent by the offline task submission platform.

[0194] The first kubernetes cluster is deployed with a variety of additional components for processing offline task dependencies, such as operator components and service components. The operator component is used to encapsulate, deploy and manage kubernetes applications; the service component is used to provide network services and load balancing, etc. The various additional components for processing offline task dependencies are deployed only in the first kubernetes cluster, without having to be deployed in multiple second kubernetes clusters, thus avoiding operation conflicts for the second kubernetes cluster and affecting the stability of the second kubernetes cluster.

[0195] The first kubernetes cluster includes multiple first scheduling nodes, each scheduling node corresponds to a second scheduling node in multiple second kubernetes clusters, and each first scheduling node is mapped to each second scheduling node in a one-to-one correspondence. The first kubernetes cluster also includes a first master node and a management node.

[0196] The node resources of the second scheduling node include online task resources, reserved resources (Buffer) and idle resources for processing online tasks. After counting the amount of idle resources, the second scheduling node can add the amount of idle resources to the node annotation (annotation) of the node attributes. The management node reads the amount of idle resources from the node annotation of the node attributes of the second scheduling node and adds the read amount of idle resources to the node resources (allocatable) of the node attributes of the corresponding first scheduling node. The figure only shows one example.

[0197] Thus, the first master node in the first kubernetes cluster can create multiple Pods based on the resource configuration data, and select a corresponding target scheduling node from multiple first scheduling nodes for each Pod based on the newly added scheduling resource amount in the node attribute. The figure only shows one as an example. The selection process can be referred to the previous introduction and will not be repeated here.

[0198] The management node in the first Kubernetes cluster updates the data of each Pod and synchronously sends the updated Pods to the corresponding second scheduling node. The node name of the corresponding second scheduling node can be carried when sending. The second scheduling node runs the received updated Pod and returns the subtask processing result. The first Kubernetes cluster generates the task processing result of the target offline task based on the received subtask processing. The figure shows only one as an example.

[0199] Based on the same inventive concept, an embodiment of the present application provides an offline task processing system, including a first resource scheduling cluster and multiple second resource scheduling clusters; the first resource scheduling cluster includes multiple first scheduling nodes, and there is a mapping relationship between the multiple first scheduling nodes and each second scheduling node from the multiple second resource scheduling clusters; wherein:

[0200] The first resource scheduling cluster receives a task processing request, which carries resource configuration data of a target offline task. Based on a mapping relationship, the first resource scheduling cluster selects idle resources in the node resources of a corresponding second scheduling node from a plurality of first scheduling nodes, and at least one target scheduling node that matches the resource configuration data. The first resource scheduling cluster schedules idle resources of a second scheduling node corresponding to at least one target scheduling node in a plurality of second resource scheduling clusters, processes the target offline task, and obtains a task processing result.

[0201] In the embodiment of the present application, the offline task processing system forms a two-layer cluster architecture. It is not necessary to manage the idle resources in each second resource scheduling cluster separately. Instead, the idle resources of multiple second resource scheduling clusters are aggregated into a first resource scheduling cluster for unified management, which reduces the complexity of offline task processing. Compared with the method in which the offline task submission platform directly submits offline tasks to the second resource scheduling cluster, in the embodiment of the present application, offline tasks do not need to be limited to processing in one second resource scheduling cluster, but can comprehensively consider the idle resources in multiple second resource scheduling clusters, effectively improving resource utilization.

[0202] According to the previous introduction, the mapping relationship can be a one-to-one relationship or a one-to-many relationship. By managing the first resource scheduling cluster, the purpose of managing the idle resources in multiple second resource scheduling clusters can be achieved. The first resource scheduling cluster provides a global view of idle resources for the offline task submission platform, and can select the optimal second scheduling node from multiple second resource scheduling clusters to process offline tasks, breaking the physical boundaries between clusters and effectively improving resource utilization.

[0203] As an embodiment, the offline task processing system may further include a management node, which obtains the amount of each scheduling resource in multiple second resource scheduling clusters for the first resource scheduling cluster through the management node, thereby achieving clear division of labor, eliminating the need to reuse the nodes in the first resource scheduling cluster, and improving the operating stability of the first resource scheduling cluster.

[0204] The management node performs the following operations on multiple first scheduling nodes:

[0205] Based on the mapping relationship, the idle resource amount is read from the node attribute of the second scheduling node corresponding to a first scheduling node in the multiple second resource scheduling clusters, and the specific details can be referred to in the previous description. The obtained idle resource amount is written into the node attribute of a first scheduling node, so that the first resource scheduling cluster obtains the scheduling resource amount corresponding to a first scheduling node, and the specific details can be referred to in the previous description.

[0206] When the first resource scheduling cluster selects a target scheduling node, it can directly refer to the node attributes of each first scheduling node without obtaining the node attributes from each second scheduling node in real time, thereby improving the efficiency of selecting the target scheduling node.

[0207] As an embodiment, when the offline task processing system also includes a management node, if the resource configuration data of the target offline task is converted into the configuration sub-data of multiple offline sub-tasks, please refer to the previous introduction for details. Then the management node can update the configuration sub-data according to the resource scheduling strategy of the corresponding second scheduling node to avoid resource scheduling conflicts and the like.

[0208] The management node can execute, for at least one target scheduling node, the following respectively: add the resource scheduling field and its field value in the configuration sub-data of an offline sub-task processed by a target scheduling node to the environment variables of the configuration sub-data, delete the resource scheduling field and its field value in the configuration sub-data, obtain configuration update data, and send the configuration update data to a second scheduling node corresponding to a target scheduling node.

[0209] The second scheduling node corresponding to at least one target scheduling node respectively executes: obtaining the resource scheduling field and its field value from the environment variables of the received configuration update data, and updating the resource scheduling configuration of the processing process assigned to the configuration update data based on the obtained resource scheduling field and its field value, and using the processing process to process the corresponding offline subtask based on the configuration update data, and generating and returning the subtask result.

[0210] For example, when the second scheduling node processes an offline subtask based on the configuration update data, it reads the resource scheduling field and its field value stored in the environment variable of the configuration update data, and reconfigures cgroups according to the resource scheduling field and its field value to ensure that the processing of the offline subtask can comply with the resource restrictions indicated by the resource scheduling field and its field value.

[0211] By deleting the resource scheduling field and its field value in the configuration sub-data, the configuration update data can be placed at a lower priority in the second scheduling node, such as the QoS attribute is changed from burstable to besteffort, which will not affect the processing of online tasks. By reconfiguring cgroups according to the resource scheduling field and its field value, the process of using the processing process to process the corresponding offline subtask based on the configuration update data can still perform resource scheduling according to the resource scheduling field and its field value, avoiding the situation of occupying a large number of idle resources and improving the processing flexibility of offline tasks.

[0212] The resource scheduling field may include a resource request field and a resource restriction field, etc., and there is no specific limitation.

[0213] Thus, the management node can generate the task processing result of the target offline task based on the received subtask results.

[0214] For example, the first resource scheduling cluster is the first Kubernetes cluster, and the multiple second resource scheduling clusters are multiple second Kubernetes clusters. Figure 7B , which is a schematic diagram of the interaction principle between an offline task submission platform and an offline task processing system.

[0215] Each first scheduling node in the first kubernetes cluster has a one-to-one mapping relationship with a second scheduling node in the plurality of second kubernetes clusters. An agent component is deployed in each second scheduling node, and the agent component periodically counts the amount of idle resources in the node resources, and writes the counted amount of idle resources into the node attributes of the corresponding second scheduling node in the form of a key-value pair, such as into the node label field of the node attributes.

[0216] In the node attributes of each second scheduling node, a scheduling switch field may be added in the node label field to indicate whether the second scheduling node is allowed to be scheduled.

[0217] The management node periodically obtains the reportable node attributes from each second scheduling node, including the node label field and the scheduling switch field. If the scheduling switch field indicates that the second scheduling node is not allowed to be scheduled, then the corresponding one-to-one correspondence is deleted from the mapping relationship. If the scheduling switch field indicates that the second scheduling node is allowed to be scheduled, then the idle resource amount recorded in the node label field is read and written into the node attribute of the corresponding first scheduling node, such as writing into the node resource field of the node attribute, so that the first scheduling node obtains the scheduling resource amount.

[0218] After generating the task processing request, the offline task submission platform may send the task processing request to the first Kubernetes cluster in the offline task processing system, and the first Kubernetes cluster receives the task processing request sent by the offline task submission platform.

[0219] The first kubernetes cluster creates multiple Pods, that is, multiple configuration sub-data, based on the resource configuration data of the target offline task carried in the task processing request. Each Pod indicates the resource request and resource limit of the corresponding offline sub-task.

[0220] The first Kubernetes cluster selects a target scheduling node that matches the Pod from each first scheduling node based on the scheduling resource amount recorded in the node attribute of each first scheduling node, and obtains multiple target scheduling nodes.

[0221] One way to manage nodes is to set the field value in the resource request field in the Pod to 0 to obtain the updated Pod, which is the configuration update data. Another way is to add the key-value pairs contained in the resource request field and the resource restriction field in the Pod to the environment variables, and delete the key-value pairs contained in the resource request field and the resource restriction field to obtain the updated Pod, which is the configuration update data.

[0222] The management node sends the updated Pod to the corresponding second scheduling node. After obtaining the updated Pod, the agent component in the second scheduling node configures the key-value pairs contained in the resource request field and the resource restriction field in the environment variables respectively in the cgroup, and then processes the offline subtask based on the updated Pod; if the key-value pairs contained in the resource request field and the resource restriction field do not exist in the environment variables respectively, then the offline subtask is directly processed based on the updated Pod.

[0223] The management node receives the subtask results of the offline subtasks returned by each second scheduling node, and generates the task processing result of the target offline task. The management node presents the task processing result on the offline task submission platform through the first resource scheduling cluster, and completes the processing process of the target offline task.

[0224] In the embodiment of the present application, a global perspective is formed through the first resource scheduling cluster, and the idle resources distributed in multiple second resource scheduling clusters can be uniformly managed. By aggregating the idle resources of multiple second resource scheduling clusters into the same first resource scheduling cluster, the physical boundaries between clusters are broken, allowing instances of offline tasks to run across clusters. A first resource scheduling cluster aggregates the idle resources of multiple second resource scheduling clusters, thereby improving the utilization rate of idle resources.

[0225] Furthermore, the offline task submission platform cannot perceive the resource attributes of the first resource scheduling cluster, and can access the first resource scheduling cluster at zero cost without additional adaptation. The first resource scheduling cluster is also transparent to multiple second resource scheduling clusters, and the processing flow of offline tasks and online tasks is decoupled. There is no need to deploy the components that the offline task operation depends on in the second resource scheduling cluster, thereby avoiding resource competition and scheduling policy conflicts, and greatly reducing interference with the operation of the second resource scheduling cluster.

[0226] Furthermore, for example, in the context of the current dual carbon goals, computing power resources are in short supply. The embodiments of the present application can use the idle resources of multiple second resource scheduling clusters without doing additional adaptation work. While improving the utilization rate of computing power resources, it can tap more low-cost computing power resources for offline tasks, helping to improve efficiency.

[0227] Based on the same inventive concept, the present application embodiment provides an offline task processing device that can implement the functions corresponding to the aforementioned offline task processing method. Figure 8 , the device includes a processing module 802 and an acquisition module 801, wherein:

[0228] Acquisition module 801: used to acquire mapping relationships between multiple first scheduling nodes and each second scheduling node in multiple second resource scheduling clusters;

[0229] Processing module 802: used to receive a task processing request; the task processing request carries resource configuration data of the target offline task;

[0230] The processing module 802 is further used to: select, from the plurality of first scheduling nodes, idle resources in the node resources of the corresponding second scheduling node and at least one target scheduling node that matches the resource configuration data based on the mapping relationship;

[0231] The processing module 802 is further used to: use at least one target scheduling node to schedule the idle resources of the corresponding second scheduling nodes to process the target offline tasks, and obtain the task processing results.

[0232] In a possible embodiment, the processing module 802 is specifically configured to:

[0233] Based on the mapping relationship, the scheduling resource amount corresponding to each of the multiple first scheduling nodes is obtained; wherein the scheduling resource amount corresponding to the first scheduling node includes: the idle resource amount of the idle resources in the node resources of the second scheduling node corresponding to the first scheduling node;

[0234] Based on the resource configuration data, create configuration sub-data of each of the multiple offline sub-tasks; wherein the configuration sub-data is used to: configure the running environment of the corresponding offline sub-task; each offline sub-task is split from the target offline task;

[0235] For multiple offline subtasks, respectively execute: based on the obtained scheduling resource amounts, select a first scheduling node whose scheduling resource amount matches the configuration sub-data of an offline subtask from multiple first scheduling nodes as a target scheduling node.

[0236] In a possible embodiment, the processing module 802 is specifically configured to:

[0237] For multiple first scheduling nodes, perform the following operations respectively:

[0238] Based on the mapping relationship, receiving an idle resource amount reported by a second scheduling node corresponding to a first scheduling node;

[0239] When it is determined that the idle resource amount is received, the idle resource amount is stored in a node attribute of a first scheduling node, and a scheduling resource amount corresponding to the first scheduling node is obtained.

[0240] In a possible embodiment, when the first resource scheduling cluster includes a management node, the processing module 802 is specifically configured to:

[0241] Using a management node, for multiple first scheduling nodes, perform the following operations respectively:

[0242] Based on the mapping relationship, read the idle resource amount from the node attribute of the second scheduling node corresponding to the first scheduling node;

[0243] The obtained idle resource amount is written into a node attribute of a first scheduling node to obtain a scheduling resource amount corresponding to the first scheduling node.

[0244] In a possible embodiment, when the resource configuration data is converted into configuration sub-data of each of the multiple offline sub-tasks, each target scheduling node is used to process an offline sub-task, and the offline sub-task is split from the target offline task;

[0245] The processing module 802 is specifically used for:

[0246] For at least one target scheduling node, perform the following operations:

[0247] According to the resource scheduling strategy of the second scheduling node corresponding to the target scheduling node, data is updated on the resource scheduling field in the configuration sub-data of the offline sub-task processed by the target scheduling node to obtain configuration update data;

[0248] Sending configuration update data to a second scheduling node corresponding to a target scheduling node, so that the second scheduling node processes the corresponding offline subtask based on the received configuration update data, and generates and returns the subtask result;

[0249] Based on the received subtask results, a task processing result of the target offline task is generated.

[0250] In a possible embodiment, when the first resource scheduling cluster includes a management node, the processing module 802 is specifically configured to perform any of the following operation modes:

[0251] Using a management node, adding a resource scheduling field and its field value in the configuration sub-data of an offline sub-task processed by a target scheduling node to the environment variable of the configuration sub-data, and deleting the resource scheduling field and its field value in the configuration sub-data to obtain configuration update data;

[0252] The management node is used to update the field value of the resource scheduling field in the configuration sub-data of an offline sub-task processed by a target scheduling node to a preset value, so as to obtain configuration update data.

[0253] In a possible embodiment, in the mapping relationship, there is a one-to-one correspondence between the multiple first scheduling nodes and the respective second scheduling nodes.

[0254] Please refer to Fig. 9 , is a computer device 900 provided in an embodiment of the present application, and the computer device 900 can be, for example, Figure 2B The control node in the first resource scheduling cluster 101 in the example may also be a management node in the first resource scheduling cluster 101; for example, it may be Figure 7B The current version and the historical version of the data storage program and the application software corresponding to the data storage program can be installed on a computer device 900, which includes a processor 980 and a memory 920.

[0255] In some embodiments, the computer device 900 may include a display unit 940 , and the display unit 940 includes a display panel 941 for displaying an interactive operation interface of a user.

[0256] In a possible embodiment, the display panel 941 may be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

[0257] The processor 980 is used to read the computer program and then execute the method defined by the computer program. For example, the processor 980 reads the data storage program or file, so as to run the data storage program on the computer device 900 and display the corresponding interface on the display unit 940. The processor 980 may include one or more general-purpose processors and may also include one or more DSPs (Digital Signal Processors) to perform related operations to implement the technical solutions provided in the embodiments of the present application.

[0258] The memory 920 generally includes internal memory and external memory, and the internal memory may be a random access memory (RAM), a read-only memory (ROM), and a cache (CACHE), etc. The external memory may be a hard disk, an optical disk, a USB disk, a floppy disk, or a tape drive, etc. The memory 920 is used to store computer programs and other data, and the computer program includes applications corresponding to each client, etc. Other data may include data generated after the operating system or application is run, and the data includes system data (such as configuration parameters of the operating system) and user data. In the embodiment of the present application, the computer program is stored in the memory 920, and the processor 980 executes the computer program in the memory 920 to implement any of the methods discussed in the previous figure.

[0259] As an embodiment, the processor 980 may obtain a mapping relationship between a plurality of first scheduling nodes included in the first resource scheduling cluster and each second scheduling node in a plurality of second resource scheduling clusters, and store the mapping relationship in the memory 920 .

[0260] The processor 980 may receive a task processing request, which carries resource configuration data of a target offline task. The processor 980 obtains a mapping relationship from the memory 920, and selects, from a plurality of first scheduling nodes, idle resources in the node resources of the corresponding second scheduling node, and at least one target scheduling node that matches the resource configuration data based on the mapping relationship. The processor 980 uses at least one target scheduling node to schedule the idle resources of the corresponding second scheduling node to process the target offline task, and obtains a task processing result.

[0261] The display unit 940 is used to receive input digital information, character information or contact touch operation / contactless gesture, and generate signal input related to user settings and function control of the computer device 900. Specifically, in the embodiment of the present application, the display unit 940 may include a display panel 941. The display panel 941 is, for example, a touch screen, which can collect user touch operations on or near it (such as operations performed by the user using fingers, stylus, or any other suitable object or accessory on or on the display panel 941), and drive corresponding connection devices according to a pre-set program.

[0262] In a possible embodiment, the display panel 941 may include two parts: a touch detection device and a touch controller. The touch detection device detects the touch position of the player, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 980, and can receive and execute the command sent by the processor 980.

[0263] The display panel 941 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the display unit 940, in some embodiments, the computer device 900 may further include an input unit 930, which may include an image input device 931 and other input devices 932, wherein the other input devices may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0264] As an embodiment, when the processor 980 obtains the task processing result, it can send the task processing result to the display unit 940, and the display unit 940 can present the task processing result. The display unit 940 can also present an interface for submitting a task processing request, etc.

[0265] In addition to the above, the computer device 900 may also include a power supply 990 for supplying power to other modules, an audio circuit 960, a near field communication module 970, and an RF circuit 910. The computer device 900 may also include one or more sensors 950, such as an acceleration sensor, a light sensor, a pressure sensor, etc. The audio circuit 960 specifically includes a speaker 961 and a microphone 962, etc. For example, the computer device 900 can collect the user's voice through the microphone 962 to perform corresponding operations, etc.

[0266] As an embodiment, the number of processors 980 may be one or more, and the processor 980 and the memory 920 may be coupled or relatively independently configured.

[0267] As an example, Fig. 9 The processor 980 in the embodiment can be used to implement the following Figure 8 The functions of the acquisition module 801 and the processing module 802 in .

[0268] As an example, Fig. 9 The processor 980 in can be used to implement the corresponding functions of the server or terminal device discussed above.

[0269] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by a computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the computer program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0270] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product, for example, through a computer program product, which is stored in a storage medium, including a computer program for a computer device to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0271] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An offline task processing method, characterized in that: The method is applied to a first resource scheduling cluster including a plurality of first scheduling nodes, comprising: Acquire a mapping relationship between the plurality of first scheduling nodes and each second scheduling node in a plurality of second resource scheduling clusters; Receive a task processing request; the task processing request carries resource configuration data of a target offline task; Based on the mapping relationship, selecting, from the plurality of first scheduling nodes, idle resources in the node resources of the corresponding second scheduling node and at least one target scheduling node matching the resource configuration data; The at least one target scheduling node is used to schedule the idle resources of the corresponding second scheduling node to process the target offline task and obtain the task processing result.

2. The method according to claim 1, characterized in that The selecting, based on the mapping relationship, idle resources in the node resources of the corresponding second scheduling node from the plurality of first scheduling nodes, and at least one target scheduling node matching the resource configuration data comprises: Based on the mapping relationship, the scheduling resource amount corresponding to each of the multiple first scheduling nodes is obtained; wherein the scheduling resource amount corresponding to the first scheduling node includes: the idle resource amount of the idle resources in the node resources of the second scheduling node corresponding to the first scheduling node; Based on the resource configuration data, create configuration sub-data of each of the multiple offline sub-tasks; wherein the configuration sub-data is used to: configure the running environment of the corresponding offline sub-task; each of the offline sub-tasks is split from the target offline task; For the multiple offline subtasks, respectively perform: based on the obtained scheduling resource amounts, select a first scheduling node whose scheduling resource amount matches the configuration sub-data of an offline subtask from the multiple first scheduling nodes as a target scheduling node.

3. The method according to claim 2, characterized in that The acquiring, based on the mapping relationship, the scheduling resource amounts corresponding to each of the plurality of first scheduling nodes includes: For the multiple first scheduling nodes, perform the following operations respectively: Based on the mapping relationship, receiving an amount of idle resources reported by a second scheduling node corresponding to a first scheduling node; When it is determined that the idle resource amount is received, the idle resource amount is stored in the node attribute of the one first scheduling node, and the scheduling resource amount corresponding to the one first scheduling node is obtained.

4. The method according to claim 2, characterized in that: When the first resource scheduling cluster includes a management node, obtaining the scheduling resource amounts corresponding to each of the plurality of first scheduling nodes based on the mapping relationship includes: The management node is used to respectively perform the following operations on the multiple first scheduling nodes: Based on the mapping relationship, read the idle resource amount from the node attribute of the second scheduling node corresponding to the first scheduling node; The obtained idle resource amount is written into the node attribute of the one first scheduling node to obtain the scheduling resource amount corresponding to the one first scheduling node.

5. The method according to any one of claims 1 to 4, characterized in that: When the resource configuration data is converted into configuration sub-data of each of the multiple offline sub-tasks, each of the target scheduling nodes is used to process an offline sub-task, and the offline sub-task is split from the target offline task; Then, the at least one target scheduling node is used to schedule the idle resources of the corresponding second scheduling node to process the target offline task and obtain the task processing result, including: For the at least one target scheduling node, perform the following operations respectively: According to the resource scheduling strategy of the second scheduling node corresponding to the target scheduling node, the resource scheduling field in the configuration sub-data of the offline sub-task processed by the target scheduling node is updated to obtain configuration update data; Sending the configuration update data to a second scheduling node corresponding to the one target scheduling node, so that the second scheduling node processes the corresponding offline subtask based on the received configuration update data, and generates and returns a subtask result; Based on the received subtask results, a task processing result of the target offline task is generated.

6. The method according to claim 5, characterized in that When the first resource scheduling cluster includes a management node, updating data in a resource scheduling field in configuration sub-data of an offline subtask processed by a target scheduling node according to a resource scheduling policy of a second scheduling node corresponding to the target scheduling node to obtain configuration update data includes any of the following operation modes: Using the management node, adding the resource scheduling field and its field value in the configuration sub-data of an offline sub-task processed by a target scheduling node to the environment variable of the configuration sub-data, and deleting the resource scheduling field and its field value in the configuration sub-data to obtain configuration update data; The management node is used to update the field value of the resource scheduling field in the configuration sub-data of an offline sub-task processed by a target scheduling node to a preset value, thereby obtaining configuration update data.

7. The method according to any one of claims 1 to 4, characterized in that: In the mapping relationship, there is a one-to-one correspondence between the multiple first scheduling nodes and each second scheduling node.

8. An offline task processing device, characterized in that: The method is applied to a first resource scheduling cluster including a plurality of first scheduling nodes, comprising: An acquisition module: used to acquire a mapping relationship between the plurality of first scheduling nodes and each second scheduling node in a plurality of second resource scheduling clusters; Processing module: used to receive a task processing request; the task processing request carries resource configuration data of the target offline task; The processing module is further used to: based on the mapping relationship, select, from the plurality of first scheduling nodes, idle resources in the node resources of the corresponding second scheduling node and at least one target scheduling node that matches the resource configuration data; The processing module is also used to: use the at least one target scheduling node to schedule the idle resources of the corresponding second scheduling nodes to process the target offline task and obtain the task processing result.

9. An offline task processing system, characterized in that: It includes a first resource scheduling cluster and multiple second resource scheduling clusters; there is a mapping relationship between multiple first scheduling nodes included in the first resource scheduling cluster and each second scheduling node in the multiple second resource scheduling clusters; wherein: The first resource scheduling cluster receives a task processing request; wherein the task processing request carries resource configuration data of a target offline task; The first resource scheduling cluster selects, from the plurality of first scheduling nodes, idle resources in the node resources of the corresponding second scheduling node and at least one target scheduling node that matches the resource configuration data based on the mapping relationship; The first resource scheduling cluster schedules idle resources of the second scheduling nodes corresponding to each of the at least one target scheduling node in the plurality of second resource scheduling clusters, processes the target offline task, and obtains a task processing result.

10. The system according to claim 9, characterized in that The system further includes a management node, and the management node performs the following operations respectively for the multiple first scheduling nodes: Based on the mapping relationship, the spatial resource amount is read from the node attribute of the second scheduling node corresponding to a first scheduling node in the plurality of second resource scheduling clusters; wherein the auxiliary information field stores the resource amount of the idle resources in the node resources counted by the corresponding second scheduling node; The obtained idle resource amount is written into the node attribute of the one first scheduling node, so that the first resource scheduling cluster obtains the scheduling resource amount corresponding to the one first scheduling node.

11. The system according to claim 9, characterized in that The system further comprises a management node, and when the resource configuration data is converted into configuration sub-data of respective multiple offline sub-tasks, each of the target scheduling nodes is used to process an offline sub-task, wherein the offline sub-task is split from the target offline task; Then, for the at least one target scheduling node, the management node respectively performs: adding the resource scheduling field and its field value in the configuration sub-data of the offline sub-task processed by one target scheduling node to the environment variable of the configuration sub-data, deleting the resource scheduling field and its field value in the configuration sub-data, obtaining configuration update data, and sending the configuration update data to the second scheduling node corresponding to the one target scheduling node; The second scheduling nodes corresponding to the at least one target scheduling node respectively perform: obtaining resource scheduling fields and field values ​​thereof from the environment variables of the received configuration update data, and updating the resource scheduling configuration of the processing process assigned to the configuration update data based on the obtained resource scheduling fields and field values ​​thereof, and using the processing process to process the corresponding offline subtask based on the configuration update data, and generating and returning the subtask result; The management node generates a task processing result of the target offline task based on the received subtask results.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

13. A computer device, characterized in that: include: A memory for storing program instructions; The processor is used to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 7 according to the obtained program instructions.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images