Task processing method and apparatus, computer-readable storage medium, and electronic device
By analyzing task execution instructions and matching target task execution nodes, the problem of low task execution efficiency in the existing technology is solved, and efficient utilization and flexible configuration of resources are achieved.
Patent Information
- Application Number
- PCT/CN2024/088591
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-04
- Filing Date
- 2024-04-18
- Publication Date
- 2024-11-07
AI Technical Summary
Existing task processing methods cannot automatically match task execution nodes based on task configuration information, resulting in low task execution efficiency.
By receiving task execution instructions sent by the terminal device, parsing the instructions to obtain task configuration information, and matching the target task execution node from the current task execution node, generating and executing scheduling tasks until the task is completed and resources are freed.
Improve task execution efficiency, avoid resource waste of target task execution nodes, and realize flexible configuration of tasks to be executed.
Smart Images

Figure CN2024088591_07112024_PF_FP_ABST
Abstract
Description
Task processing method and device, computer-readable storage medium, and electronic device
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This disclosure claims priority from application number 202310491574.2, application date May 4, 2023, and invention title: Task processing method and device, computer-readable storage medium, and electronic device. The entire contents of this Chinese patent application are incorporated herein by reference. Technical Field
[0003] The embodiments of the present disclosure relate to the technical field of computer resource scheduling, and in particular, to a task processing method, a task processing device, a computer-readable storage medium, and an electronic device. Background Art
[0004] In the existing method, it is impossible to automatically match the task execution node for the task to be executed according to the task configuration information, thereby resulting in low task execution efficiency.
[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in this field.
[0006] Summary of the Invention
[0007] According to one aspect of the present disclosure, a task processing method is provided, comprising:
[0008] Receiving a first task execution instruction sent by a terminal device; wherein the first task execution instruction is generated by the terminal device in response to an input event on a task configuration interface;
[0009] Parsing the first task execution instruction to obtain a task to be executed and first task configuration information corresponding to the task to be executed;
[0010] According to the first task configuration information, matching a target task execution node corresponding to the task to be executed from the current task execution node;
[0011] A scheduling task is generated according to the first task configuration information and the target task execution node, and the scheduling task is executed in the target task execution node to obtain a task execution result corresponding to the task to be executed.
[0012] In an exemplary embodiment of the present disclosure, matching a target task execution node corresponding to the to-be-executed task from a current task execution node according to the first task configuration information includes:
[0013] Acquire current task type information included in the first task configuration information, and match a first task execution node that executes a historically executed task consistent with the current task type information from a task scheduling template;
[0014] Acquire a current task execution node in an idle state from a resource pool, and determine whether the first task execution node exists in the current task execution node;
[0015] If the first task execution node exists in the current task execution node, taking the first task execution node as the target task execution node corresponding to the task to be executed;
[0016] If the first task execution node does not exist in the current task execution node, a target task execution node corresponding to the to-be-executed task is matched from the current task execution node according to the first task configuration information.
[0017] In an exemplary embodiment of the present disclosure, matching a target task execution node corresponding to the to-be-executed task from the current task execution node according to the first task configuration information includes:
[0018] Obtaining current node configuration information of the current task execution node, and determining whether there is target node configuration information matching the first task configuration information in the current node configuration information;
[0019] If the current node configuration information contains target node configuration information that matches the first task configuration information, the current task execution node corresponding to the target node configuration information is used as the target task execution node corresponding to the task to be executed.
[0020] In an exemplary embodiment of the present disclosure, the task processing method further includes:
[0021] If the target node configuration information matching the first task configuration information does not exist in the current node configuration information, generating first failure prompt information corresponding to the target task execution node, and sending the first failure prompt information to the terminal device;
[0022] Receive the second task execution instruction sent by the terminal device after dynamically adjusting the first task configuration information in the first task execution instruction according to the first failure prompt information, and match the target task execution node corresponding to the task to be executed from the current task execution node according to the second task configuration information included in the second task execution instruction.
[0023] In an exemplary embodiment of the present disclosure, matching a target task execution node corresponding to the to-be-executed task from the current task execution node according to the first task configuration information includes:
[0024] Calculating, based on the first task configuration information and the current node configuration information, a network configuration compatibility and / or a resource configuration compatibility between the task to be executed and the current task execution node;
[0025] According to the network configuration adaptability and / or resource configuration adaptability, a target task execution node corresponding to the task to be executed is matched from the current task execution nodes.
[0026] In an exemplary embodiment of the present disclosure, matching a target task execution node corresponding to the task to be executed from the current task execution node according to the network configuration adaptability and / or resource configuration adaptability includes:
[0027] Sorting the current task execution nodes according to the network configuration adaptability to obtain a first node sorting result, and selecting the current task execution node with the largest network adaptability from the first node sorting result as the target task execution node corresponding to the task to be executed; or
[0028] Sorting the current task execution nodes according to the resource configuration adaptability to obtain a second node sorting result, and selecting the current task execution node with the largest resource adaptability from the second node sorting result as the target task execution node corresponding to the task to be executed; or
[0029] Calculating the total fitness between the task to be executed and the current task execution node according to the network configuration fitness and the resource configuration fitness, and sorting the current task execution nodes according to the total fitness to obtain a third node sorting result;
[0030] The current task execution node with the largest resource adaptability is selected from the third node sorting result as the target task execution node corresponding to the task to be executed.
[0031] In an exemplary embodiment of the present disclosure, calculating the total fitness between the to-be-executed task and the current task execution node according to the network configuration fitness and the resource configuration fitness includes:
[0032] Configuring a first weight value and a second weight value for the network configuration adaptability and the resource configuration adaptability; wherein the first weight value is greater than or equal to the second weight value;
[0033] Calculating a first product operation result between the first weight value and the network configuration adaptability, and calculating a second product operation result between the second weight value and the resource configuration adaptability;
[0034] A first sum operation result between the first product operation result and the second product operation result is calculated, and a total fitness between the task to be executed and the current task execution node is obtained according to the first sum operation result.
[0035] In an exemplary embodiment of the present disclosure, the first task configuration information includes first network configuration information and / or first resource configuration information; the current node configuration information includes second network configuration information and / or second resource configuration information;
[0036] Calculating the network configuration compatibility and / or resource configuration compatibility between the task to be executed and the current task execution node according to the first task configuration information and the current node configuration information includes:
[0037] Calculating the network configuration compatibility between the task to be executed and the current task execution node based on the first network configuration information and the second network configuration information; and / or
[0038] The resource configuration compatibility between the task to be executed and the current task execution node is calculated according to the first resource configuration information and the second resource configuration information.
[0039] In an exemplary embodiment of the present disclosure, the first network configuration information includes a first network bandwidth and / or a first network delay, and the second network configuration information includes a second network bandwidth and / or a second network delay;
[0040] Calculating the network configuration compatibility between the task to be executed and the current task execution node according to the first network configuration information and the second network configuration information includes:
[0041] Calculating a first quotient operation result between the first network bandwidth and the second network bandwidth; and / or
[0042] Calculating a second quotient operation result between the first network delay and the second network delay;
[0043] The network configuration compatibility between the task to be executed and the current task execution node is obtained according to the first quotient operation result and / or the second quotient operation result.
[0044] In an exemplary embodiment of the present disclosure, the first resource configuration information includes one or more of a resource configuration of a first central processing unit, a resource configuration of a first memory, and a resource configuration of a first graphics processor;
[0045] The second resource configuration information includes one or more of a resource configuration of a second central processing unit, a resource configuration of a second memory, and a resource configuration of a second graphics processing unit;
[0046] Calculating the resource configuration compatibility between the task to be executed and the current task execution node according to the first resource configuration information and the second resource configuration information includes:
[0047] Calculating a third quotient operation result between the resource configuration of the second central processing unit and the resource configuration of the first central processing unit; and / or
[0048] Calculating a fourth quotient operation result between the resource configuration of the second memory and the resource configuration of the first memory; and / or
[0049] calculating a fifth quotient operation result between the resource configuration of the second graphics processor and the resource configuration of the first graphics processor;
[0050] According to the third quotient operation result and / or the fourth quotient operation result and / or the fifth quotient operation result, the resource configuration adaptability between the task to be executed and the current task execution node is obtained.
[0051] In an exemplary embodiment of the present disclosure, executing the scheduled task in the target task execution node to obtain a task execution result corresponding to the task to be executed includes:
[0052] Sending the scheduling task to the target task execution node, and submitting the container task image included in the scheduling task to the task execution tool in the container orchestration engine through the target task execution node;
[0053] The container task image is executed by the task execution tool to obtain a task execution result corresponding to the task to be executed.
[0054] In an exemplary embodiment of the present disclosure, executing the container task image by the task execution tool to obtain a task execution result corresponding to the task to be executed includes:
[0055] Executing the container task image by the task execution tool to obtain a task execution container;
[0056] The executable image included in the task execution container is started and executed to obtain a task execution result corresponding to the task to be executed.
[0057] In an exemplary embodiment of the present disclosure, the task processing method further includes:
[0058] During the operation of the executable image, monitoring the first operating state of the task execution container and the second operating state of the task execution tool in real time, and generating a monitoring log according to the first operating state and the second operating state; and / or
[0059] The first operating state and the second operating state are displayed.
[0060] In an exemplary embodiment of the present disclosure, the task processing method further includes:
[0061] When it is determined that the execution of the executable image is completed, deleting the executable image in the task execution container, and releasing the second network occupied resources and the second memory occupied resources of the target task execution node occupied by executing the scheduled task;
[0062] The node state of the target task execution node is updated from an occupied state to an idle state, and the target task execution node in the idle state is placed in a resource pool.
[0063] In an exemplary embodiment of the present disclosure, the task processing method further includes:
[0064] The task callback interface corresponding to the terminal device is called, and the task execution result is fed back to the terminal device through the task callback interface.
[0065] In an exemplary embodiment of the present disclosure, the first task execution instruction is generated by the terminal device in the following manner:
[0066] Displaying a task execution interface on the display interface of the terminal device;
[0067] In response to a first input event on a task execution interface, determining a task to be executed, and displaying a task configuration interface corresponding to the task to be executed; wherein the task to be executed includes a model training task and / or a data analysis task;
[0068] In response to a second input event acting on the task configuration interface, determining first task configuration information of a task to be executed;
[0069] In response to a third input event acting on the task configuration interface, a first task execution instruction is generated according to the task to be executed and the first task configuration information, and the first task execution instruction is sent to a task execution framework.
[0070] According to one aspect of the present disclosure, there is provided a task processing device, comprising:
[0071] A first receiving module is configured to receive a first task execution instruction sent by a terminal device; wherein the first task execution instruction is generated by the terminal device in response to an input event on a task configuration interface;
[0072] a first parsing module, configured to parse the first task execution instruction to obtain a task to be executed and first task configuration information corresponding to the task to be executed;
[0073] a task execution node determination module, configured to match a target task execution node corresponding to the task to be executed from current task execution nodes according to the first task configuration information;
[0074] The task execution module is used to generate a scheduling task according to the first task configuration information and a target task execution node, and execute the scheduling task in the target task execution node to obtain a task execution result corresponding to the task to be executed.
[0075] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements any one of the task processing methods described above.
[0076] According to one aspect of the present disclosure, there is provided an electronic device, including:
[0077] processor; and
[0078] a memory for storing executable instructions of the processor;
[0079] The processor is configured to execute any one of the above-mentioned task processing methods by executing the executable instructions.
[0080] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0081] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.
[0082] FIG1 schematically shows a flow chart of a task processing method according to an exemplary embodiment of the present disclosure.
[0083] FIG2 schematically shows a block diagram of a task processing system according to an exemplary embodiment of the present disclosure.
[0084] FIG3 schematically shows an exemplary structural diagram of a server according to an exemplary embodiment of the present disclosure.
[0085] FIG4 schematically illustrates a structural example diagram of a containerized application management system Kubernetes according to an exemplary embodiment of the present disclosure.
[0086] FIG5 schematically illustrates an example diagram of a specific generation process of a first task execution instruction according to an example embodiment of the present disclosure.
[0087] FIG6 schematically shows an example interface diagram of a task configuration interface according to an exemplary embodiment of the present disclosure.
[0088] FIG7 schematically shows a flow chart of a method for matching a target task execution node corresponding to the to-be-executed task from among current task execution nodes according to the first task configuration information according to an exemplary embodiment of the present disclosure.
[0089] FIG8 schematically illustrates an example interface diagram of a display interface obtained when a scheduling task fails to be created according to an exemplary embodiment of the present disclosure.
[0090] FIG9 schematically illustrates an example interface diagram of a usage status of a first central processing unit, a usage status of a first memory, and a usage status of a first graphics processing unit according to an exemplary embodiment of the present disclosure.
[0091] FIG10 schematically shows an example diagram of an interface for monitoring logs according to an exemplary embodiment of the present disclosure.
[0092] FIG11 schematically shows a flowchart of dynamically adjusting a node with abnormal operation according to an exemplary embodiment of the present disclosure.
[0093] FIG12 schematically shows an interface display diagram of a resource release situation according to an exemplary embodiment of the present disclosure.
[0094] FIG13 schematically shows a block diagram of a task processing device according to an exemplary embodiment of the present disclosure.
[0095] FIG14 schematically illustrates an electronic device for implementing the above-mentioned task processing method according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0096] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.
[0097] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0098] Some Kubernetes-based resource scheduling solutions, while offering job grouping algorithms and resource allocation strategies, only consider the use and recovery of some resources, resulting in an incomplete resource matching approach. Furthermore, since scheduling strategies are limited to existing job groups, ease of use is often overlooked. Furthermore, some intelligent scheduling solutions, while offering intelligent scheduling strategies, lack corresponding flexibility mechanisms for other areas of distributed collaboration. Furthermore, these solutions lack user support for decision-making, preventing users from fine-tuning strategies in the event of errors and requiring them to regenerate resource allocation strategies.
[0099] Based on this, this exemplary embodiment first provides a task processing method that can be run on a server, server cluster, or cloud server. Of course, those skilled in the art can also run the disclosed method on other platforms as needed, and this exemplary embodiment does not specifically limit this. Specifically, referring to FIG1 , the task processing method may include the following steps:
[0100] Step S110. Receive a first task execution instruction sent by the terminal device; wherein the first task execution instruction is generated by the terminal device in response to an input event on the task configuration interface;
[0101] Step S120: Parse the first task execution instruction to obtain a task to be executed and first task configuration information corresponding to the task to be executed;
[0102] Step S130. According to the first task configuration information, matching the target task execution node corresponding to the task to be executed from the current task execution node;
[0103] Step S140: Generate a scheduling task according to the first task configuration information and the target task execution node, and execute the scheduling task in the target task execution node to obtain a task execution result corresponding to the task to be executed.
[0104] In the above-mentioned task processing method, on the one hand, a first task execution instruction sent by a terminal device is received; then the first task execution instruction is parsed to obtain the task to be executed and the first task configuration information corresponding to the task to be executed; and then, according to the first task configuration information, the target task execution node corresponding to the task to be executed is matched from the current task execution node; finally, a scheduling task is generated according to the first task configuration information and the target task execution node, and the scheduling task is executed in the target task execution node to obtain a task execution result corresponding to the task to be executed; since the target task execution node can be matched according to the first task configuration information, and then the task is executed in the target task execution node, the problem in the prior art that the task execution node cannot be automatically matched for the task to be executed according to the task configuration information, thereby making the task execution efficiency low, is solved; on the other hand, the problem of node resource waste of the target task execution node can also be avoided; on the other hand, since the first task execution instruction is generated by the terminal device in response to an input event for the task configuration interface, flexible configuration of the task to be executed is realized.
[0105] Hereinafter, the task processing method described in the exemplary embodiments of the present disclosure will be further explained and illustrated with reference to the accompanying drawings.
[0106] First, the specific terms involved in the exemplary embodiments of the present disclosure are explained and illustrated.
[0107] Kubernetes: Also known as K8s, it is an open source system for automatically deploying, scaling, and managing containerized applications.
[0108] Interface callback: also known as callback, refers to the feedback mechanism triggered after the completion of some tasks.
[0109] Container monitoring: mainly refers to status monitoring and operation status checking of containers generated by k8s services.
[0110] Secondly, the technical implementation principle of the example embodiment of the present disclosure is explained and illustrated. Specifically, a task processing method recorded in the example embodiment of the present disclosure proposes an implementation method of a kubernetes resource scheduling system that supports multi-node task collaboration, aiming to solve the problem of how to select execution nodes, monitor node execution status, and simplify Kubernetes management in large-scale deep learning training tasks or multi-node concurrent collaborative tasks; at the same time, it can also solve complex scheduling problems such as Kubernetes service calls and container management in actual production practices; in the process of actual application, the task management method can involve multiple different aspects such as task management, interface callbacks, and container monitoring.
[0111] Furthermore, the task processing system involved in the exemplary embodiments of the present disclosure is explained and illustrated. Specifically, as shown in FIG2 , the task processing system may include a terminal device 210 and a server 220 ; wherein the terminal device may be connected to the server via a wired network or a wireless network; and, in actual application, the terminal device may be used to generate task execution instructions and send the task execution instructions to the server; and the server may be used to implement the task processing method described in the exemplary embodiments of the present disclosure.
[0112] In an exemplary embodiment, as shown in reference to FIG3 , the server side described here can also be considered as the server side where the task execution framework is located; at the same time, in the task execution framework described in the exemplary embodiment of the present disclosure, it can include a task registration module 310, a deployment scheduling module 320, a task management module 330, a resource monitoring module 340, a resource recovery module 350, a callback management module 360, a containerized application management system Kubernetes 370 and a call service interface 380; wherein, the connection relationship between each module can be referred to as shown in FIG3 . Among them, the task registration module 310 can be used to read the task configuration information in the task execution instruction and register the task configuration information to the task management module 330; the task management module 330 can be used to load the task configuration information and read the Kubernetes node status, and then generate a scheduling task based on the task configuration information and the node status, combined with the intelligent scheduling strategy, and send the scheduling task to the deployment scheduling module 320; the deployment scheduling module 320 reads the scheduling task information and sends the Job to Kubernetes. After the Job runs successfully, the Job and the pod information included in the Job are registered with the resource monitoring module 340, and the resource monitoring module 340 monitors the subsequent Job, P od and other running status; the resource monitoring module 340 can be used to monitor the running status of Job, Pod, etc., and provide real-time feedback to the task management module 330, and then display the task running status through the task management module 330; the resource recovery module 350 can be used to release the node resources used for task execution after receiving the task completion information sent by the task management module, and return the node resources to the resource pool; the callback management module 360 can be used to allow the user to register a callback address, and after receiving the callback request, the execution information and execution results of the task sent by the task management module are fed back to the external service in the form of a callback task by calling the service interface 380, thereby facilitating the docking and integration of external services.
[0113] In an example embodiment, the task registration module described above may include a configuration parsing module, a subtask decomposition and registration module, and a callback registration module; wherein the configuration parsing module may be used to parse the first task execution instruction to obtain the first task configuration information; the subtask decomposition module may be used to perform task decomposition and / or task registration based on the first task configuration information; at the same time, the callback registration module may be used to receive a registration request for a callback interface sent by a user and to call a service interface registration, and then to feedback the task execution result or character execution log based on the call service interface. Further, in actual application, the task may be transmitted in the form of a data packet via an information channel (such as a message queue telemetry transmission MQTT); for example, the scheduling task may be transmitted from the task registration module to the task management module via the information channel, or from the task management module to the deployment scheduling module, etc. At the same time, the data packet described here may include resource information such as the current task execution node in an idle state; and the resource monitoring module described above may monitor node information such as the current task execution node and the idle task execution node, and then write the corresponding information into the data packet.
[0114] In an example embodiment, the resource pool described herein is a resource pool set in the task execution framework for placing the current task execution node in an idle state; the resource pool may include one or more current task execution nodes; at the same time, in the current task execution node, it may be based on Kubernetes technology, providing services based on many nodes under a microservice architecture, and the service may be based on collaboration between multiple nodes; at the same time, in the process of deploying the current task execution node, there may be the following two different situations: one is that each current task execution node can be set in a different server respectively, that is, only one current task execution node is set in a server; in this scenario, when performing actual task management or task execution, it may include Tasks that rely on collaborative operations between different servers may also include tasks that can be independently executed by a single server; and, in scenarios that rely on collaborative operations between different servers, it can be applicable to large-scale cluster deployment environments with massive data, which can be beneficial to the operations of smart parks / smart cities; another is that the current task execution nodes can be set in the same server, that is, multiple current task execution nodes can be set in a server, or the current task execution nodes can be evenly set in a certain number of servers; in this scenario, when the task is actually executed, it relies on the collaborative operation of the virtual memory opened for different nodes under the server. This situation is used in deployment environments involving less data, such as banking scenarios under a single function (such as passenger flow identification).
[0115] In an example embodiment, a containerized application management system Kubernetes can be used to perform scheduling tasks. As shown in FIG4 , the containerized application management system Kubernetes may include a container (Pod) 410, a service port (Service) 420 exposed to the outside by the container, a container load balancing (Ingress) 430, a first management tool (Deployment) 440, and a second management tool (Job) 450; wherein, the container can be used to perform corresponding tasks; the service port is used to provide a communication connection between the container and the outside world, such as writing a scheduling task to the Pod through the service port; the container load balancing can be used to achieve load balancing of the Pod, thereby achieving the purpose of balancing the load of each Pod; the first management tool can be used to manage the Pod, the service port, and the container load balancing, and the second management tool can be used to control the Pod to perform corresponding tasks.
[0116] Based on the above records, it can be known that the task processing system recorded in the example embodiment of the present disclosure has the following characteristics compared to other Kubernetes-based scheduling systems: on the one hand, the configuration is simple and flexible; specifically, the scheduling is managed from the task dimension, and only the basic information and requirements of the task operation need to be provided, and the Kubernetes configuration file can be automatically generated and the task can be issued; on the other hand, the monitoring system is complete; specifically, the execution status can be monitored from the perspective of the entire life cycle of the task, starting from the verification of the task creation conditions, to the task creation status, execution status, completion status, and resource recovery; on the other hand, it can run through the entire task process, and any problems will alert the user and allow the user to make corresponding decisions; further, the system can also automatically generate a scheduling strategy based on the user-configured operating requirement information, select the node to execute the task, and complete the task issuance, thereby avoiding the problem of user execution failure due to the selected resources not meeting the requirements.
[0117] The following will explain and illustrate the specific generation process of the first task execution instruction involved in this disclosure in conjunction with Figure 5. Specifically, referring to Figure 5, the specific generation process of the first task execution instruction may include the following steps:
[0118] Step S510: displaying a task execution interface on the display interface of the terminal device.
[0119] Step S520, in response to the first input event to the task execution interface, determining the task to be executed, and displaying the task configuration interface corresponding to the task to be executed; wherein the task to be executed includes a model training task and / or a data analysis task.
[0120] Step S530 : determining first task configuration information of the task to be executed in response to a second input event acting on the task configuration interface.
[0121] Step S540 : In response to a third input event acting on the task configuration interface, a first task execution instruction is generated according to the task to be executed and the first task configuration information, and the first task execution instruction is sent to a task execution framework.
[0122] Hereinafter, steps S510 to S540 will be explained and illustrated. Specifically, in the actual application process, due to the computing power limitation or data security limitation on the terminal device side, there is usually a scenario where the corresponding task needs to be executed on the server side; based on this scenario, when it is necessary to request the server side to assist in executing the task, first, the task execution interface can be displayed on the display interface of the terminal device; wherein, the task execution interface recorded here can be implemented based on a certain client; that is, the task execution interface can be displayed by opening the corresponding client; then, the user can determine the corresponding task to be executed according to actual needs, and then display the task configuration interface corresponding to the task to be executed; wherein, the task to be executed recorded here can include model training tasks, data analysis tasks, and of course interface testing tasks, etc., and this example does not impose special restrictions on this; at the same time, the task configuration interface recorded here can be specifically shown in Figure 6; further, the user can also enter the corresponding task configuration information according to the displayed task configuration interface; wherein, the task configuration information recorded here can include image information, mount file information, resource configuration information, node type information, node specification information, delayed start and stop information, run command information, network bandwidth information, network delay information, run parameter mode information, run parameter information and run command preview information, etc. Among them, the image information recorded here may include the image name and image version, etc., and the mounted file may include the corresponding script file, etc.; the resource configuration information may include single-machine training or multi-machine multi-card training, etc., and the node type may include CPU or GPU, etc.; at the same time, after the task configuration information is entered, you can click the task issuance control on the task configuration interface to obtain the first task execution instruction and submit the task instruction to the task execution framework, that is, the server side.
[0123] It should be noted here that, since tasks can include different types, the task configuration information included in the task configuration interface recorded here is illustrated using the model training task as an example. For other tasks, such as data request tasks or interface testing tasks, the corresponding task configuration interface can be displayed according to actual conditions. This example does not impose any special restrictions on this. At the same time, the model training tasks recorded here may include deep neural network model training tasks, or federated model training tasks and cross-modal model training tasks (which can be understood as comprehensive semantic understanding based on the fusion of multiple single-modal features, and then completing specific classification, matching, generation and other tasks) and so on.
[0124] The task processing method shown in FIG1 will be further explained and illustrated below in conjunction with FIG2 to FIG6. Specifically:
[0125] In step S110 , a first task execution instruction sent by a terminal device is received; wherein the first task execution instruction is generated by the terminal device in response to an input event on a task configuration interface.
[0126] Specifically, the reason why the first task execution instruction is limited here to be generated by the terminal device in response to an input event for the task configuration interface is so that during the creation of the scheduled task, if it is detected that the scheduled task creation fails, the task configuration information included in the first task execution instruction can be manually adjusted, thereby avoiding the problem of being unable to execute the pending task due to the failure of the scheduled task creation.
[0127] In step S120 , the first task execution instruction is parsed to obtain a task to be executed and first task configuration information corresponding to the task to be executed.
[0128] Specifically, the first task configuration information recorded here may include task image information, mount file information, first network configuration information, and first resource configuration information, etc.; wherein, the first network configuration information may include first network bandwidth information and first network delay information, etc., and the first resource configuration information may include first CPU information, first GPU information, and first memory information, etc. In actual application, the task configuration information can be configured according to actual needs, and this example does not impose any special restrictions on this; at the same time, for different types of tasks to be executed, the corresponding first task configuration information may be partially the same or completely different, etc., and this example does not impose any special restrictions on this.
[0129] In step S130 , according to the first task configuration information, a target task execution node corresponding to the task to be executed is matched from the current task execution nodes.
[0130] Specifically, referring to FIG7 , according to the first task configuration information, matching the target task execution node corresponding to the task to be executed from the current task execution node may include the following steps:
[0131] Step S710: obtaining the current task type information included in the first task configuration information, and matching a first task execution node for executing a historically executed task consistent with the current task type information from a task scheduling template;
[0132] Step S720: obtaining a current task execution node in an idle state from a resource pool, and determining whether the first task execution node exists in the current task execution node;
[0133] Step S730: If the first task execution node exists in the current task execution node, the first task execution node is used as the target task execution node corresponding to the task to be executed;
[0134] Step S740 : If the first task execution node does not exist in the current task execution node, a target task execution node corresponding to the to-be-executed task is matched from the current task execution node according to the first task configuration information.
[0135] Below, steps S710 to S740 will be explained and illustrated. Specifically, in the actual application process, the first task execution node can be matched according to the current task type information of the task to be executed; that is, it is determined whether there is a historical task consistent with the current task type information in the task scheduling template; if so, the first task execution node to which the historical task is assigned is determined; of course, it is also necessary to determine whether the first task execution node is in an idle state. If so, the first task execution node can be directly used as the target task execution node; if the first task execution node does not exist or the first task execution node is in an occupied state, it is also necessary to match the target task execution node corresponding to the task to be executed from the current task execution node according to the first task configuration information.
[0136] In an example embodiment, matching the target task execution node corresponding to the task to be executed from the current task execution node according to the first task configuration information can be achieved as follows: first, obtaining the current node configuration information of the current task execution node, and determining whether there is target node configuration information matching the first task configuration information in the current node configuration information; second, if there is target node configuration information matching the first task configuration information in the current node configuration information, the current task execution node corresponding to the target node configuration information is used as the target task execution node corresponding to the task to be executed; further, if there is no target node configuration information matching the first task configuration information in the current node configuration information, generating first failure prompt information corresponding to the target task execution node, and sending the first failure prompt information to the terminal device; finally, receiving the second task execution instruction sent by the terminal device after dynamically adjusting the first task configuration information in the first task execution instruction according to the first failure prompt information, and matching the target task execution node corresponding to the task to be executed from the current task execution node according to the second task configuration information included in the second task execution instruction. Specifically, in the process of actual application, it is possible to first determine whether the current node configuration information contains target node configuration information corresponding to the first task configuration information required to execute the task to be executed; that is, determine whether the current task execution node contains a target task execution node capable of executing the task to be executed; if so, directly determine the target task execution node; if not, a first failure prompt message can be generated based on the specific mismatch reason; for example, the required network bandwidth or network delay does not match, or the required CPU resources or GPU resources or memory resources do not match, etc.; then, the first failure prompt message is sent to the terminal device; after the terminal device receives the first failure prompt message, the first task configuration information can be adaptively and dynamically adjusted based on the specific mismatch reason, thereby obtaining the second task configuration information, and then generating the second task execution instruction based on the second task configuration information and re-matching the corresponding target task execution node. Among them, the display interface obtained when the scheduling task fails to be created can be specifically shown in Figure 8.
[0137] In an example embodiment, in the display interface of the scheduling task shown in Figure 8, it may include multiple different scheduling tasks, for example, the scheduling tasks may include V0001-V0012, etc.; in the actual application process, in some special application scenarios, there will be scenarios where multiple scheduling tasks need to be executed at the same time; for example, in the scenarios of smart banking or smart medical care, it is necessary to execute multiple related tasks such as face recognition, body recognition, passenger flow recognition, etc. that are simultaneously set up in the smart park, etc.; in this scenario, the tasks to be executed may include a platform for developers focusing on the field of image recognition to conduct model training at the same time, and simultaneously execute convolutional neural network models, attention mechanism models, bidirectional long short-term memory network models, recurrent neural network models, etc.; in this case, it is necessary to reasonably allocate resources under limited computing power, so that the corresponding model can be trained at the fastest speed, so as to achieve the purpose of face recognition, body recognition or passenger flow recognition, and perform corresponding measures based on the face recognition results, body recognition results or passenger flow recognition results, such as VIP services or general user services, etc.
[0138] In an example embodiment, according to the first task configuration information, matching the target task execution node corresponding to the task to be executed from the current task execution node can be achieved in the following manner: first, according to the first task configuration information and the current node configuration information, calculating the network configuration adaptability and / or resource configuration adaptability between the task to be executed and the current task execution node; secondly, according to the network configuration adaptability and / or resource configuration adaptability, matching the target task execution node corresponding to the task to be executed from the current task execution node. That is, if it is necessary to match the target task execution node corresponding to the task to be executed from the current task execution node, it can be considered from two dimensions, namely network configuration adaptability and resource configuration adaptability; at the same time, in the actual application process, a smart scheduling strategy can be generated based on the network configuration adaptability and resource configuration adaptability; among them, the smart scheduling here can be reflected in the following aspects: on the one hand, if only the first network configuration information appears in the first task configuration information but the first resource configuration information does not appear, the target task execution node can be matched only based on the dimension of network configuration adaptability; on the other hand, if only the first resource configuration information appears in the first task configuration information but the first network configuration information does not appear, the target task execution node can be matched only based on the dimension of resource configuration adaptability; on the other hand, if both the first resource configuration information and the first network configuration information appear in the first task configuration information, the target task execution node can be matched from two dimensions.
[0139] In an example embodiment, matching the target task execution node corresponding to the task to be executed from the current task execution node according to the network configuration adaptability and / or resource configuration adaptability can be achieved in the following ways: one implementation method is to sort the current task execution nodes according to the network configuration adaptability to obtain a first node sorting result, and select the current task execution node with the largest network adaptability from the first node sorting result as the target task execution node corresponding to the task to be executed; another implementation method is to sort the current task execution nodes according to the resource configuration adaptability to obtain a second node sorting result, and select the current task execution node with the largest resource adaptability from the second node sorting result as the target task execution node corresponding to the task to be executed; yet another implementation method is to calculate the total adaptability between the task to be executed and the current task execution node according to the network configuration adaptability and the resource configuration adaptability, and sort the current task execution nodes according to the total adaptability to obtain a third node sorting result; and select the current task execution node with the largest resource adaptability from the third node sorting result as the target task execution node corresponding to the task to be executed. That is, in the actual application process, if the first task configuration information only includes the first network configuration information, the current task execution node with the largest network configuration adaptability can be directly selected as the target task execution node; if the first task configuration information only includes the first resource configuration information, the current task execution node with the largest resource configuration adaptability can be directly selected as the target task execution node; of course, if the first task configuration information includes both the first resource configuration information and the first network configuration information, the resource configuration adaptability and the network configuration adaptability can be weighted and summed to determine the corresponding target task execution node.
[0140] In an example embodiment, the total fitness between the task to be executed and the current task execution node is calculated based on the network configuration fitness and the resource configuration fitness, which can be achieved in the following manner: first, a first weight value and a second weight value are configured for the network configuration fitness and the resource configuration fitness; wherein the first weight value is greater than or equal to the second weight value; secondly, a first product operation result between the first weight value and the network configuration fitness, and a second product operation result between the second weight value and the resource configuration fitness; then, a first sum operation result between the first product operation result and the second product operation result is calculated, and the total fitness between the task to be executed and the current task execution node is obtained based on the first sum operation result. Specifically, if the corresponding target task execution node is determined by taking a weighted sum of the resource configuration adaptability and the network configuration adaptability, it is necessary to configure the corresponding first weight value and the second weight value for the network configuration adaptability and the resource configuration adaptability; wherein, the specific configuration process of the first weight value and the second weight value can be configured through manual experience or through model prediction, and this example does not impose any special restrictions on this; at the same time, when configured through model prediction, the network model used here may include but is not limited to a convolutional neural network model, a recurrent neural network model, a deep neural network model, and a decision tree model, etc., and this example does not impose any special restrictions on this; at the same time, the reason why the first weight value is limited to be greater than or equal to the second weight value is because in actual application, the network configuration adaptability needs to take precedence over the resource configuration adaptability, so as to achieve the purpose of improving task execution efficiency.
[0141] In an example embodiment, the current node configuration information recorded above may include second network configuration information and / or second resource configuration information; wherein, based on the first task configuration information and the current node configuration information, calculating the network configuration adaptability and / or resource configuration adaptability between the task to be executed and the current task execution node can be achieved in the following manner: based on the first network configuration information and the second network configuration information, calculating the network configuration adaptability between the task to be executed and the current task execution node; and / or based on the first resource configuration information and the second resource configuration information, calculating the resource configuration adaptability between the task to be executed and the current task execution node.
[0142] In an example embodiment, the second network configuration information described above may include a second network bandwidth and / or a second network delay; wherein, based on the first network configuration information and the second network configuration information, calculating the network configuration adaptability between the task to be executed and the current task execution node can be achieved by: calculating the first quotient operation result between the first network bandwidth and the second network bandwidth; and / or calculating the second quotient operation result between the first network delay and the second network delay; and obtaining the network configuration adaptability between the task to be executed and the current task execution node based on the first quotient operation result and / or the second quotient operation result. The specific calculation method of the network configuration adaptability can be referred to as shown in the following formula (1):
[0143] Among them, Fitness is the network configuration adaptability, Bandwith require The first network bandwidth, that is, the required network bandwidth; Bandwith actual It is the second network bandwidth, that is, the actual network bandwidth; Delay require is the first network delay, that is, the required network delay; Delay actual is the second network delay, also known as the actual network delay. It's important to note that since network bandwidth and delay are generally consistent within the same region, the network bandwidth and delay for different regions can be calculated in advance, and the matching degree can be calculated based on the target bandwidth and delay. Furthermore, the sum of the quotient of the required bandwidth and the average actual bandwidth of each node is added to the quotient of the required delay and the average actual delay of each node. A larger sum indicates a higher degree of fit.
[0144] In an example embodiment, the first resource configuration information recorded above may include the resource configuration of the first central processing unit, the resource configuration of the first memory, and the resource configuration of the first graphics processing unit, etc., and the second resource configuration information includes the resource configuration of the second central processing unit, the resource configuration of the second memory, and the resource configuration of the second graphics processing unit, etc.; wherein, according to the first resource configuration information and the second resource configuration information, the resource configuration adaptability between the task to be executed and the current task execution node is calculated, which can be achieved in the following manner: calculating the third quotient operation result between the resource configuration of the second central processing unit and the resource configuration of the first central processing unit; and / or calculating the fourth quotient operation result between the resource configuration of the second memory and the resource configuration of the first memory; and / or calculating the fifth quotient operation result between the resource configuration of the second graphics processing unit and the resource configuration of the first graphics processing unit; according to the third quotient operation result and / or the fourth quotient operation result and / or the fifth quotient operation result, the resource configuration adaptability between the task to be executed and the current task execution node is obtained. wherein, the specific calculation formula of the resource configuration adaptability can refer to the following formula (2):
[0145] Among them, Fitness node Is the resource configuration adaptability; CPU require It is the resource configuration of the first central processing unit, that is, the resource configuration of the required central processing unit; CPU node is the resource configuration of the second CPU and the actual resource configuration of the CPU of the target task execution node; MEM require It is the resource configuration of the first memory, that is, the resource configuration of the required memory; MEM node is the resource configuration of the second memory, that is, the actual memory resource configuration of the target task execution node; GPU require It is the resource configuration of the first graphics processor, that is, the resource configuration of the required graphics processor; GPU node It is the resource configuration of the second graphics processor, that is, the actual resource configuration of the graphics processor of the target task execution node.
[0146] It should be noted that the calculation method of resource adaptability is mainly determined by the matching of hardware resource requirements. For example, if the requirement is 4 CPUs and 8GB of memory, then all configurations greater than or equal to this meet this adaptation requirement. At the same time, the greater the node resource value is than the required resource value, the higher the adaptability. Of course, in actual application, this calculation method may have a situation where one value is very high and the other values do not meet the requirements. Therefore, it is necessary to verify whether each value is greater than or equal to 1 before calculating, and then sum it. That is, only when the third quotient operation result, the fourth quotient operation result, and the fifth quotient operation result are all greater than or equal to 1, can the subsequent summation operation be performed. Of course, the reason why the subsequent summation operation can only be performed when the third quotient operation result, the fourth quotient operation result, and the fifth quotient operation result are all greater than or equal to 1 is to enable the target task execution node to smoothly implement the specific execution of the task based on its own CPU resources, GPU resources, and memory resources, thereby avoiding the problem of task execution failure.
[0147] It should be further explained here that when performing node scheduling, the region is determined based on the network adaptability first, and then the priority is selected based on the adaptability of the resource configuration; at the same time, after each successful scheduling, the result of this scheduling will be saved as a scheduling template. When a similar scheduling task is executed again, the node of the scheduling template will be directly used. When the node is occupied, the node matching will be performed again.
[0148] In step S140 , a scheduling task is generated according to the first task configuration information and the target task execution node, and the scheduling task is executed in the target task execution node to obtain a task execution result corresponding to the task to be executed.
[0149] In this example embodiment, first, a scheduling task is generated based on the first task configuration information and the target task execution node; wherein the scheduling task includes not only the node identifier (Node_id) of the target task execution node, but also the first task configuration information of the task to be executed, the task identifier, and the device identifier of the terminal device that sends the first task execution instruction, and of course other information may also be included, and this example does not impose any special restrictions on this; at the same time, the target task execution node recorded here may be a server with different functions, such as an interface testing server, a model training server, and a data request server, etc., and may be equipped with corresponding servers according to actual needs, and this example does not impose any special restrictions on this.
[0150] Secondly, the scheduling task is executed in the target task execution node to obtain a task execution result corresponding to the task to be executed. Specifically, it can be achieved in the following ways: first, the scheduling task is sent to the target task execution node, and the container task image included in the scheduling task is submitted to the task execution tool in the container orchestration engine through the target task execution node; secondly, the container task image is executed by the task execution tool to obtain a task execution result corresponding to the task to be executed. Among them, executing the container task image by the task execution tool to obtain a task execution result corresponding to the task to be executed can be achieved in the following ways: first, executing the container task image by the task execution tool to obtain a task execution container; secondly, starting and executing the executable image included in the task execution container to obtain a task execution result corresponding to the task to be executed. Among them, the executable image recorded here can be obtained from the corresponding image platform according to the image name and image version included in the first task configuration information; at the same time, in the actual application process, the obtained container task image can be instantiated according to the mount file included in the first task configuration information to obtain a task execution container, and then the executable image included in the task execution container is started and executed to obtain the corresponding task execution result; among them, the task execution results recorded here may include model training results, data analysis results or interface test results, etc., and this example does not impose any special restrictions on this.
[0151] In an example embodiment, whether it is the current task execution node recorded above or the target task execution node recorded here, a daemon program can be set in each node, and then the daemon program is started to monitor whether a scheduling task is issued in the target task execution node through a monitoring function. If a scheduling task is issued, the library file can be directly called to execute the scheduling task; in another possible example embodiment, the execution of the task can also be achieved in other ways, such as establishing a communication link between the target task execution node and the deployment scheduling module, and then after the deployment scheduling module issues the scheduling task to the target task execution node, the corresponding task execution instruction is issued to the target task execution node, so that the target task execution node executes the scheduling task based on the task execution instruction; in the actual application process, it can be determined by itself according to actual needs, and this example does not impose any special restrictions on this.
[0152] In an example embodiment, the task processing method may also include: during the operation of the executable image, monitoring the first operating state of the task execution container and the second operating state of the task execution tool in real time, and generating a monitoring log based on the first operating state and the second operating state; and / or displaying the first operating state and the second operating state. That is, during the task processing process, it is also necessary to detect the first operating state of the task execution container Pod and the second operating state of the task execution tool Job in real time; wherein, the operating state recorded here may include the usage state of the first central processing unit, the usage state of the first memory, the usage state of the first graphics processor, the operating state of the container, and the monitoring log, etc.; at the same time, it is also necessary to feedback the corresponding operating state to the task management module in real time, and display the task running status through the task management module; wherein, the usage state of the first central processing unit, the usage state of the first memory, and the usage state of the first graphics processor can be specifically shown in Figure 9, and the specific monitoring log can be specifically shown in Figure 10. Among them, in the actual application process, the resource monitoring module can monitor the running status of Job, Pod, etc., and provide real-time feedback to the task management module 330, and then display the task running status through the task management module 330; among them, the specific display situation can be referred to as shown in Figure 10.
[0153] It should be noted here that if any target task execution node is monitored to have an operational abnormality, the information of the operational abnormality can be directly fed back to the corresponding terminal device, so that the user can dynamically adjust the parameter information included in the first task configuration information through the terminal device, and then redeploy the nodes in the task; the specific operation process can be referred to as shown in Figure 11.
[0154] In an example embodiment, the task processing method may also include: when it is determined that the execution of the executable image is completed, deleting the executable image in the task execution container, and releasing the second network occupied resources and the second memory occupied resources of the target task execution node occupied by executing the scheduled task; updating the node state of the target task execution node from the occupied state to the idle state, and placing the target task execution node in the idle state into the resource pool. That is, in the actual application process, when the execution of the task to be executed is completed, the corresponding node resources need to be released and the node is fed back to the resource pool; at the same time, the user can view the resource release status of the node in real time through the cluster status interface; wherein, the specific interface display diagram can be shown in Figure 12. wherein, after receiving the task completion information issued by the task management module, the resource recovery module 350 releases the node resources used for task execution and returns the node resources to the resource pool; wherein, the specific display of the specific resource release situation can be shown in Figure 12.
[0155] In an exemplary embodiment, the task processing method may further include: invoking a task callback interface corresponding to the terminal device, and feeding back the task execution result to the terminal device via the task callback interface. Specifically, in actual application, users may be allowed to register a callback address, and then use the callback address to view task execution results and task execution log information, thereby facilitating task docking and integration with external services for the pending task.
[0156] At this point, the task execution method recorded in the example embodiment of the present disclosure has been fully realized. Based on the above-mentioned recorded content, it can be known that the task execution method recorded in the example embodiment of the present disclosure has at least the following advantages: on the one hand, the focus can be shifted to the task execution itself; that is, in the actual application process, users do not need to care about how many task groups Kubernetes scheduling will be split into and which nodes they will be assigned to; at the same time, the premise for user use is simple configuration and a complete monitoring system, which is simpler to configure and use under the premise of making the best use of resources; that is, the user only needs to provide the first task configuration information to automatically realize the allocation and execution of tasks, thereby achieving the purpose of improving the execution efficiency of tasks; on the other hand, an intelligent scheduling model; that is, it can be based on the user The system can tailor the scheduling strategy to the operation requirements of the task, ensuring that the running nodes meet the task requirements and are as optimal as possible; on the other hand, it allows users to make decisions; that is, when problems arise during task issuance and task execution, users can make decisions to adjust whether the resource allocation mode is strict or relatively loose. When errors occur, users can fine-tune the allocation strategy; further, component-based access is implemented without the need for independent deployment; that is, it can provide Kubernetes container management as an independent service, and can also be integrated into the development of other container-related business systems as a development component; at the same time, the entire Kubernetes container life cycle can be maintained from the perspective of a single task, reducing the complexity of use.
[0157] The exemplary embodiments of the present disclosure further provide a task processing device. Specifically, as shown in FIG13 , the task processing device may include a first receiving module 1310, a first parsing module 1320, a task execution node determination module 1330, and a task execution module 1340.
[0158] The first receiving module 1310 may be configured to receive a first task execution instruction sent by a terminal device; wherein the first task execution instruction is generated by the terminal device in response to an input event on a task configuration interface;
[0159] A first parsing module 1320 may be configured to parse the first task execution instruction to obtain a task to be executed and first task configuration information corresponding to the task to be executed;
[0160] The task execution node determination module 1330 may be configured to match a target task execution node corresponding to the task to be executed from the current task execution nodes according to the first task configuration information;
[0161] The task execution module 1340 may be configured to generate a scheduling task according to the first task configuration information and the target task execution node, and execute the scheduling task in the target task execution node to obtain a task execution result corresponding to the task to be executed.
[0162] In an exemplary embodiment of the present disclosure, the task execution node determination module 1330 can also be configured to: obtain the current task type information included in the first task configuration information, and match the first task execution node that executes the historical executed task consistent with the current task type information from the task scheduling template; obtain the current task execution node in an idle state from the resource pool, and determine whether the first task execution node exists in the current task execution node; if the first task execution node exists in the current task execution node, then use the first task execution node as the target task execution node corresponding to the task to be executed; if the first task execution node does not exist in the current task execution node, then match the target task execution node corresponding to the task to be executed from the current task execution node according to the first task configuration information.
[0163] In an exemplary embodiment of the present disclosure, the task execution node determination module 1330 can also be configured to: obtain the current node configuration information of the current task execution node, and determine whether there is target node configuration information matching the first task configuration information in the current node configuration information; if there is target node configuration information matching the first task configuration information in the current node configuration information, then the current task execution node corresponding to the target node configuration information will be used as the target task execution node corresponding to the task to be executed.
[0164] In an exemplary embodiment of the present disclosure, the task execution node determination module 1330 can also be configured as follows: if there is no target node configuration information matching the first task configuration information in the current node configuration information, then generating first failure prompt information corresponding to the target task execution node, and sending the first failure prompt information to the terminal device; receiving a second task execution instruction sent by the terminal device after dynamically adjusting the first task configuration information in the first task execution instruction according to the first failure prompt information, and matching the target task execution node corresponding to the task to be executed from the current task execution node according to the second task configuration information included in the second task execution instruction.
[0165] In an exemplary embodiment of the present disclosure, the task execution node determination module 1330 can also be configured to: calculate the network configuration adaptability and / or resource configuration adaptability between the task to be executed and the current task execution node based on the first task configuration information and the current node configuration information; and match the target task execution node corresponding to the task to be executed from the current task execution node based on the network configuration adaptability and / or resource configuration adaptability.
[0166] In an exemplary embodiment of the present disclosure, the task execution node determination module 1330 can also be configured to: sort the current task execution nodes according to the network configuration adaptability to obtain a first node sorting result, and select the current task execution node with the largest network adaptability from the first node sorting result as the target task execution node corresponding to the task to be executed; or sort the current task execution nodes according to the resource configuration adaptability to obtain a second node sorting result, and select the current task execution node with the largest resource adaptability from the second node sorting result as the target task execution node corresponding to the task to be executed; or calculate the total adaptability between the task to be executed and the current task execution node according to the network configuration adaptability and the resource configuration adaptability, and sort the current task execution nodes according to the total adaptability to obtain a third node sorting result; select the current task execution node with the largest resource adaptability from the third node sorting result as the target task execution node corresponding to the task to be executed.
[0167] In an exemplary embodiment of the present disclosure, the task execution node determination module 1330 can also be configured to: configure a first weight value and a second weight value for the network configuration adaptability and the resource configuration adaptability; wherein the first weight value is greater than or equal to the second weight value; calculate a first product operation result between the first weight value and the network configuration adaptability, and calculate a second product operation result between the second weight value and the resource configuration adaptability; calculate a first sum operation result between the first product operation result and the second product operation result, and obtain the total adaptability between the task to be executed and the current task execution node based on the first sum operation result.
[0168] In an exemplary embodiment of the present disclosure, the first task configuration information includes first network configuration information and / or first resource configuration information; the current node configuration information includes second network configuration information and / or second resource configuration information; wherein, the task execution node determination module 1330 can also be configured to: calculate the network configuration adaptability between the task to be executed and the current task execution node based on the first network configuration information and the second network configuration information; and / or calculate the resource configuration adaptability between the task to be executed and the current task execution node based on the first resource configuration information and the second resource configuration information.
[0169] In an exemplary embodiment of the present disclosure, the first network configuration information includes a first network bandwidth and / or a first network delay, and the second network configuration information includes a second network bandwidth and / or a second network delay; wherein the task execution node determination module 1330 can also be configured to: calculate a first quotient operation result between the first network bandwidth and the second network bandwidth; and / or calculate a second quotient operation result between the first network delay and the second network delay; and obtain the network configuration adaptability between the task to be executed and the current task execution node based on the first quotient operation result and / or the second quotient operation result.
[0170] In an exemplary embodiment of the present disclosure, the first resource configuration information includes one or more of the resource configuration of the first central processing unit, the resource configuration of the first memory, and the resource configuration of the first graphics processor; the second resource configuration information includes one or more of the resource configuration of the second central processing unit, the resource configuration of the second memory, and the resource configuration of the second graphics processor; wherein, the task execution node determination module 1330 can also be configured to: calculate the third quotient operation result between the resource configuration of the second central processing unit and the resource configuration of the first central processing unit; and / or calculate the fourth quotient operation result between the resource configuration of the second memory and the resource configuration of the first memory; and / or calculate the fifth quotient operation result between the resource configuration of the second graphics processor and the resource configuration of the first graphics processor; based on the third quotient operation result and / or the fourth quotient operation result and / or the fifth quotient operation result, obtain the resource configuration adaptability between the task to be executed and the current task execution node.
[0171] In an exemplary embodiment of the present disclosure, the task execution module 1340 can also be configured to: send the scheduling task to the target task execution node, and submit the container task image included in the scheduling task to the task execution tool in the container orchestration engine through the target task execution node; execute the container task image through the task execution tool to obtain the task execution result corresponding to the task to be executed.
[0172] In an exemplary embodiment of the present disclosure, the task execution module 1340 can also be configured to: execute the container task image through the task execution tool to obtain a task execution container; start and execute the executable image included in the task execution container to obtain a task execution result corresponding to the task to be executed.
[0173] In an exemplary embodiment of the present disclosure, the task processing device further includes:
[0174] An operation status monitoring module may be used to monitor the first operation status of the task execution container and the second operation status of the task execution tool in real time during the operation of the executable image, and generate a monitoring log based on the first operation status and the second operation status; and / or
[0175] The operating status display module can be used to display the first operating status and the second operating status.
[0176] In an exemplary embodiment of the present disclosure, the task processing device further includes:
[0177] A resource release module may be configured to delete the executable image in the task execution container and release the second network occupied resources and the second memory occupied resources of the target task execution node occupied by executing the scheduled task when it is determined that the execution of the executable image is completed;
[0178] The node placement module can be used to update the node state of the target task execution node from an occupied state to an idle state, and place the target task execution node in the idle state into a resource pool.
[0179] In an exemplary embodiment of the present disclosure, the task processing device further includes:
[0180] The task execution result feedback module can be used to call the task callback interface corresponding to the terminal device, and feed back the task execution result to the terminal device through the task callback interface.
[0181] In an exemplary embodiment of the present disclosure, the first task execution instruction is generated by a terminal device in the following manner: displaying a task execution interface on a display interface of the terminal device; determining a task to be executed in response to a first input event to the task execution interface, and displaying a task configuration interface corresponding to the task to be executed; wherein the task to be executed includes a model training task and / or a data analysis task; determining first task configuration information of the task to be executed in response to a second input event acting on the task configuration interface; generating a first task execution instruction based on the task to be executed and the first task configuration information in response to a third input event acting on the task configuration interface, and sending the first task execution instruction to the task execution framework.
[0182] The specific details of each module in the above-mentioned task processing device have been described in detail in the corresponding task processing method, so they will not be repeated here.
[0183] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.
[0184] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0185] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.
[0186] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."
[0187] An electronic device 1400 according to this embodiment of the present disclosure is described below with reference to FIG 14. The electronic device 1400 shown in FIG 14 is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.
[0188] As shown in FIG14 , electronic device 1400 is implemented as a general-purpose computing device. Components of electronic device 1400 may include, but are not limited to, the aforementioned at least one processing unit 1410, the aforementioned at least one storage unit 1420, a bus 1430 connecting various system components (including storage unit 1420 and processing unit 1410), and a display unit 1440.
[0189] Wherein, the storage unit stores a program code, and the program code can be executed by the processing unit 1410, so that the processing unit 1410 performs the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1410 can execute step S110 as shown in Figure 1: receiving a first task execution instruction sent by a terminal device; wherein the first task execution instruction is generated by the terminal device in response to an input event for a task configuration interface; step S120: parsing the first task execution instruction to obtain a task to be executed and first task configuration information corresponding to the task to be executed; step S130: matching a target task execution node corresponding to the task to be executed from the current task execution node according to the first task configuration information; step S140: generating a scheduling task according to the first task configuration information and the target task execution node, and executing the scheduling task in the target task execution node to obtain a task execution result corresponding to the task to be executed.
[0190] The storage unit 1420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 14201 and / or a cache memory unit 14202 , and may further include a read-only memory unit (ROM) 14203 .
[0191] The storage unit 1420 may also include a program / utility 14204 having a set (at least one) of program modules 14205, such program modules 14205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0192] The bus 1430 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0193] Electronic device 1400 can also communicate with one or more external devices 1500 (e.g., a keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1400, and / or any device that enables electronic device 1400 to communicate with one or more other computing devices (e.g., a router, modem, etc.). Such communication can occur via input / output (I / O) interface 1450. Furthermore, electronic device 1400 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via network adapter 1460. As shown, network adapter 1460 communicates with other modules of electronic device 1400 via bus 1430. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with electronic device 1400, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0194] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
[0195] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the aforementioned methods of this specification. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present disclosure.
[0196] According to an embodiment of the present disclosure, a program product for implementing the above-mentioned method can be a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0197] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0198] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0199] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0200] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0201] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.
[0202] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not invented herein. The specification and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.
Claims
1. A task processing method, characterized in that: include: Receiving a first task execution instruction sent by a terminal device; wherein the first task execution instruction is generated by the terminal device in response to an input event for a task configuration interface; Parsing the first task execution instruction to obtain a task to be executed and first task configuration information corresponding to the task to be executed; According to the first task configuration information, matching a target task execution node corresponding to the task to be executed from the current task execution node; A scheduling task is generated according to the first task configuration information and the target task execution node, and the scheduling task is executed in the target task execution node to obtain a task execution result corresponding to the task to be executed.
2. The task processing method according to claim 1, characterized in that: According to the first task configuration information, matching a target task execution node corresponding to the task to be executed from the current task execution node includes: Acquire current task type information included in the first task configuration information, and match a first task execution node that executes a historically executed task that is consistent with the current task type information from a task scheduling template; Acquire a current task execution node in an idle state from a resource pool, and determine whether the first task execution node exists in the current task execution node; If the first task execution node exists in the current task execution node, taking the first task execution node as the target task execution node corresponding to the task to be executed; If the first task execution node does not exist in the current task execution node, a target task execution node corresponding to the task to be executed is matched from the current task execution node according to the first task configuration information.
3. The task processing method according to claim 2, characterized in that: According to the first task configuration information, matching a target task execution node corresponding to the task to be executed from the current task execution node includes: Obtaining current node configuration information of the current task execution node, and determining whether there is target node configuration information matching the first task configuration information in the current node configuration information; If there is target node configuration information matching the first task configuration information in the current node configuration information, the current task execution node corresponding to the target node configuration information is used as the target task execution node corresponding to the task to be executed.
4. The task processing method according to claim 3, characterized in that: The task processing method further includes: If the target node configuration information matching the first task configuration information does not exist in the current node configuration information, a first failure prompt information corresponding to the target task execution node is generated, and the first failure prompt information is sent to the terminal node. Terminal equipment; Receive the second task execution instruction sent by the terminal device after dynamically adjusting the first task configuration information in the first task execution instruction according to the first failure prompt information, and match the target task execution node corresponding to the task to be executed from the current task execution node according to the second task configuration information included in the second task execution instruction.
5. The task processing method according to claim 2, characterized in that: According to the first task configuration information, matching a target task execution node corresponding to the task to be executed from the current task execution node includes: Calculating the network configuration adaptability and / or resource configuration adaptability between the task to be executed and the current task execution node according to the first task configuration information and the current node configuration information; According to the network configuration adaptability and / or resource configuration adaptability, a target task execution node corresponding to the task to be executed is matched from the current task execution node.
6. The task processing method according to claim 5, characterized in that: According to the network configuration adaptability and / or resource configuration adaptability, matching a target task execution node corresponding to the task to be executed from the current task execution node includes: Sort the current task execution nodes according to the network configuration adaptability to obtain a first node sorting result, and select the current task execution node with the largest network adaptability from the first node sorting result as the target task execution node corresponding to the task to be executed; or Sorting the current task execution nodes according to the resource configuration adaptability to obtain a second node sorting result, and selecting the current task execution node with the largest resource adaptability from the second node sorting result as the target task execution node corresponding to the task to be executed; or Calculating the total fitness between the task to be executed and the current task execution node according to the network configuration fitness and the resource configuration fitness, and sorting the current task execution nodes according to the total fitness to obtain a third node sorting result; A current task execution node with the largest resource adaptability is selected from the third node sorting result as the target task execution node corresponding to the task to be executed.
7. The task processing method according to claim 6, characterized in that: Calculating the total fitness between the task to be executed and the current task execution node according to the network configuration fitness and the resource configuration fitness, including: Configuring a first weight value and a second weight value for the network configuration adaptability and the resource configuration adaptability; wherein the first weight value is greater than or equal to the second weight value; Calculate the first product operation result between the first weight value and the network configuration adaptability, and calculate the second weight value and the resource The result of the second product operation between the configuration fitness degrees; A first sum operation result between the first product operation result and the second product operation result is calculated, and a total fitness between the task to be executed and the current task execution node is obtained according to the first sum operation result.
8. The task processing method according to claim 5, characterized in that: The first task configuration information includes first network configuration information and / or first resource configuration information; the current node configuration information includes second network configuration information and / or second resource configuration information; Wherein, calculating the network configuration adaptability and / or resource configuration adaptability between the task to be executed and the current task execution node according to the first task configuration information and the current node configuration information includes: Calculating the network configuration compatibility between the task to be executed and the current task execution node according to the first network configuration information and the second network configuration information; and / or The resource configuration compatibility between the task to be executed and the current task execution node is calculated according to the first resource configuration information and the second resource configuration information.
9. The task processing method according to claim 8, characterized in that: The first network configuration information includes a first network bandwidth and / or a first network delay, and the second network configuration information includes a second network bandwidth and / or a second network delay; The step of calculating the network configuration compatibility between the task to be executed and the current task execution node according to the first network configuration information and the second network configuration information includes: Calculating a first quotient operation result between the first network bandwidth and the second network bandwidth; and / or Calculating a second quotient operation result between the first network delay and the second network delay; The network configuration adaptability between the task to be executed and the current task execution node is obtained according to the first quotient calculation result and / or the second quotient calculation result.
10. The task processing method according to claim 8, characterized in that: The first resource configuration information includes one or more of a resource configuration of a first central processing unit, a resource configuration of a first memory, and a resource configuration of a first graphics processing unit; The second resource configuration information includes one or more of a resource configuration of a second central processing unit, a resource configuration of a second memory, and a resource configuration of a second graphics processing unit; The step of calculating the resource configuration adaptability between the task to be executed and the current task execution node according to the first resource configuration information and the second resource configuration information includes: Calculating a third quotient operation result between the resource configuration of the second central processing unit and the resource configuration of the first central processing unit; and / or Calculating a fourth quotient operation result between the resource configuration of the second memory and the resource configuration of the first memory; and / or Calculating a fifth quotient operation result between the resource configuration of the second graphics processor and the resource configuration of the first graphics processor; According to the third quotient calculation result and / or the fourth quotient calculation result and / or the fifth quotient calculation result, the resource configuration adaptability between the task to be executed and the current task execution node is obtained.
11. The task processing method according to claim 1, characterized in that: Executing the scheduled task in the target task execution node to obtain a task execution result corresponding to the task to be executed includes: Sending the scheduling task to the target task execution node, and submitting the container task image included in the scheduling task to the task execution tool in the container orchestration engine through the target task execution node; The container task image is executed by the task execution tool to obtain a task execution result corresponding to the task to be executed.
12. The task processing method according to claim 11, characterized in that: Executing the container task image through the task execution tool to obtain a task execution result corresponding to the task to be executed includes: Executing the container task image by the task execution tool to obtain a task execution container; The executable image included in the task execution container is started and executed to obtain a task execution result corresponding to the task to be executed.
13. The task processing method according to claim 12, characterized in that: The task processing method further includes: During the operation of the executable image, a first operating state of the task execution container and a second operating state of the task execution tool are monitored in real time, and a monitoring log is generated according to the first operating state and the second operating state; and / or The first operating state and the second operating state are displayed.
14. The task processing method according to claim 12, characterized in that: The task processing method further includes: When it is determined that the execution of the executable image is completed, deleting the executable image in the task execution container, and releasing the second network occupied resources and the second memory occupied resources of the target task execution node occupied by executing the scheduled task; The node state of the target task execution node is updated from an occupied state to an idle state, and the target task execution node in the idle state is placed in a resource pool.
15. The task processing method according to claim 1, characterized in that: The task processing method further includes: The task callback interface corresponding to the terminal device is called, and the task execution result is fed back to the terminal device through the task callback interface.
16. The task processing method according to claim 1, characterized in that: The first task execution instruction is generated by the terminal device in the following manner: Displaying a task execution interface on the display interface of the terminal device; In response to a first input event to a task execution interface, determining a task to be executed, and displaying a task configuration interface corresponding to the task to be executed; wherein the task to be executed includes a model training task and / or a data analysis task; In response to a second input event acting on the task configuration interface, determining first task configuration information of the task to be executed; In response to a third input event acting on the task configuration interface, a first task execution instruction is generated according to the task to be executed and the first task configuration information, and the first task execution instruction is sent to the task execution framework.
17. A task processing device, characterized in that: include: A first receiving module, configured to receive a first task execution instruction sent by a terminal device; wherein the first task execution instruction is generated by the terminal device in response to an input event to a task configuration interface; A first parsing module, used to parse the first task execution instruction to obtain a task to be executed and first task configuration information corresponding to the task to be executed; A task execution node determination module, used to match a target task execution node corresponding to the task to be executed from the current task execution nodes according to the first task configuration information; The task execution module is used to generate a scheduling task according to the first task configuration information and a target task execution node, and execute the scheduling task in the target task execution node to obtain a task execution result corresponding to the task to be executed.
18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the task processing method according to any one of claims 1 to 16 is implemented.
19. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the task processing method according to any one of claims 1 to 16 by executing the executable instructions.
Citation Information
Patent Citations
Distributed application intelligent deployment method and system in cloud platform
CN103810023A
Distributed task scheduling method, system and device and medium
CN109656706A
Hadoop cluster task scheduling method based on positive and negative feedback load scheduling algorithm
CN113934525A
Calculation task scheduling method and device, electronic equipment and storage medium
CN115080197A
Task processing method and device, computer readable storage medium and electronic equipment
CN116700919A
Cited By
Distributed task scheduling execution method and device, computer equipment and storage medium
CN119781927A
Distributed task scheduling execution method and device, computer device, and storage medium
CN119781927B
Node sorting method and device based on multi-dimensional data and medium
CN120849042A
