Distributed task scheduling method and device and storage medium

Abstract

The invention discloses a distributed task scheduling method and device and a storage medium, and relates to the technical field of distributed computation.The distributed task scheduling method comprises the steps that an obtained model task is decomposed into a plurality of sub-tasks, and the dependency relationship between the sub-tasks is determined; according to the dependency relationship and the available resource information of each container node, distributing each subtask to each container node; determining a configuration strategy of each container node according to the sub-task allocated to each container node; and scheduling each container node according to the dependency relationship and the configuration strategy. According to the method, the resource required by the container node is determined according to the sub-task allocated to the container node, so that unnecessary resource waste is avoided, and the resource utilization rate is improved.

Description

Technical Field

[0001] The present application relates to the field of distributed computing technology, and in particular to a distributed task scheduling method, device and storage medium. Background Art

[0002] With the rapid development of information technology, cross-node distributed task scheduling technology has become an important part of modern computing. Early distributed computing systems achieved the initial sharing and utilization of computing resources through simple task allocation and node management. With the rise of containerization technology, the deployment and execution of AI (Artificial Intelligence) tasks in distributed environments have been promoted.

[0003] Currently, container management platforms such as Docker and Kubernetes provide basic scheduling capabilities, which can allocate tasks to different container nodes according to resource requirements and constraints. However, these platforms mainly focus on the deployment, expansion and management of containers. In terms of distributed task scheduling, these platforms only provide basic scheduling capabilities, which allocate tasks to different nodes according to resource requirements and constraints, and provide sufficient resources for container nodes to successfully execute tasks. This will cause a waste of resources to a certain extent, resulting in low resource utilization.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a distributed task scheduling method, device and storage medium, aiming to improve resource utilization.

[0006] To achieve the above objectives, the present application proposes a distributed task scheduling method, the method comprising: Decomposing the acquired model task into multiple subtasks, and determining the dependency relationship between the subtasks; Allocate each of the subtasks to each of the container nodes according to the dependency relationship and available resource information of each container node; Determining a configuration strategy for each of the container nodes according to the subtasks assigned to each of the container nodes; The container nodes are scheduled according to the dependency relationship and the configuration strategy.

[0007] In one embodiment, the step of allocating each of the subtasks to each of the container nodes according to the dependency relationship and the available resource information of each of the container nodes includes: Determine at least one candidate solution according to the dependency relationship and the available resource information, wherein the candidate solution is used for allocating each of the subtasks; Determine a target solution from among the candidate solutions according to the resource utilization rate and task completion time of each candidate solution; According to the target solution, each of the subtasks is allocated to each of the container nodes.

[0008] In one embodiment, the configuration strategy includes an isolation environment configuration, and after the step of determining the configuration strategy of each container node, the step further includes: According to the isolation environment configuration and the container configuration of each container node, the resource space of each container node is adjusted so that the resource space of each container node has no overlap.

[0009] In one embodiment, the configuration strategy includes a resource allocation strategy, and after the step of determining the configuration strategy of each container node, the step further includes: The resource quota of each container node is determined according to the resource allocation strategy and the resource usage requirement of each container node.

[0010] In one embodiment, the configuration policy includes a security policy, and after the step of determining the configuration policy of each container node, the step further includes: The access rights of each of the container nodes are determined according to the security policy and the security requirements of each of the container nodes.

[0011] In one embodiment, the configuration strategy includes network requirements, and after the step of determining the configuration strategy of each container node, the step further includes: According to the network requirements and the network isolation rules of each container node, a communication interface is configured for each container node.

[0012] In one embodiment, the configuration strategy includes encryption requirements, and after the step of scheduling each of the container nodes, further includes: Encrypting the task data of the subtask according to the encryption requirement and a preset encryption key; According to a preset key distribution principle, a decryption key corresponding to the encryption key is allocated to the subtask.

[0013] In addition, to achieve the above purpose, the present application also proposes a distributed task scheduling device, the distributed task scheduling device comprising: A decomposition module is used to decompose the acquired model task into multiple subtasks and determine the dependency relationship between the subtasks; An allocation module, configured to allocate each of the subtasks to each of the container nodes according to the dependency relationship and available resource information of each container node; A configuration module, used to determine a configuration strategy for each of the container nodes according to the subtasks assigned to each of the container nodes; A scheduling module is used to schedule each of the container nodes according to the dependency relationship and the configuration strategy.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a distributed task scheduling device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the distributed task scheduling method described above.

[0015] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the distributed task scheduling method described above are implemented.

[0016] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the distributed task scheduling method described above are implemented.

[0017] One or more technical solutions proposed in the present application have at least the following technical effects: first, the acquired model task is decomposed into multiple subtasks, the dependency relationship between the subtasks is determined, and the subtasks that can be executed in parallel are clarified, which is conducive to reducing the overall execution time of all subtasks; then, according to the dependency relationship and the available resource information of each container node, each subtask is allocated to each container node to ensure that the subtask can be executed normally on the allocated container node without insufficient resources. At the same time, the subtask also consumes the resources on the container node to ensure that the resources are effectively utilized and avoid resource waste; then, according to the subtasks allocated to each container node, Task, determine the configuration strategy of each container node. The configuration strategy refers to the requirements that the container node needs to meet when it is scheduled or executed, such as resource requirements, security requirements, and isolation configuration. Limit each container node according to the requirements of the assigned subtasks to ensure the minimum usage principle, that is, each container node only obtains the resources and permissions required to complete the task, etc., to avoid resource waste; finally, schedule each container node and execute each subtask in the order corresponding to the dependency relationship. During the execution process, limit the allocation and use of resources according to the configuration strategy. On the basis of being able to complete the execution of the entire model task, minimize resource waste and improve resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the distributed task scheduling method of the present application; Figure 2 A schematic diagram of the dependency relationship between subtasks provided in Example 1 of the present application; Figure 3 A schematic diagram of the module structure of the distributed task scheduling method provided in Example 2 of the present application; Figure 4 This is a schematic diagram of the module structure of the distributed task scheduling device according to an embodiment of the present application; Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the distributed task scheduling method in the embodiment of the present application. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] In this embodiment, for the convenience of description, the following description is made with the terminal as the execution subject.

[0024] Conventional technologies are difficult to effectively optimize resource utilization in the scheduling and security of distributed AI tasks, especially for the computing requirements of AI tasks; and the current security isolation mechanism mainly relies on namespaces and control groups, lacks deep integration with higher-level security mechanisms, and is difficult to ensure data security when facing scenarios with higher security requirements. In terms of auditing and monitoring, conventional technologies are mainly focused on the single-node level, and are unable to fully monitor and audit the execution of distributed tasks. They lack a unified log management and event backtracking mechanism, making it difficult to effectively track and analyze security incidents after they occur, thus affecting the reliability and security of distributed task scheduling and execution.

[0025] The present application provides a solution, which decomposes the acquired model task into multiple subtasks, determines the dependency relationship between the subtasks, and clarifies the subtasks that can be executed in parallel, which is conducive to reducing the overall execution time of all subtasks; then, according to the dependency relationship and the available resource information of each container node, multiple candidate schemes for allocating each subtask to each container node are determined, and the target scheme for allocation can be determined according to the resource utilization rate and task completion time of each candidate scheme, such as selecting the scheme with the highest resource utilization rate and the shortest task completion time to complete the allocation of subtasks, and further improve the resource utilization rate and task execution efficiency; then, according to the subtasks allocated to each container node, the configuration strategy of each container node can be determined, and the configuration strategy can include security strategy and encryption requirements, etc., and then the security of task execution can be improved by reasonably setting resource quotas, determining access rights and encrypting task data; finally, each container node is scheduled to execute each subtask in the order corresponding to the dependency relationship. During the execution process, the allocation and use of resources are restricted according to the configuration strategy. On the basis of being able to complete the execution of the entire model task, resource waste is reduced as much as possible and resource utilization is improved.

[0026] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes the terminal as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0027] Based on this, the embodiment of the present application provides a distributed task scheduling method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the distributed task scheduling method of the present application.

[0028] In this embodiment, the distributed task scheduling method includes steps S10 to S40: Step S10, decomposing the acquired model task into multiple subtasks, and determining the dependency relationship between the subtasks; It should be noted that a model task refers to a computing task completed using an artificial intelligence model, which obtains the expected output or implements a specific function through training or reasoning processes; a subtask refers to the decomposition result of a model task, which is responsible for processing a smaller and more specific function in the model task.

[0029] In addition, it should be noted that the dependency relationship refers to the execution order and conditions between subtasks, that is, some subtasks may need to be executed after other subtasks are completed, or require the output results of other subtasks as input.

[0030] For example, the task of identifying the type of objects in an image (model task) can be decomposed into at least three subtasks: image preprocessing, feature extraction, and classification. Furthermore, image preprocessing can be decomposed into subtasks such as noise removal and image contrast enhancement. Since the output of image preprocessing is the input of feature extraction, and the output of feature extraction is the input of classification, the dependency relationship between the subtasks can be determined as feature extraction depends on image preprocessing, and classification depends on feature extraction.

[0031] For example, the model task can be decomposed according to the model structure. For the object type recognition task (model task) in the image, the model task can be decomposed into different subtasks according to the model structure, such as convolution layer, pooling layer and fully connected layer, including: input layer, two convolution layers and fully connected layer, where the two convolution layers are used for feature extraction and object detection respectively; and for the convolution layer subtask, different subtasks can be further decomposed according to different filters. Then construct DAG (Directed Acyclic Graph), please refer to Figure 2 , Figure 2 A dependency diagram of the model tasks is provided. First, subtask 101 is performed, image preprocessing of the input layer is used to provide preprocessed images for subsequent tasks; subtask 102 feature extraction and subtask 103 object detection can be executed in parallel because feature extraction and object detection can be performed independently; subtask 104 classification requires features and detection results for classification, so subtask 104 depends on subtasks 102 and 103.

[0032] It can be understood that by decomposing the model task into subtasks and determining the dependencies, subtasks that do not have dependencies can be executed in parallel, thereby reducing the overall execution time; at the same time, clarifying the dependencies between subtasks helps to reasonably allocate computing resources and avoid resource waste.

[0033] Step S20, allocating each subtask to each container node according to the dependency relationship and available resource information of each container node; It should be noted that a container node refers to a physical or virtual server that runs a containerized application. Each container node can have specific computing resources and storage capabilities, etc.; available resource information refers to the computing resources, storage resources, and network resources available on the container node, such as memory, CPU (Central Processing Unit), network bandwidth, etc.

[0034] When a large model task is received, it is split into multiple subtasks and the dependencies of each subtask are obtained. Then, the current resource availability information of each container node is queried, including CPU usage, content occupancy, etc., and the best allocation plan is selected according to the preset strategy (such as giving priority to idle nodes). Finally, the subtasks are allocated to the corresponding container nodes according to the allocation plan, and the relevant nodes are notified to start executing the tasks according to the dependencies.

[0035] Exemplarily, for an image object detection task (model task) that can be decomposed into subtasks such as image preprocessing and feature extraction, the dependency relationship between the subtasks is obtained as the feature extraction subtask depends on the image preprocessing subtask. The image preprocessing subtask is assigned to container node 1 with abundant CPU resources, and the feature extraction subtask is assigned to container node 2 with sufficient memory. The output of container node 1 is set as the input of container node 2 according to the dependency relationship.

[0036] It can be understood that, according to the dependency relationship, reasonable setting of parallel nodes can effectively shorten the task completion time; reasonable allocation of subtasks according to resource information can ensure that the resources on the container node are effectively utilized and avoid resource waste.

[0037] Step S30: determining a configuration strategy for each container node according to the subtasks assigned to each container node.

[0038] It should be noted that the configuration policy refers to the requirements that need to be met when the container node is scheduled or executed, including resource requirements, security requirements, and isolation configuration.

[0039] Exemplarily, information about one or more subtasks that each container node needs to execute is obtained, including the subtask type, complexity, and expected load; then, based on this information, the resource requirements of each container node are determined, and configuration strategies such as resource allocation, load balancing, and security settings for each node are automatically or manually determined to ensure that each container node can efficiently process the assigned subtasks.

[0040] Step S40: Schedule each container node according to the dependency relationship and configuration strategy.

[0041] It should be noted that scheduling refers to the process of determining the order of task execution and arranging the corresponding container nodes to execute tasks based on task priority, dependencies, and resource requirements.

[0042] Exemplarily, the dependency relationship between image preprocessing and feature extraction is that the input of feature extraction is the output of image preprocessing, and the configuration strategy of the container node where the image preprocessing subtask is located includes encrypting the data obtained after executing the corresponding subtask. Therefore, the corresponding container node is first arranged to execute the image preprocessing subtask, and the execution result is encrypted and sent to the container node where the feature extraction subtask is located.

[0043] This embodiment provides a distributed task scheduling method, which can process these subtasks in parallel by decomposing a model task into multiple subtasks, thereby reducing the task completion time; intelligently allocate subtasks to container nodes based on dependencies and available resources of the container, thereby improving the effective utilization of resources; at the same time, adjust the relevant configuration strategy of the container according to the assigned subtasks, limit the resources of the container according to actual needs, etc., to further improve resource utilization and reduce resource waste.

[0044] In a feasible implementation manner, step S20 includes: Step S21, determining at least one candidate solution according to the dependency relationship and available resource information, wherein the candidate solution is used for the allocation of each subtask; It should be noted that the candidate solution refers to multiple possible subtask allocation solutions generated by algorithms or manual decisions based on dependencies and available resources, which determine which subtasks should be assigned to which container nodes for execution.

[0045] When only one candidate solution is generated, the subtasks are directly assigned to the container nodes according to the candidate solution.

[0046] Exemplarily, there are four subtasks A, B, C, and D, whose dependencies are B depends on A, C depends on A, and D depends on B and C. If resource 1 or 2 is required to complete A, resource 3, 4, or 5 is required to complete B or C, and resource 6 is required to complete D, candidate solution 1 can be generated: container node 1 uses resource 1 to complete A, container node 2 uses resource 3 to complete B and C, and container node 3 uses resource 6 to complete D; candidate solution 2 can also be generated: container node 1 uses resource 1 to complete A, container node 2 uses resource 3 to complete B, container node 4 uses resource 4 to complete C, and container node 3 uses resource 6 to complete D; more candidate solutions can be generated in actual situations.

[0047] Step S22, determining a target solution from each candidate solution according to the resource utilization rate and task completion time of each candidate solution; It should be noted that resource utilization refers to the proportion of container node resources (such as CPU, memory, etc.) used under a specific plan; task completion time refers to the total time required for the model task to be executed from start to finish, that is, the total time required for all subtasks to be completed.

[0048] When the dependencies are met, the candidate solution with the highest resource utilization and the shortest task completion time is selected as the target solution. If both cannot be met at the same time, the priority of task completion time and resource utilization is determined according to the weight parameters of the scheduling strategy to determine the target solution. Of course, a solution that is relatively balanced between the two can also be selected, and this embodiment does not specifically limit this.

[0049] It can be understood that by evaluating the performance indicators (resource utilization and task completion time) of each candidate solution, the optimal subtask allocation solution is selected, thereby achieving efficient resource utilization and rapid task completion.

[0050] Step S23: Allocate each subtask to each container node according to the target solution.

[0051] For example, for a machine learning model training task, according to the target plan, the data preprocessing subtask is assigned to a container node with high I / O (Input / Output) performance, the model training subtask is assigned to a container node with a high-performance GPU, and the result verification subtask is assigned to a container node with sufficient memory to ensure that each subtask can be executed normally.

[0052] In this implementation, accurate subtask allocation ensures that each subtask can be executed on a suitable container node, which can fully utilize the resources of each container node and improve resource utilization and task execution efficiency.

[0053] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, the configuration strategy includes the isolation environment configuration, and after step S30, it also includes: Step A10: adjusting the resource space of each container node according to the isolation environment configuration and the container configuration of each container node, so that the resource space of each container node has no overlap.

[0054] It should be noted that the isolated environment configuration refers to the parameter requirements of the independent operating environment determined according to the subtasks assigned to the container node, including resource usage, process space, and file system. The container configuration of the container node refers to the parameter requirements of the operating environment determined according to the hardware or software characteristics of the container node, including the resources, environment variables, and storage mounts required for container operation.

[0055] In addition, it should be noted that resource space refers to the sum of computing resources, storage resources, network resources, etc. allocated to container nodes; no overlap means that in resource allocation, the resource spaces of different containers have no intersection, ensuring the independence and security of resources.

[0056] For example, the namespace mechanism can be used to ensure the isolation of task processes, file systems, networks, etc. in a containerized environment. For example, the process namespace is used to ensure that the processes within the container are only visible within the namespace and cannot be discovered by other containers or the host machine; the network namespace is used to assign an independent network interface to each container to ensure that containers cannot communicate directly; the user namespace is used to ensure that the user permissions within the container are isolated from the user permissions of the host machine.

[0057] For example, for each subtask ti, the following isolation conditions need to be ensured:

[0058] Among them, Ci and Cj represent the resource space of container i and container j respectively, ensuring that there is no overlap between the processes, file systems, and network spaces of different container nodes.

[0059] In this implementation, by ensuring that there is no overlap in resource space, the independence of containers is guaranteed, avoiding system instability due to resource competition, and helping to prevent data leakage and malicious attacks between containers, thereby improving the security of task execution.

[0060] In a feasible implementation manner, the configuration strategy includes a resource allocation strategy, and after step S30, further includes: Step A20: Determine the resource quota of each container node according to the resource allocation strategy and the resource usage requirement of each container node.

[0061] It should be noted that the resource allocation strategy refers to the demand of container nodes for CPU, memory, I / O and other resources determined by the subtasks assigned to the container nodes. The resource usage requirements of container nodes refer to the usage scope of related resources determined by system configuration or business requirements. For example, the CPU usage of each container can be set to no more than 60%.

[0062] In addition, it should be noted that the resource quota refers to the upper limit of resources allocated to each container node, ensuring that the node can use resources without exceeding this limit.

[0063] It should be noted that the sum of the resource usage of the subtasks allocated to each container node cannot exceed the resource capacity of the container node.

[0064] For example, each task ti requires resources r(ti), which can include multiple types of resources such as r_cpu(ti), r_mem(ti), r_io(ti), etc. To meet the resource requirements of all tasks, resources can be evenly allocated so that each task can be optimally completed with given resources:

[0065] Among them, Minimize means minimization, and N represents the task set assigned to the current container node. The purpose of this formula is to minimize the maximum value of resources required to complete the task and ensure load balancing.

[0066] In this implementation, by reasonably setting resource quotas, resource utilization efficiency is improved and waste is reduced.

[0067] In a feasible implementation manner, the configuration strategy includes security configuration, and after step S30, further includes: Step A30: Determine the access rights of each container node according to the security policy and the security requirements of each container node.

[0068] It should be noted that the security policy refers to the security access rules of the container node, which are determined according to the security requirements of the subtasks assigned to the container node, including access control, identity authentication, etc. The security requirements of the container node refer to the security conditions that the container node set by the system needs to meet during operation, such as file system access rights, system call permissions, etc.

[0069] It should be noted that in the case of a conflict between the security policy and the security requirements of the container, the stricter security policy will usually be implemented first. Administrators can also manually intervene and adjust the security policy or the security requirements of the container through log analysis and alert notifications.

[0070] For example, a mandatory access control mechanism can be used to ensure that tasks in a container can only access authorized resources, preventing unauthorized operations and security vulnerabilities. Assuming that task ti needs to access specific files and system resources, assign access rights Pi to it so that access rights meet the principle of minimization: R Among them, Pi is the permission set of task ti, and R is the access permission of all resources. Each task can only access the resources defined in Pi to prevent over-authorization of tasks.

[0071] In this implementation, access rights are determined based on security policies and security requirements to ensure that only authorized users or system entities can access container nodes and their resources, protect sensitive data, and thus improve the security of task execution.

[0072] In a feasible implementation manner, the configuration strategy includes network requirements, and after step S30, further includes: Step A40: configure a communication interface for each container node according to network requirements and network isolation rules of each container node.

[0073] It should be noted that network requirements refer to the requirements for data exchange between container nodes or between container nodes and external networks, which are determined according to the assigned subtasks, including data transmission rate, bandwidth, and security policies. The network isolation rules of container nodes refer to the network communication rules set in a containerized environment to ensure the security and independence of container nodes, including communication permissions and paths between container nodes and between container nodes and external networks.

[0074] Although network namespaces can achieve basic network isolation, independent communication interface configurations can provide more advanced network control and security features, such as encrypted communications and more fine-grained network policies. Compared with network namespaces, independent communication interface configurations are more secure in complex communication scenarios.

[0075] Exemplarily, the network requirements of the container node are obtained, including which external networks need to be accessed, which other container nodes need to communicate with, etc.; then, the network isolation rules of the container itself are obtained to determine which inter-node communications need to be allowed or restricted; then, according to the network requirements and isolation rules, a virtual network interface (communication interface) is set on the container node, and an appropriate communication protocol (such as TCP / IP), listening address and port number are selected to achieve communication with other container nodes or external networks.

[0076] In this implementation, by configuring each container node with its own communication interface, the communication data of different tasks are isolated, unauthorized access and data leakage are prevented, and the security of task execution is enhanced.

[0077] In a feasible implementation manner, the configuration strategy includes encryption requirements, and after step S30, further includes: Step A50, encrypting the task data of the subtask according to the encryption requirement and the preset encryption key; It should be noted that encryption requirements refer to specific encryption requirements set to protect the data security of subtasks, including the selection of encryption algorithms, the setting of encryption strength, etc. Task data refers to the information generated or required to be processed during the execution of subtasks.

[0078] Exemplarily, the task data di of task ti is encrypted using a symmetric encryption algorithm:

[0079] Among them, ci is the encrypted ciphertext, E represents the encryption algorithm, and k is the secret key. Since a symmetric encryption algorithm is used, k is both the encryption key and the decryption key. Through encryption, task data can be protected from unauthorized access during storage or transmission.

[0080] Step A51, according to the preset key distribution principle, allocate a decryption key corresponding to the encryption key to the subtask.

[0081] It should be noted that the key distribution principle refers to the rules for key distribution and management formulated to ensure key security. The acquisition of keys and key distribution principles can be set according to user input or automatically generated according to system configuration or encryption requirements.

[0082] For example, key distribution follows the principle of minimizing exposure, and the key set (including encryption key and decryption key) assigned to each task must meet the following requirements:

[0083] Among them, Ki and Kj represent the key sets of task ti and task tj respectively. By ensuring that each task supports the required keys, key leakage is prevented, which may cause task data to be stolen.

[0084] In this implementation, by encrypting the task data, the confidentiality and integrity of the data are enhanced; by formulating the key distribution principle, the unified management and distribution of the key can be achieved to ensure the safe execution of the task.

[0085] In a feasible implementation manner, after step S40, the method further includes: Step A60, comparing the obtained actual execution order of each subtask and the dependency relationship to see whether they are consistent; Step A61: When the actual execution order is inconsistent with the dependency relationship, output a warning message.

[0086] It should be noted that the actual execution order refers to the execution order of the subtasks in chronological order during the actual execution process.

[0087] For example, the execution order of each subtask is monitored in real time, and it is determined whether the actual execution order of the subtasks is consistent with the dependency relationship:

[0088] Among them, ti and tj represent subtasks, and the symbol Indicates a partial order relationship "priority or earlier than", L(ti) and L(tj) represent the actual execution time of subtask ti and subtask tj respectively. If subtask ti takes precedence over tj, then the actual execution time of ti should be earlier than the actual execution time of tj. When the "if-then" in the formula is not true, that is, when the actual execution order of the subtasks is inconsistent with the dependency relationship, an alarm message is output to prompt the operator to pay attention and correct the error.

[0089] In this implementation, by comparing the actual execution order and dependencies, errors in task execution can be discovered and corrected in a timely manner, ensuring that tasks are performed in the expected order and dependencies, avoiding errors or failures caused by violations of dependencies, and improving the reliability of the overall task execution process.

[0090] For example, to help understand the implementation process of the distributed task scheduling method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 3 , Figure 3 A schematic diagram of the module structure of a distributed task scheduling method is provided, specifically: The task scheduling module is responsible for decomposing the acquired model tasks into multiple subtasks, and automatically selecting suitable container nodes to deploy subtasks based on factors such as the complexity of the subtasks, data location, and resource availability. In the process of deploying subtasks, it is necessary to simultaneously satisfy the dependencies between subtasks and load balancing (maximize resource utilization and minimize task completion time); after deployment, the resource allocation strategy, security strategy and other configuration strategies of the container node are determined based on the subtasks assigned to the container node, so that other modules can adjust the relevant configurations of the container node based on the configuration strategy.

[0091] The namespace module is responsible for creating an independent isolation environment for the container based on the isolation environment configuration provided by the task scheduling module and the container configuration of the container itself, ensuring that the network, process, file system and other resources between different containers and between the container and the host are completely isolated, and ensuring that the execution environment of each subtask is independent.

[0092] The resource control module is responsible for managing the resource allocation of containers, including the use of CPU, memory, network, and disk I / O, according to the resource allocation strategy provided by the task scheduling module and the resource usage requirements of the container itself, to prevent a container from overusing resources and affecting the execution of subtasks of other containers. In addition, the resource control module will also provide the security audit module with the resource usage of each container node.

[0093] The access control module is responsible for fine-grained permission management of access to processes, files, and networks inside the container based on the security policies provided by the task scheduling module and the security requirements of the container nodes through a mandatory access control mechanism to prevent unauthorized operations or data leakage. In addition, the access control module will also report the permission usage to the security audit module, so that the security audit module can detect whether the container node has abnormal behavior according to the security policy.

[0094] The network isolation module is responsible for managing network communications between containers based on the network requirements provided by the task scheduling module and the network isolation rules of the container nodes, ensuring the isolation of communication data between different tasks. In addition, the network isolation module can also provide network access logs for the security audit module.

[0095] The data encryption module is responsible for encrypting task data according to the encryption requirements provided by the task scheduling module, ensuring that sensitive data processed by the container is always transmitted and stored in encrypted form, and ensuring key security through secure key distribution principles.

[0096] The security audit module is responsible for real-time monitoring of the running status, resource usage, and security events of containers in the distributed system, and records all operation behaviors to ensure that the audit is traceable. The security audit module can also receive audit logs provided by each module, including resource usage, network activity, file access, etc., and output an audit log containing the behavior records of all subtasks.

[0097] In addition, the task scheduling module can adjust the resource allocation strategy and security strategy in real time according to the monitoring information provided by the audit module to ensure that all subtasks can be successfully executed.

[0098] The task scheduling module is equivalent to a main program, providing information such as resource allocation strategy and security strategy; other modules are equivalent to subroutines, which perform specific tasks such as resource control, isolation and encryption based on the information provided by the main program.

[0099] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the distributed task scheduling method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0100] The present application also provides a distributed task scheduling device. Figure 4 , the distributed task scheduling device comprises: A decomposition module 10 is used to decompose the acquired model task into multiple subtasks and determine the dependency relationship between the subtasks; An allocation module 20, configured to allocate each subtask to each container node according to the dependency relationship and available resource information of each container node; A configuration module 30, configured to determine a configuration strategy for each container node according to the subtasks assigned to each container node; The scheduling module 40 is used to schedule each container node according to the dependency relationship and configuration strategy.

[0101] The distributed task scheduling device provided in the embodiment of the present application adopts the distributed task scheduling method in the above embodiment to improve resource utilization. Compared with the prior art, the beneficial effects of the distributed task scheduling device provided in the present application are the same as the beneficial effects of the distributed task scheduling method provided in the above embodiment, and other technical features in the distributed task scheduling device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0102] An embodiment of the present application provides a distributed task scheduling device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the distributed task scheduling method in the above-mentioned embodiment one.

[0103] Reference below Figure 5 , which shows a schematic diagram of the structure of a distributed task scheduling device suitable for implementing the embodiment of the present application. The distributed task scheduling device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The distributed task scheduling device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0104] like Figure 5As shown, the distributed task scheduling device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the distributed task scheduling device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the distributed task scheduling device to communicate wirelessly or wired with other devices to exchange data. Although the distributed task scheduling device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0105] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0106] The distributed task scheduling device provided in the embodiment of the present application adopts the distributed task scheduling method in the above embodiment to improve resource utilization. Compared with the prior art, the beneficial effects of the distributed task scheduling device provided in the present application are the same as the beneficial effects of the distributed task scheduling method provided in the above embodiment, and other technical features in the distributed task scheduling device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0107] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0108] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0109] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, and the computer-readable program instructions are used to execute the distributed task scheduling method in the above embodiment.

[0110] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0111] The computer-readable storage medium may be included in the distributed task scheduling device; or may exist independently without being assembled into the distributed task scheduling device.

[0112] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a distributed task scheduling device, the distributed task scheduling device: decomposes the acquired model task into multiple subtasks, and determines the dependency relationship between each subtask; allocates each subtask to each container node according to the dependency relationship and the available resource information of each container node; determines the configuration strategy of each container node according to the subtasks allocated to each container node; and schedules each container node according to the dependency relationship and the configuration strategy.

[0113] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0116] The readable storage medium provided in the embodiment of the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned distributed task scheduling method, and can improve resource utilization. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the distributed task scheduling method provided in the above-mentioned embodiment, and will not be described in detail here.

[0117] An embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned distributed task scheduling method when executed by a processor.

[0118] The computer program product provided in the embodiment of the present application can improve resource utilization. Compared with the prior art, the beneficial effects of the computer program product provided in the present application are the same as the beneficial effects of the distributed task scheduling method provided in the above embodiment, which will not be described in detail here.

[0119] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A distributed task scheduling method, characterized in that: The distributed task scheduling method comprises: Decomposing the acquired model task into multiple subtasks, and determining the dependency relationship between the subtasks; Allocate each of the subtasks to each of the container nodes according to the dependency relationship and available resource information of each container node; Determining a configuration strategy for each of the container nodes according to the subtasks assigned to each of the container nodes; The container nodes are scheduled according to the dependency relationship and the configuration strategy.

2. The distributed task scheduling method according to claim 1, characterized in that: The step of allocating each of the subtasks to each of the container nodes according to the dependency relationship and the available resource information of each container node includes: Determine at least one candidate solution according to the dependency relationship and the available resource information, wherein the candidate solution is used for allocating each of the subtasks; Determine a target solution from among the candidate solutions according to the resource utilization rate and task completion time of each candidate solution; According to the target solution, each of the subtasks is allocated to each of the container nodes.

3. The distributed task scheduling method according to claim 1, characterized in that: The configuration strategy includes an isolation environment configuration, and after the step of determining the configuration strategy of each container node, further includes: According to the isolation environment configuration and the container configuration of each container node, the resource space of each container node is adjusted so that the resource space of each container node has no overlap.

4. The distributed task scheduling method according to claim 1, characterized in that: The configuration strategy includes a resource allocation strategy, and after the step of determining the configuration strategy of each container node, further comprising: The resource quota of each container node is determined according to the resource allocation strategy and the resource usage requirement of each container node.

5. The distributed task scheduling method according to claim 1, characterized in that: The configuration policy includes a security policy, and after the step of determining the configuration policy of each container node, the step further includes: The access rights of each of the container nodes are determined according to the security policy and the security requirements of each of the container nodes.

6. The distributed task scheduling method according to claim 1, characterized in that: The configuration strategy includes network requirements, and after the step of determining the configuration strategy of each container node, further comprising: According to the network requirements and the network isolation rules of each container node, a communication interface is configured for each container node.

7. The distributed task scheduling method according to claim 1, characterized in that: The configuration strategy includes encryption requirements, and after the step of determining the configuration strategy of each container node, further comprising: Encrypting the task data of the subtask according to the encryption requirement and a preset encryption key; According to a preset key distribution principle, a decryption key corresponding to the encryption key is allocated to the subtask.

8. The distributed task scheduling method according to claim 1, characterized in that: After the step of scheduling each of the container nodes, the method further includes: Comparing the obtained actual execution order of each of the subtasks with the dependency relationship to see whether they are consistent; When the actual execution order is inconsistent with the dependency relationship, an alarm message is output.

9. A distributed task scheduling device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the distributed task scheduling method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the distributed task scheduling method according to any one of claims 1 to 8 are implemented.

Images