Task processing method and device, storage medium and processor

Abstract

The application discloses a task processing method and device, a storage medium and a processor. The method comprises the following steps: obtaining a target task, wherein the target task is at least one task to be scheduled in a distributed system; determining a plurality of schedulable nodes in the distributed system based on the target task; determining a target node according to a load value of each schedulable node; scheduling the target task to the target node and executing the target task in the target node. Through the application, the method for scheduling tasks in the distributed system in the related art is solved, and the problem of low task execution efficiency is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method and apparatus for processing a task, a storage medium, and a processor. Background Technology

[0002] A cloud-native security scanning system is typically a distributed system consisting of one or more (primary and backup) central control terminals and a large number of security scanning programs installed on worker nodes. The central control terminal communicates with multiple security scanning programs via a network, configuring and distributing security scanning tasks (such as scanning for vulnerabilities in system repository images, and scanning images for viruses) to the relevant security scanning programs to achieve the purpose of security scanning of repository images. For example, Figure 1 This is a schematic diagram illustrating the execution of a security scanning task in a cloud-native system based on existing technology, such as... Figure 1 As shown, ① at the central control terminal, the administrator can issue scanning task instructions. The scheduler on the central control terminal schedules the scanning tasks to the security scanning program on the worker node according to a certain algorithm. ② The program puts the assigned tasks into the execution queue and performs scanning or antivirus tasks on the images in the image repository in a serial or parallel manner. ③ Then the execution results are returned to the central control terminal.

[0003] However, due to the varying resources of each worker node and the constantly changing workload on those nodes, the execution efficiency of protection tasks is affected. Furthermore, resource contention between programs on the same node can impact the node's own business operations. In another related technology, a round-robin task scheduling algorithm is used to schedule tasks in a distributed system. Specifically, the scheduler distributes tasks to be scheduled evenly among currently available worker nodes in a round-robin manner. Assuming there are three worker nodes (A, B, and C) and 10 tasks to be scheduled, the scheduler schedules task 1 to node A, task 2 to node B, task 3 to node C, task 4 to node A, and so on. Ultimately, A is allocated 4 tasks, while B and C are each allocated 3 tasks. However, this technology has the following drawbacks: without considering the resources and workload of the worker nodes, such task scheduling can easily lead to resource waste or low task execution efficiency. For example, if the resource configurations of nodes A, B, and C do not match the workload required by the business, some tasks may be assigned to nodes with fewer available resources, resulting in low protection efficiency and impacting the resource consumption of existing business operations.

[0004] Furthermore, related technology two employs a single-dimensional task scheduling algorithm to schedule tasks in a distributed system. That is, the scheduler only considers the utilization of a single resource for task scheduling. For example, the scheduler might simply assign tasks to the worker node with the lowest current CPU utilization. Suppose there are three worker nodes A, B, and C, and 10 tasks to be scheduled. The scheduler samples the data and determines that worker node A has the lowest CPU utilization, so it schedules all 10 tasks to that node. However, using a scheduling algorithm like related technology two can easily lead to a situation where a particular worker node's resources in other dimensions, such as memory, are not considered. Additionally, the degree of variation in node workload can also cause unfairness in task scheduling, resulting in low task execution efficiency.

[0005] There is currently no effective solution to the problem of low task execution efficiency caused by the methods of scheduling tasks in distributed systems in related technologies. Summary of the Invention

[0006] The main objective of this application is to provide a task processing method, apparatus, storage medium, and processor to solve the problem that the methods for scheduling tasks in distributed systems in related technologies result in low task execution efficiency.

[0007] To achieve the above objectives, according to one aspect of this application, a task processing method is provided. The method includes: obtaining a target task, wherein the target task is at least one task to be scheduled in a distributed system; determining a plurality of schedulable nodes in the distributed system based on the target task; determining a target node based on the load value of each schedulable node; scheduling the target task to the target node; and executing the target task on the target node.

[0008] Further, before determining the target node based on the load value of each schedulable node, the method further includes: obtaining a first arithmetic mean, a second arithmetic mean, a third arithmetic mean, and a fourth arithmetic mean, and the number of first tasks in each schedulable node, wherein the first arithmetic mean is the arithmetic mean of the CPU utilization of each schedulable node, the second arithmetic mean is the arithmetic mean of the memory utilization of each schedulable node, the third arithmetic mean is the arithmetic mean of the disk I / O utilization of each schedulable node, the fourth arithmetic mean is the arithmetic mean of the network bandwidth utilization of each schedulable node, and the first task is an unfinished task in each schedulable node; determining the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node; and calculating the load value of each schedulable node based on the first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean, the number of first tasks in each schedulable node, and the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node.

[0009] Further, obtaining the first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean respectively includes: obtaining multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates for each schedulable node; and determining the first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean based on the multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates for each schedulable node.

[0010] Further, determining the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node includes: obtaining the type information of the target task; and determining the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node based on the type information of the target task.

[0011] Further, determining the target node based on the load value of each schedulable node includes: determining the schedulable node with the smallest load value among multiple schedulable nodes based on the load value of each schedulable node; and taking the schedulable node with the smallest load value among the multiple schedulable nodes as the target node.

[0012] Further, determining multiple schedulable nodes in the distributed system based on the target task includes: obtaining type information of the target task; obtaining the minimum CPU requirement and minimum memory requirement required to execute the target task according to the type information of the target task; calculating the arithmetic mean of CPU resources and the arithmetic mean of memory resources of each node in the distributed system; determining multiple first nodes in the distributed system, and using the multiple first nodes as the multiple schedulable nodes, wherein the arithmetic mean of CPU resources of each first node is not less than the minimum CPU requirement required to execute the target task, and the arithmetic mean of memory resources of each first node is not less than the minimum memory requirement required to execute the target task.

[0013] Furthermore, after scheduling the target task to the target node, the method further includes: updating the number of second tasks in the target node, wherein the second task is a task in the target node that has not been completed; scheduling a third task based on the updated number of second tasks in the target node, wherein the third task is a task that needs to be scheduled after scheduling the target task in the distributed system.

[0014] To achieve the above objectives, according to another aspect of this application, a task processing apparatus is provided. The apparatus includes: a first acquisition module for acquiring a target task, wherein the target task is at least one task to be scheduled in a distributed system; a first determination module for determining a plurality of schedulable nodes in the distributed system based on the target task; a second determination module for determining a target node based on the load value of each schedulable node; and a first processing module for scheduling the target task to the target node and executing the target task on the target node.

[0015] Furthermore, the apparatus further includes: a second acquisition module, configured to acquire, before determining the target node based on the load value of each schedulable node, a first arithmetic mean, a second arithmetic mean, a third arithmetic mean, and a fourth arithmetic mean, and the number of first tasks in each schedulable node, respectively, wherein the first arithmetic mean is the arithmetic mean of the CPU utilization of each schedulable node, the second arithmetic mean is the arithmetic mean of the memory utilization of each schedulable node, the third arithmetic mean is the arithmetic mean of the disk I / O utilization of each schedulable node, and the fourth arithmetic mean is the arithmetic mean of the network bandwidth utilization of each schedulable node. The arithmetic mean of utilization, the first task is the unfinished task in each schedulable node; the third determining module is used to determine the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization and network bandwidth utilization of each schedulable node respectively; the first calculation module is used to calculate the load value of each schedulable node based on the first arithmetic mean, the second arithmetic mean, the third arithmetic mean and the fourth arithmetic mean, the number of the first task in each schedulable node, and the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization and network bandwidth utilization of each schedulable node.

[0016] Further, the second acquisition module includes: a first acquisition unit, configured to acquire multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates for each schedulable node; and a first determination unit, configured to determine the first arithmetic average, the second arithmetic average, the third arithmetic average, and the fourth arithmetic average based on the multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates of each schedulable node.

[0017] Furthermore, the third determining module includes: a second acquiring unit, used to acquire the type information of the target task; and a second determining unit, used to determine the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization and network bandwidth utilization of each schedulable node based on the type information of the target task.

[0018] Further, the second determining module includes: a third determining unit, used to determine the schedulable node with the smallest load value among multiple schedulable nodes based on the load value of each schedulable node; and a fourth determining unit, used to take the schedulable node with the smallest load value among the multiple schedulable nodes as the target node.

[0019] Further, the first determining module includes: a third obtaining unit, used to obtain the type information of the target task; a fourth obtaining unit, used to obtain the minimum CPU requirement and minimum memory requirement required to execute the target task based on the type information of the target task; a first calculating unit, used to calculate the arithmetic mean of CPU resources and the arithmetic mean of memory resources of each node in the distributed system; and a fifth determining unit, used to determine a plurality of first nodes in the distributed system and to designate the plurality of first nodes as the plurality of schedulable nodes, wherein the arithmetic mean of CPU resources of each first node is not less than the minimum CPU requirement required to execute the target task, and the arithmetic mean of memory resources of each first node is not less than the minimum memory requirement required to execute the target task.

[0020] Furthermore, the device further includes: a first update module, configured to update the number of second tasks in the target node after scheduling the target task to the target node, wherein the second task is a task in the target node that has not been completed; and a first scheduling module, configured to schedule a third task based on the updated number of second tasks in the target node, wherein the third task is a task that needs to be scheduled after scheduling the target task in the distributed system.

[0021] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is provided, the storage medium storing a program, wherein the program performs a processing method for any of the tasks described above.

[0022] To achieve the above objectives, according to another aspect of this application, a processor is provided for running a program, wherein the program, when running, performs the processing method of any of the tasks described above.

[0023] This application employs the following steps: obtaining a target task, wherein the target task is at least one task to be scheduled in a distributed system; determining multiple schedulable nodes in the distributed system based on the target task; determining a target node based on the load value of each schedulable node; scheduling the target task to the target node and executing the target task on the target node. This solves the problem of low task execution efficiency caused by methods for scheduling tasks in distributed systems in related technologies. By determining multiple schedulable nodes in the distributed system based on the obtained target task, determining the target node based on the load value of each schedulable node, and then scheduling the target task to the target node and executing the target task on the target node, the execution efficiency of the task is improved. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 This is a schematic diagram illustrating the execution of a security scanning task in a cloud-native system based on existing technology.

[0026] Figure 2 This is a flowchart of a task processing method provided according to an embodiment of this application;

[0027] Figure 3 This is a flowchart of an optional task processing method provided according to an embodiment of this application;

[0028] Figure 4 This is a schematic diagram of a task processing apparatus provided according to an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent information from the aforementioned user or organization.

[0034] The present invention will now be described in conjunction with preferred implementation steps. Figure 2 This is a flowchart of a task processing method provided according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0035] Step S201: Obtain the target task, wherein the target task is at least one task to be scheduled in the distributed system.

[0036] For example, the aforementioned distributed system could be a cloud-native security scanning system. Furthermore, the security scanning tasks (the aforementioned target tasks) configured and issued by the central control unit within the cloud-native security scanning system can be obtained first, and then these security scanning tasks can be scheduled and processed.

[0037] Step S202: Based on the target task, determine multiple schedulable nodes in the distributed system.

[0038] For example, the aforementioned multiple schedulable nodes can be multiple schedulable worker nodes in a cloud-native security scanning system. Furthermore, these worker nodes can be servers. For instance, multiple schedulable nodes can be selected from the multiple worker nodes based on the information related to the security scanning tasks configured and issued by the central control terminal and the status of the multiple worker nodes in the cloud-native security scanning system.

[0039] Step S203: Determine the target node based on the load value of each schedulable node.

[0040] For example, the load value of each schedulable node is calculated, and then, based on the load value of each schedulable node, the working node (the target node mentioned above) is selected from multiple schedulable nodes to finally perform the security scanning task.

[0041] Step S204: Schedule the target task to the target node and execute the target task on the target node.

[0042] For example, a security scan task can be scheduled to the finally selected worker node to perform the security scan task, and then the security scan task can be executed on that worker node.

[0043] Through the above steps S201 to S204, based on the target task obtained in the distributed system, multiple schedulable nodes in the distributed system are determined, and the target node is determined according to the load value of each schedulable node. The target task is then scheduled to the target node and executed in the target node, thereby improving the execution efficiency of the task.

[0044] To quickly and accurately determine multiple schedulable nodes in a distributed system, the task processing method provided in this application embodiment can also determine multiple schedulable nodes in the distributed system through the following steps: obtaining the type information of the target task; obtaining the minimum CPU requirement and minimum memory requirement required to execute the target task based on the type information of the target task; calculating the arithmetic mean of the CPU resources and the arithmetic mean of the memory resources of each node in the distributed system; determining multiple first nodes in the distributed system, and using the multiple first nodes as multiple schedulable nodes, wherein the arithmetic mean of the CPU resources of each first node is not less than the minimum CPU requirement required to execute the target task, and the arithmetic mean of the memory resources of each first node is not less than the minimum memory requirement required to execute the target task.

[0045] For example, sampling of worker nodes can be performed first. Worker node sampling can include: idle CPU (central processing unit) resources (Ci), idle memory resources (Mi), CPU utilization (C), memory utilization (M), disk I / O (input and output) utilization (W), interface network bandwidth utilization (B), and the number of unfinished tasks (J) as a set of sampling results.

[0046] In addition, the sampling interval is t (in seconds), and the sampling results are synchronized to the central control terminal in real time. For each working node, the central control terminal can save up to N sets of sampling results, and the results are in chronological order, that is, the Nth result is the latest.

[0047] Then, based on different task types (vulnerability scanning, compliance checks, virus scanning, etc.), the task can be split according to the number of images. The central control terminal can then specify the minimum amount of resources required for each task to run, including the minimum CPU resource requirement (Cr) and the minimum memory requirement (Mr).

[0048] Furthermore, when calculating and filtering schedulable nodes at the central control end, the arithmetic mean of Ci (Cia) and the arithmetic mean of Mi (Mia) of the working nodes can be calculated first.

[0049] If Cin is the sampling result of the nth Ci:

[0050] Cia=(Ci1+Ci2+Ci3+…+Ci N ) / N;

[0051] If Min is the sampling result of the nth Mi:

[0052] Mia=(Mi1+Mi2+Mi3+…+Mi N ) / N;

[0053] If Cia>=Cr and Mia>=Mr, then the node is a schedulable node. This means that the idle resources of the worker node during the time period Nt can meet the minimum requirements for the task to run.

[0054] All schedulable nodes are then selected using the method described above. Additionally, if no node meets the above conditions, the task is suspended and rescheduled after a time of Nt.

[0055] Using the above method, multiple schedulable nodes can be quickly and accurately identified based on CPU and memory resources.

[0056] To quickly and accurately determine the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node, the task processing method provided in this application embodiment can also determine the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node through the following steps: obtaining the type information of the target task; and determining the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node based on the type information of the target task.

[0057] For example, the weight values ​​for a node's CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization can be set according to different types of tasks. For instance, resource download tasks are most affected by network bandwidth and disk I / O, followed by memory, and lastly CPU. Therefore, the weight values ​​for these four resources can be set to 10%, 20%, 35%, and 35%, respectively.

[0058] Using the above method, the weight values ​​corresponding to CPU, memory, disk I / O, and network bandwidth can be quickly and accurately set according to the type of task.

[0059] To quickly and accurately obtain the first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean, the task processing method provided in this application embodiment can also obtain the first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean through the following steps: obtaining multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates for each schedulable node; and determining the first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean based on the multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates for each schedulable node.

[0060] For example, calculate the arithmetic mean (Ca) of C, the arithmetic mean (Ma) of M, the arithmetic mean (Wa) of W, and the arithmetic mean (Ba) of B for schedulable nodes.

[0061] For example, if Cn is the sampling result of the nth C:

[0062] Ca=(C1+C2+C3+…+C N ) / N;

[0063] If Mn is the sampling result of the nth M:

[0064] Ma=(M1+M2+M3+…+M N ) / N;

[0065] If Wn is the sampling result of the nth W:

[0066] Wa=(W1+W2+W3+…+W N ) / N;

[0067] If Bn is the sampling result of the nth B:

[0068] Ba = (B1 + B2 + B3 + ... + B) N ) / N.

[0069] The above method can quickly and accurately calculate the arithmetic average of CPU, memory, disk I / O, and network bandwidth.

[0070] To quickly and accurately calculate the load value of each schedulable node, the task processing method provided in this application embodiment can also calculate the load value of each schedulable node through the following steps: obtaining a first arithmetic mean, a second arithmetic mean, a third arithmetic mean, and a fourth arithmetic mean, as well as the number of first tasks in each schedulable node, wherein the first arithmetic mean is the arithmetic mean of the CPU utilization of each schedulable node, the second arithmetic mean is the arithmetic mean of the memory utilization of each schedulable node, the third arithmetic mean is the arithmetic mean of the disk I / O utilization of each schedulable node, the fourth arithmetic mean is the arithmetic mean of the network bandwidth utilization of each schedulable node, and the first task is an unfinished task in each schedulable node; determining the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node; and calculating the load value of each schedulable node based on the first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean, the number of first tasks in each schedulable node, and the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node.

[0071] For example, the formula for calculating node load is as follows:

[0072] (Ca*(Cr / Cia)*a+Ma*(Mr / Mia)*b+Wa*c+Ba*d)*(J N +1)

[0073] Among them, J N This represents the number of tasks that have not yet been completed in the latest sample result (Nth J). a, b, c, and d represent the weights of four resources: CPU, memory, disk I / O, and network bandwidth, respectively, with a sum of 100%. Different values ​​can be set depending on the type of task. Ca is the arithmetic mean of CPU utilization, Ma is the arithmetic mean of memory utilization, Wa is the arithmetic mean of disk I / O utilization, and Ba is the arithmetic mean of interface network bandwidth. Cr is the minimum CPU resource requirement for a single task execution, and Mr is the minimum memory requirement for a single task execution. Cia is the arithmetic mean of CPU resource usage, and Mia is the arithmetic mean of memory resource usage.

[0074] The above method allows for the rapid and accurate calculation of the load value of each node.

[0075] In order to quickly and accurately determine the target node, the task processing method provided in this application embodiment can also determine the target node through the following steps: based on the load value of each schedulable node, determine the schedulable node with the smallest load value among multiple schedulable nodes; and take the schedulable node with the smallest load value among multiple schedulable nodes as the target node.

[0076] For example, after calculating the load values ​​of all schedulable nodes, find the schedulable node with the smallest load value (the target node mentioned above) and schedule the task to the schedulable node with the smallest load value.

[0077] Using the above method, based on the load value of each node, it is possible to quickly and accurately determine which working node the task needs to be scheduled to.

[0078] To quickly and accurately schedule the third task, the task processing method provided in this application embodiment can also schedule the third task through the following steps: updating the number of second tasks in the target node, wherein the second task is a task in the target node that has not been completed; scheduling the third task based on the updated number of second tasks in the target node, wherein the third task is a task that needs to be scheduled after the target task is scheduled in the distributed system.

[0079] For example, update the number of tasks to be executed in the worker node with the smallest load value, and the updated value J N '=J N +1, that is, J N This represents the number of tasks awaiting execution in the worker node with the lowest load value after the task is scheduled. Since the task is scheduled to the worker node with the lowest load value, the number of tasks awaiting execution in that worker node should increase by one. Furthermore, if there are still tasks to be scheduled before the new sampling results are synchronized to the control terminal, J should be used when calculating the load value. N The value of '.

[0080] The above solution can ensure the accuracy of subsequent task scheduling.

[0081] For example, Figure 3 This is a flowchart of an optional task processing method provided according to an embodiment of this application, such as... Figure 3 As shown, the optional task processing methods include the following steps:

[0082] (1) Sampling of working nodes

[0083] The sampled results include: idle CPU resources (Ci), idle memory resources (Mi), CPU utilization (C), memory utilization (M), disk I / O utilization (W), interface network bandwidth utilization (B), and number of unfinished tasks (J).

[0084] (2) Sampling interval: t (in seconds)

[0085] The sampling results are synchronized to the central control terminal in real time. For each working node, the central control terminal can store a maximum of N sets of sampling results, which are arranged in chronological order, i.e., the Nth result is the latest.

[0086] (3) Task Type

[0087] Based on different task types (vulnerability scanning, compliance checks, virus removal, etc.), the system is split according to the number of images; the central control terminal specifies the minimum resource requirements for each task execution, including minimum CPU resource requirements (Cr) and minimum memory requirements (Mr). For example... Figure 3 As shown, after a new task is generated, it is arranged in the queue of tasks to be scheduled, from the tail to the head, to get task 1. Then, according to the type of task 1, the minimum amount of resources required to run task 1 once is set.

[0088] (4) Task scheduling

[0089] Step 1: The central control unit calculates and filters schedulable nodes:

[0090] Calculate the arithmetic mean (Cia) of Ci and the arithmetic mean (Mia) of Mi for the working nodes.

[0091] If Cin is the sampling result of the nth Ci:

[0092] Cia=(Ci1+Ci2+Ci3+…+Ci N ) / N;

[0093] If Min is the sampling result of the nth Mi:

[0094] Mia=(Mi1+Mi2+Mi3+…+Mi N ) / N;

[0095] If Cia>=Cr and Mia>=Mr, then the node is a schedulable node. This means that the idle resources of the worker node during the time period Nt can meet the minimum requirements for the task to run.

[0096] After filtering out all schedulable nodes using the method described above, proceed to step 2.

[0097] In addition, if no node meets the above conditions (i.e., there are no schedulable nodes), the task is suspended and rescheduled after Nt time.

[0098] Step 2: Complete task scheduling according to the algorithm:

[0099] Calculate the arithmetic mean (Ca), arithmetic mean (Ma), arithmetic mean (Wa), and arithmetic mean (Ba) of the schedulable nodes C.

[0100] If Cn is the sampling result of the nth C:

[0101] Ca=(C1+C2+C3+…+C N ) / N;

[0102] If Mn is the sampling result of the nth M:

[0103] Ma=(M1+M2+M3+…+M N ) / N;

[0104] If Wn is the sampling result of the nth W:

[0105] Wa=(W1+W2+W3+…+W N ) / N;

[0106] If Bn is the sampling result of the nth B:

[0107] Ba = (B1 + B2 + B3 + ... + B) N ) / N;

[0108] The formula for calculating node load is:

[0109] (Ca*(Cr / Cia)*a+Ma*(Mr / Mia)*b+Wa*c+Ba*d)*(J N +1)

[0110] Among them, J N This represents the number of unfinished tasks in the latest sample result (Nth J). a, b, c, and d represent the weights of four resources: CPU, memory, disk I / O, and network bandwidth, respectively. Their sum is 100%, and different values ​​can be set depending on the type of task. For example, for resource download tasks, network bandwidth and disk I / O have the greatest impact, followed by memory, and lastly CPU. Therefore, the values ​​of a, b, c, and d can be set to 10%, 20%, 35%, and 35%, respectively.

[0111] Calculate the load values ​​of all schedulable nodes, schedule the task to the node with the lowest load value, and continue to step 3.

[0112] Step 3, update the J of the scheduled node. N value:

[0113] The updated value J N '=J N +1.

[0114] Furthermore, if there are still tasks to be scheduled before the new sampling results are synchronized to the control terminal, J should be used when calculating the load value. N The value of '.

[0115] The above solution optimizes the task scheduling method of the distributed system, decomposes the granularity of scanning tasks, and makes more rational and full use of cluster resources, thereby improving task execution efficiency. Furthermore, this algorithm provides a relatively fair multi-dimensional task scheduling algorithm at the central control end and offers efficient security protection capabilities for the entire cloud-native business system under real-time business workloads. In addition, the algorithm comprehensively considers various resources with weights on worker nodes and dynamically allocates tasks to different worker nodes at the individual task level, which more closely resembles the actual use cases of cloud-native workloads, achieving low-interference and high-efficiency scheduling of security scanning tasks.

[0116] For example, using the method provided in this application embodiment, in a cloud-native compilation environment, some worker nodes primarily provide users with software package query and download services, consuming significant amounts of network bandwidth and node memory; other worker nodes primarily compile code into usable software packages by pulling relevant code and repository images, consuming significant amounts of CPU resources. When an administrator issues relevant security scanning tasks from the central control terminal, the central control terminal can use this algorithm to schedule the tasks to the optimal worker nodes.

[0117] Therefore, the method provided in this application embodiment can be applied to the following scenarios:

[0118] (1) In cloud-native scenarios, there are a large number of working nodes, and no dedicated resources are reserved for security scanning.

[0119] (2) The business being run has key business indicators, and it is not allowed to cause significant impact on the existing business provided by the system while performing security scanning tasks.

[0120] In summary, the task processing method provided in this application, by obtaining a target task (which is at least one task to be scheduled in a distributed system), determining multiple schedulable nodes in the distributed system based on the target task, determining a target node based on the load value of each schedulable node, scheduling the target task to the target node, and executing the target task on the target node, solves the problem of low task execution efficiency in related technologies for scheduling tasks in distributed systems. By determining multiple schedulable nodes in the distributed system based on the obtained target task, determining the target node based on the load value of each schedulable node, and then scheduling the target task to the target node and executing the target task on the target node, the method improves the execution efficiency of the task.

[0121] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0122] This application also provides a task processing apparatus. It should be noted that the task processing apparatus of this application can be used to execute the task processing method provided in this application. The task processing apparatus provided in this application will be described below.

[0123] Figure 4 This is a schematic diagram of a task processing apparatus according to an embodiment of this application. Figure 4 As shown, the device includes: a first acquisition module 401, a first determination module 402, a second determination module 403, and a first processing module 404.

[0124] Specifically, the first acquisition module 401 is used to acquire the target task, wherein the target task is at least one task to be scheduled in the distributed system;

[0125] The first determining module 402 is used to determine multiple schedulable nodes in the distributed system based on the target task.

[0126] The second determining module 403 is used to determine the target node based on the load value of each schedulable node;

[0127] The first processing module 404 is used to schedule the target task to the target node and execute the target task in the target node.

[0128] In summary, the task processing apparatus provided in this application embodiment acquires a target task through a first acquisition module 401, wherein the target task is at least one task to be scheduled in a distributed system; a first determination module 402 determines multiple schedulable nodes in the distributed system based on the target task; a second determination module 403 determines a target node based on the load value of each schedulable node; and a first processing module 404 schedules the target task to the target node and executes the target task on the target node. This solves the problem of low task execution efficiency caused by methods for scheduling tasks in distributed systems in related technologies. By determining multiple schedulable nodes in the distributed system based on the acquired target task, determining the target node based on the load value of each schedulable node, scheduling the target task to the target node, and executing the target task on the target node, the execution efficiency of the task is improved.

[0129] Optionally, in the task processing apparatus provided in the embodiments of this application, the apparatus further includes: a second acquisition module, configured to acquire, before determining the target node based on the load value of each schedulable node, a first arithmetic mean, a second arithmetic mean, a third arithmetic mean, and a fourth arithmetic mean, and the number of first tasks in each schedulable node, respectively, wherein the first arithmetic mean is the arithmetic mean of the CPU utilization of each schedulable node, the second arithmetic mean is the arithmetic mean of the memory utilization of each schedulable node, the third arithmetic mean is the arithmetic mean of the disk I / O utilization of each schedulable node, and the fourth arithmetic mean is the arithmetic mean of the first tasks in each schedulable node. The system is composed of: an arithmetic mean of the network bandwidth utilization of the scheduling nodes; a first task that is not yet completed in each schedulable node; a third determining module that determines the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node; and a first calculation module that calculates the load value of each schedulable node based on the first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean, the number of first tasks in each schedulable node, and the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node.

[0130] For example, the formula for calculating node load is as follows:

[0131] (Ca*(Cr / Cia)*a+Ma*(Mr / Mia)*b+Wa*c+Ba*d)*(J N +1)

[0132] Among them, J N This represents the number of tasks that have not yet been completed in the latest sample result (Nth J). a, b, c, and d represent the weights of four resources: CPU, memory, disk I / O, and network bandwidth, respectively, with a sum of 100%. Different values ​​can be set depending on the type of task. Ca is the arithmetic mean of CPU utilization, Ma is the arithmetic mean of memory utilization, Wa is the arithmetic mean of disk I / O utilization, and Ba is the arithmetic mean of interface network bandwidth. Cr is the minimum CPU resource requirement for a single task execution, and Mr is the minimum memory requirement for a single task execution. Cia is the arithmetic mean of CPU resource usage, and Mia is the arithmetic mean of memory resource usage.

[0133] In summary, the load value of each node can be calculated quickly and accurately.

[0134] Optionally, in the task processing apparatus provided in the embodiments of this application, the second acquisition module includes: a first acquisition unit, configured to acquire multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates of each schedulable node; and a first determination unit, configured to determine a first arithmetic average, a second arithmetic average, a third arithmetic average, and a fourth arithmetic average based on the multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates of each schedulable node.

[0135] For example, calculate the arithmetic mean (Ca) of C, the arithmetic mean (Ma) of M, the arithmetic mean (Wa) of W, and the arithmetic mean (Ba) of B for schedulable nodes.

[0136] For example, if Cn is the sampling result of the nth C:

[0137] Ca=(C1+C2+C3+…+C N ) / N;

[0138] If Mn is the sampling result of the nth M:

[0139] Ma=(M1+M2+M3+…+M N ) / N;

[0140] If Wn is the sampling result of the nth W:

[0141] Wa=(W1+W2+W3+…+W N ) / N;

[0142] If Bn is the sampling result of the nth B:

[0143] Ba = (B1 + B2 + B3 + ... + B) N ) / N.

[0144] In summary, the arithmetic averages of CPU, memory, disk I / O, and network bandwidth can be calculated quickly and accurately.

[0145] Optionally, in the task processing apparatus provided in the embodiments of this application, the third determining module includes: a second obtaining unit, used to obtain type information of the target task; and a second determining unit, used to determine the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization and network bandwidth utilization of each schedulable node according to the type information of the target task.

[0146] For example, the weight values ​​for a node's CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization can be set according to different types of tasks. For instance, resource download tasks are most affected by network bandwidth and disk I / O, followed by memory, and lastly CPU. Therefore, the weight values ​​for these four resources can be set to 10%, 20%, 35%, and 35%, respectively.

[0147] In summary, based on the type of task, the weight values ​​corresponding to CPU, memory, disk I / O, and network bandwidth can be set quickly and accurately.

[0148] Optionally, in the task processing apparatus provided in the embodiments of this application, the second determining module includes: a third determining unit, used to determine the schedulable node with the smallest load value among multiple schedulable nodes based on the load value of each schedulable node; and a fourth determining unit, used to take the schedulable node with the smallest load value among multiple schedulable nodes as the target node.

[0149] For example, after calculating the load values ​​of all schedulable nodes, find the schedulable node with the smallest load value (the target node mentioned above) and schedule the task to the schedulable node with the smallest load value.

[0150] In summary, based on the load value of each node, it is possible to quickly and accurately determine which worker node the task needs to be scheduled to.

[0151] Optionally, in the task processing apparatus provided in this application embodiment, the first determining module includes: a third obtaining unit, used to obtain type information of the target task; a fourth obtaining unit, used to obtain the minimum CPU requirement and minimum memory requirement required to execute the target task according to the type information of the target task; a first calculating unit, used to calculate the arithmetic mean of CPU resources and the arithmetic mean of memory resources of each node in the distributed system; and a fifth determining unit, used to determine multiple first nodes in the distributed system and treat the multiple first nodes as multiple schedulable nodes, wherein the arithmetic mean of CPU resources of each first node is not less than the minimum CPU requirement required to execute the target task, and the arithmetic mean of memory resources of each first node is not less than the minimum memory requirement required to execute the target task.

[0152] For example, sampling of worker nodes can be performed first. Worker node sampling can include: idle CPU (central processing unit) resources (Ci), idle memory resources (Mi), CPU utilization (C), memory utilization (M), disk I / O (input and output) utilization (W), interface network bandwidth utilization (B), and the number of unfinished tasks (J) as a set of sampling results.

[0153] In addition, the sampling interval is t (in seconds), and the sampling results are synchronized to the central control terminal in real time. For each working node, the central control terminal can save up to N sets of sampling results, and the results are in chronological order, that is, the Nth result is the latest.

[0154] Then, based on different task types (vulnerability scanning, compliance checks, virus scanning, etc.), the task can be split according to the number of images. The central control terminal can then specify the minimum amount of resources required for each task to run, including the minimum CPU resource requirement (Cr) and the minimum memory requirement (Mr).

[0155] Furthermore, when calculating and filtering schedulable nodes at the central control end, the arithmetic mean of Ci (Cia) and the arithmetic mean of Mi (Mia) of the working nodes can be calculated first.

[0156] If Cin is the sampling result of the nth Ci:

[0157] Cia=(Ci1+Ci2+Ci3+…+Ci N ) / N;

[0158] If Min is the sampling result of the nth Mi:

[0159] Mia=(Mi1+Mi2+Mi3+…+Mi N ) / N;

[0160] If Cia>=Cr and Mia>=Mr, then the node is a schedulable node. This means that the idle resources of the worker node during the time period Nt can meet the minimum requirements for the task to run.

[0161] All schedulable nodes are then selected using the method described above. Additionally, if no node meets the above conditions, the task is suspended and rescheduled after a time of Nt.

[0162] In summary, based on the amount of CPU and memory resources, multiple schedulable nodes can be quickly and accurately identified.

[0163] Optionally, in the task processing apparatus provided in the embodiments of this application, the apparatus further includes: a first update module, configured to update the number of second tasks in the target node after scheduling the target task to the target node, wherein the second task is a task in the target node that has not been completed; and a first scheduling module, configured to schedule a third task according to the updated number of second tasks in the target node, wherein the third task is a task that needs to be scheduled after scheduling the target task in the distributed system.

[0164] For example, update the number of tasks to be executed in the worker node with the smallest load value, and the updated value J N '=J N +1, that is, J N This represents the number of tasks awaiting execution in the worker node with the lowest load value after the task is scheduled. Since the task is scheduled to the worker node with the lowest load value, the number of tasks awaiting execution in that worker node should increase by one. Furthermore, if there are still tasks to be scheduled before the new sampling results are synchronized to the control terminal, J should be used when calculating the load value. N The value of '.

[0165] In summary, this ensures the accuracy of subsequent task scheduling.

[0166] The task processing device includes a processor and a memory. The first acquisition module 401, the first determination module 402, the second determination module 403, and the first processing module 404 are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0167] A processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and task execution efficiency can be improved by adjusting kernel parameters.

[0168] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0169] This invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a method for processing the task.

[0170] This invention provides a processor for running a program, wherein the program executes a method for processing the task during runtime.

[0171] like Figure 5As shown, an embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: obtaining a target task, wherein the target task is at least one task to be scheduled in a distributed system; determining multiple schedulable nodes in the distributed system based on the target task; determining a target node based on the load value of each schedulable node; scheduling the target task to the target node and executing the target task in the target node.

[0172] When the processor executes the program, it also performs the following steps: Before determining the target node based on the load value of each schedulable node, the method further includes: obtaining a first arithmetic mean, a second arithmetic mean, a third arithmetic mean, and a fourth arithmetic mean, and the number of first tasks in each schedulable node, wherein the first arithmetic mean is the arithmetic mean of the CPU utilization of each schedulable node, the second arithmetic mean is the arithmetic mean of the memory utilization of each schedulable node, the third arithmetic mean is the arithmetic mean of the disk I / O utilization of each schedulable node, the fourth arithmetic mean is the arithmetic mean of the network bandwidth utilization of each schedulable node, and the first task is an unfinished task in each schedulable node; determining the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node; and calculating the load value of each schedulable node based on the first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean, the number of first tasks in each schedulable node, and the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node.

[0173] When the processor executes the program, it also performs the following steps: obtaining the first arithmetic average, the second arithmetic average, the third arithmetic average, and the fourth arithmetic average respectively includes: obtaining multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates for each schedulable node; and determining the first arithmetic average, the second arithmetic average, the third arithmetic average, and the fourth arithmetic average based on the multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates for each schedulable node.

[0174] When the processor executes the program, it also performs the following steps: determining the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node, including: obtaining the type information of the target task; and determining the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node based on the type information of the target task.

[0175] When the processor executes the program, it also performs the following steps: determining the target node based on the load value of each schedulable node includes: determining the schedulable node with the smallest load value among multiple schedulable nodes based on the load value of each schedulable node; and taking the schedulable node with the smallest load value among the multiple schedulable nodes as the target node.

[0176] When the processor executes the program, it also performs the following steps: determining multiple schedulable nodes in the distributed system based on the target task includes: obtaining type information of the target task; obtaining the minimum CPU requirement and minimum memory requirement required to execute the target task according to the type information of the target task; calculating the arithmetic mean of CPU resources and the arithmetic mean of memory resources of each node in the distributed system; determining multiple first nodes in the distributed system, and using the multiple first nodes as the multiple schedulable nodes, wherein the arithmetic mean of CPU resources of each first node is not less than the minimum CPU requirement required to execute the target task, and the arithmetic mean of memory resources of each first node is not less than the minimum memory requirement required to execute the target task.

[0177] When the processor executes the program, it also performs the following steps: after scheduling the target task to the target node, the method further includes: updating the number of second tasks in the target node, wherein the second task is a task in the target node that has not been completed; scheduling a third task based on the updated number of second tasks in the target node, wherein the third task is a task that needs to be scheduled after scheduling the target task in the distributed system.

[0178] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0179] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: obtaining a target task, wherein the target task is at least one task to be scheduled in a distributed system; determining multiple schedulable nodes in the distributed system based on the target task; determining a target node based on the load value of each schedulable node; scheduling the target task to the target node, and executing the target task in the target node.

[0180] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: before determining the target node based on the load value of each schedulable node, the method further includes: obtaining a first arithmetic mean, a second arithmetic mean, a third arithmetic mean, and a fourth arithmetic mean, and the number of first tasks in each schedulable node, wherein the first arithmetic mean is the arithmetic mean of the CPU utilization of each schedulable node, the second arithmetic mean is the arithmetic mean of the memory utilization of each schedulable node, the third arithmetic mean is the arithmetic mean of the disk I / O utilization of each schedulable node, and the fourth arithmetic mean is the arithmetic mean of the CPU utilization of each schedulable node. The arithmetic mean is the arithmetic mean of the network bandwidth utilization of each schedulable node, and the first task is the unfinished task in each schedulable node. The weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization and network bandwidth utilization of each schedulable node are determined respectively. Based on the first arithmetic mean, the second arithmetic mean, the third arithmetic mean and the fourth arithmetic mean, the number of the first task in each schedulable node, and the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization and network bandwidth utilization of each schedulable node, the load value of each schedulable node is calculated.

[0181] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: obtaining a first arithmetic mean, a second arithmetic mean, a third arithmetic mean, and a fourth arithmetic mean respectively, including: obtaining multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates for each schedulable node; and determining the first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean respectively based on the multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates for each schedulable node.

[0182] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: determining the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node, including: obtaining the type information of the target task; and determining the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node based on the type information of the target task.

[0183] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: determining a target node based on the load value of each schedulable node, including: determining the schedulable node with the smallest load value among a plurality of schedulable nodes based on the load value of each schedulable node; and taking the schedulable node with the smallest load value among the plurality of schedulable nodes as the target node.

[0184] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: determining multiple schedulable nodes in the distributed system based on the target task, including: obtaining type information of the target task; obtaining the minimum CPU requirement and minimum memory requirement required to execute the target task according to the type information of the target task; calculating the arithmetic mean of CPU resources and the arithmetic mean of memory resources of each node in the distributed system; determining multiple first nodes in the distributed system, and using the multiple first nodes as the multiple schedulable nodes, wherein the arithmetic mean of CPU resources of each first node is not less than the minimum CPU requirement required to execute the target task, and the arithmetic mean of memory resources of each first node is not less than the minimum memory requirement required to execute the target task.

[0185] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: after scheduling the target task to the target node, the method further includes: updating the number of second tasks in the target node, wherein the second task is a task in the target node that has not been completed; scheduling a third task based on the updated number of second tasks in the target node, wherein the third task is a task that needs to be scheduled after scheduling the target task in the distributed system.

[0186] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0187] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0188] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0189] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0190] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0191] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0192] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0193] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0194] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0195] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for processing a task, characterized in that, include: Obtain a target task, wherein the target task is at least one task to be scheduled in a distributed system, the distributed system is a cloud-native security scanning system, and the target task is a security scanning task configured and issued by the central control terminal of the cloud-native security scanning system. Based on the target task, multiple schedulable nodes are identified in the distributed system; The target node is determined based on the load value of each schedulable node; The target task is scheduled to the target node and executed on the target node; Before determining the target node based on the load value of each schedulable node, the method further includes: The first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean, as well as the number of first tasks in each schedulable node, are obtained respectively. The first arithmetic mean is the arithmetic mean of the CPU utilization of each schedulable node, the second arithmetic mean is the arithmetic mean of the memory utilization of each schedulable node, the third arithmetic mean is the arithmetic mean of the disk I / O utilization of each schedulable node, the fourth arithmetic mean is the arithmetic mean of the network bandwidth utilization of each schedulable node, and the first task is an unfinished task in each schedulable node. Determine the weight values ​​for CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization for each schedulable node; Based on the first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean, the number of first tasks in each schedulable node, and the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node, the load value of each schedulable node is calculated. The weight values ​​for CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of each schedulable node are determined as follows: Obtain the type information of the target task; Based on the type information of the target task, the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization and network bandwidth utilization of each schedulable node are determined respectively. Based on the target task, the multiple schedulable nodes in the distributed system are determined to include: Obtain the type information of the target task; Based on the type information of the target task, obtain the minimum CPU requirement and minimum memory requirement required to execute the target task. Calculate the arithmetic mean of CPU resources and the arithmetic mean of memory resources for each node in the distributed system. A plurality of first nodes are identified in the distributed system, and the plurality of first nodes are designated as the plurality of schedulable nodes, wherein the arithmetic mean of the CPU resources of each first node is not less than the minimum CPU requirement required to execute the target task, and the arithmetic mean of the memory resources of each first node is not less than the minimum memory requirement required to execute the target task. The method further includes sampling the working nodes in the distributed system at a preset sampling interval, and synchronizing the sampling results to the central control terminal in real time. The central control terminal saves N sets of sampling results and calculates the arithmetic average of the CPU resources and the arithmetic average of the memory resources of each working node based on the N sets of sampling results. Based on the load value of each schedulable node, the target nodes are determined as follows: Based on the load value of each schedulable node, determine the schedulable node with the smallest load value among multiple schedulable nodes; The schedulable node with the smallest load value among the plurality of schedulable nodes is selected as the target node. After scheduling the target task to the target node, the method further includes: Update the number of second tasks in the target node, wherein the second task is the task in the target node that has not been completed; Based on the updated number of second tasks in the target node, the third task is scheduled, wherein the third task is the task that needs to be scheduled after the target task is scheduled in the distributed system.

2. The method according to claim 1, characterized in that, Obtaining the first arithmetic mean, the second arithmetic mean, the third arithmetic mean, and the fourth arithmetic mean respectively includes: Obtain multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates for each schedulable node; Based on multiple CPU utilization rates, multiple memory utilization rates, multiple disk I / O utilization rates, and multiple network bandwidth utilization rates of each schedulable node, the first arithmetic average, the second arithmetic average, the third arithmetic average, and the fourth arithmetic average are determined respectively.

3. A task processing apparatus, characterized in that, include: The first acquisition module is used to acquire the target task, wherein the target task is at least one task to be scheduled in a distributed system, the distributed system is a cloud-native security scanning system, and the target task is a security scanning task configured and issued by the central control terminal of the cloud-native security scanning system. The first determining module is used to determine multiple schedulable nodes in the distributed system based on the target task. The second determining module is used to determine the target node based on the load value of each schedulable node; The first processing module is used to schedule the target task to the target node and execute the target task in the target node; The apparatus further includes: a second acquisition module, configured to acquire, before determining the target node based on the load value of each schedulable node, a first arithmetic mean, a second arithmetic mean, a third arithmetic mean, and a fourth arithmetic mean, and the number of first tasks in each schedulable node, respectively, wherein the first arithmetic mean is the arithmetic mean of the CPU utilization of each schedulable node, the second arithmetic mean is the arithmetic mean of the memory utilization of each schedulable node, the third arithmetic mean is the arithmetic mean of the disk I / O utilization of each schedulable node, and the fourth arithmetic mean is the network bandwidth utilization of each schedulable node. The arithmetic mean of the first task is an unfinished task in each schedulable node; the third determining module is used to determine the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization and network bandwidth utilization of each schedulable node respectively; the first calculation module is used to calculate the load value of each schedulable node based on the first arithmetic mean, the second arithmetic mean, the third arithmetic mean and the fourth arithmetic mean, the number of first tasks in each schedulable node, and the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization and network bandwidth utilization of each schedulable node. The third determining module includes: a second obtaining unit, used to obtain the type information of the target task; and a second determining unit, used to determine the weight values ​​corresponding to the CPU utilization, memory utilization, disk I / O utilization and network bandwidth utilization of each schedulable node based on the type information of the target task. The apparatus further includes: a first update module, configured to update the number of second tasks in the target node after scheduling the target task to the target node, wherein the second task is a task in the target node that has not been completed; and a first scheduling module, configured to schedule a third task based on the updated number of second tasks in the target node, wherein the third task is a task that needs to be scheduled after scheduling the target task in the distributed system. The device is also used to sample the working nodes in the distributed system at a preset sampling interval and synchronize the sampling results to the central control terminal in real time. The central control terminal saves N sets of sampling results and calculates the arithmetic average of the CPU resources and the arithmetic average of the memory resources of each working node based on the N sets of sampling results. The second determining module includes: a third determining unit, used to determine the schedulable node with the smallest load value among multiple schedulable nodes based on the load value of each schedulable node; and a fourth determining unit, used to take the schedulable node with the smallest load value among multiple schedulable nodes as the target node. The device further includes: a first update module, used to update the number of second tasks in the target node after scheduling the target task to the target node, wherein the second task is a task in the target node that has not been completed; and a first scheduling module, used to schedule a third task based on the updated number of second tasks in the target node, wherein the third task is a task that needs to be scheduled after scheduling the target task in the distributed system.

4. A computer-readable storage medium, characterized in that, The storage medium stores a program, wherein the program executes the processing method of the task according to any one of claims 1 to 2.

5. A processor for a task processing method, characterized in that, The processor is used to run a program, wherein the program, when running, performs the processing method of the task according to any one of claims 1 to 2.

Images