A distributed resource management method and system

Abstract

The present application relates to the field of electronic digital processing technology, and specifically to a distributed resource management method and system, the method comprising: obtaining the maximum computing speed of each node in the distributed processing system and the communication delay between any two nodes, as well as the task generation time, task generation node, task size, and remaining processing time of each task at the current moment; determining the task resolution time; determining the node response balance, absolute communication consumption, and communication delay ratio, and determining the fitness of each particle; combining intelligent optimization algorithms for iterative calculations, obtaining the optimal solution allocation vector, and allocating the newly added tasks at the current moment. The present application can effectively avoid the problem of node paralysis caused by unreasonable distributed resource allocation due to communication delays between nodes, improve the speed of distributed computing's response to tasks, and improve the processing efficiency of the entire distributed computing.

Description

Technical Field

[0001] The present application relates to the field of electronic digital processing technology, and in particular to a distributed resource management method and system. Background Art

[0002] Distributed computing technology distributes computing tasks to multiple computing nodes, which ensures that the entire distributed processing system can quickly process tasks when faced with a large number of computing tasks, avoiding the problem of insufficient speed when a single machine processes tasks. When distributed computing technology processes computing tasks, it is necessary to summarize the computing tasks collected by each node and then allocate them to the surrounding nodes for distributed computing, thereby completing distributed processing. When computing tasks are allocated, it is equivalent to allocating and managing the computing resources of the computing nodes.

[0003] When managing distributed resources, traditional methods allocate resources based on ensuring load balancing of computing nodes to prevent tasks from piling up on a certain computing node, which causes some computing tasks to fail to respond for a long time and lead to node paralysis. When allocating tasks, traditional load balancing methods only focus on the load of distributed computing devices to allocate tasks, without fully considering the communication delay between nodes and the generation rules of computing tasks, resulting in unreasonable resource allocation, which causes some computing tasks to take too long to process due to communication delays during task allocation, thereby reducing the overall processing efficiency of the entire distributed computing device, causing some communication nodes to fail to respond for a long time and causing node paralysis. Summary of the invention

[0004] In order to solve the above technical problems, a distributed resource management method and system are provided to solve the existing problems.

[0005] The solution to the technical problem of this application is to provide a distributed resource management method and system, including the following steps:

[0006] In a first aspect, an embodiment of the present application provides a distributed resource management method, the method comprising the following steps:

[0007] Obtain the maximum computing speed of each node in the distributed processing system and the communication delay between any two nodes, as well as the task generation time, task generation node, task size, and remaining processing time of each task at the current moment; record all newly generated tasks of all nodes at the current moment as each newly added task at the current moment;

[0008] Determine the task resolution time at which each newly added task is processed by any node at the current moment according to the queuing waiting time required for each newly added task to be processed by any node at the current moment, the communication delay between the task generation node of each newly added task and any node, and the time required for each newly added task to be calculated and processed by any node;

[0009] Randomly assign nodes to all newly added tasks at the current moment, and obtain multiple solution allocation vectors as the initial population of the intelligent optimization algorithm; analyze the task solution time described in the solution allocation vector corresponding to each particle, and the speed of node generating tasks, and determine the solution efficiency sequence and generation speed sequence corresponding to each particle respectively; based on the difference between the solution efficiency sequence and the generation speed sequence, determine the node response balance corresponding to each particle;

[0010] Determine the absolute communication consumption of each newly added task in the solution allocation vector corresponding to each particle according to the time consumed in the round-trip communication transmission between the task generation node and the node corresponding to the task being processed by each newly added task in the solution allocation vector corresponding to each particle; determine the communication delay proportion corresponding to each particle based on the absolute communication consumption of all newly added tasks in the solution allocation vector corresponding to each particle and the task solution time thereof;

[0011] Based on the node response balance and the communication delay ratio, the fitness of each particle in the intelligent optimization algorithm is determined; the intelligent optimization algorithm is combined with iterative calculation to obtain the optimal solution allocation vector, and the newly added tasks at the current moment are allocated.

[0012] Preferably, determining the task resolution time at which each newly added task at the current moment is processed by any node includes:

[0013] Calculate the queuing waiting time of each newly added task before it is processed by any node based on the remaining processing time required to calculate and process the remaining tasks before each newly added task is processed by any node;

[0014] The current moment The new task was The time it takes to complete the task processing by each node The calculation formula is: ,in, For the current moment The task size of the newly added task, For the The maximum computing speed of each node, For the current moment Task generation node for new tasks To the corresponding The communication delay of each node is For the current moment The node will process the completed The newly added tasks are transferred to Task generation node for new tasks The communication delay, For the current moment The new task was The waiting time before a node is processed. is the maximum value function.

[0015] Preferably, the queue waiting time is the cumulative sum of the remaining processing time of all tasks arranged before each newly added task waiting to be processed at any node at the current moment.

[0016] Preferably, the obtaining of multiple scheme allocation vectors includes:

[0017] Construct a multidimensional vector with the same number as all newly added tasks at the current moment, and record it as the solution allocation vector;

[0018] All nodes are numbered, and a node is randomly assigned to the newly added task corresponding to each dimension in the solution allocation vector to obtain a solution allocation vector; this step is repeated to obtain multiple solution allocation vectors.

[0019] Preferably, the step of determining the solution efficiency sequence and the generation speed sequence corresponding to each particle includes:

[0020] The average of the task solving time of all newly added tasks corresponding to the same task generation node in the solution allocation vector and processed by the corresponding node is recorded as the average solving time; the average solving time of all task generation nodes in the solution allocation vector is negatively mapped and normalized to form a solving efficiency sequence;

[0021] Based on the task generation time of each task, the number of tasks generated by each node in multiple moments before the current moment is counted; the task generation speed of each node at the current moment is calculated; and the task generation speeds of all nodes in the scheme allocation vector are normalized to form a generation speed sequence.

[0022] Preferably, the node response balance corresponding to each particle is the cross entropy between the solution efficiency sequence and the generation speed sequence corresponding to each particle.

[0023] Preferably, The solution allocation vector corresponding to the particle Absolute communication cost of each new task The calculation method is: ,in, For the The solution allocation vector corresponding to the particle The number of newly added tasks to be processed corresponds to the Node to Task generation node for new tasks The communication delay between For the The solution allocation vector corresponding to the particle Task generation node for new tasks To The number of newly added tasks to be processed corresponds to the The communication delay between nodes is The solution corresponding to each particle is assigned to the first The node processes The waiting time before adding a new task, is the maximum value function.

[0024] Preferably, determining the communication delay proportion corresponding to each particle includes:

[0025] The mean of the absolute communication consumption of all newly added tasks in the solution allocation vector corresponding to each particle is recorded as the first mean;

[0026] The average of the task solution time of all newly added tasks in the solution allocation vector corresponding to each particle and processed by the corresponding node is recorded as the second average;

[0027] The ratio of the first mean value to the second mean value is used as the communication delay proportion corresponding to each particle.

[0028] Preferably, the calculation formula for the fitness of each particle in the intelligent optimization algorithm is: ,in, For the The fitness of a particle, For the The node response balance corresponding to each particle, For the The communication delay ratio corresponding to each particle is The default value is greater than 0.

[0029] In a second aspect, an embodiment of the present application also provides a distributed resource management system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of any one of the above-mentioned distributed resource management methods when executing the computer program.

[0030] This application has at least the following beneficial effects:

[0031] This application determines the task resolution time for each new task at the current moment to be processed by any node based on the queue waiting time required for each new task at the current moment to be processed by any node, the communication delay between the task generation node of each new task and any node, and the time required for each new task to be calculated and processed by any node. The beneficial effect is that it takes into account the time consumed by the task resolution after each new task is assigned to different nodes, thereby reflecting the response time of the distributed processing system to the task, and further illustrating the efficiency of the distributed processing and computing tasks; randomly assign nodes to all new tasks at the current moment, and obtain multiple The scheme allocation vectors are used as the initial population of the intelligent optimization algorithm; the task solution time in the scheme allocation vector corresponding to each particle and the speed of node generating tasks are analyzed, and the solution efficiency sequence and generation speed sequence corresponding to each particle are determined respectively. The beneficial effect is that the efficiency of calculating and processing each new task in the scheme allocation vector corresponding to each particle and the speed of node generating tasks are considered, and the possibility of task accumulation and node paralysis at different nodes is evaluated; based on the solution efficiency sequence and generation speed sequence, the node response balance corresponding to each particle is determined, and the beneficial effect is that the tasks generated by the nodes with faster task generation are considered to be faster. The speed response situation is reflected in the solution allocation vector of the corresponding particle, thereby reflecting whether the solution allocation vector of the corresponding particle can respond to all the newly added tasks evenly when allocating computing resources, so as to avoid the situation where a single node task responds too slowly, resulting in the node paralysis of a single node; according to the time consumed by each newly added task in the solution allocation vector corresponding to each particle in the round-trip communication transmission between the task generation node and the node corresponding to the processed node, the absolute communication consumption of each newly added task in the solution allocation vector corresponding to each particle is determined; based on the absolute communication consumption of all the newly added tasks in the solution allocation vector corresponding to each particle and the task solution time thereof, the communication delay proportion corresponding to each particle is determined, and its beneficial effect is The proportion of time spent on communication by the newly added tasks is taken into consideration to avoid excessive communication delay caused by task allocation according to the corresponding scheme allocation vector, and to screen out some allocation schemes with excessively high communication costs; based on the node response balance and the communication delay proportion, the fitness of each particle in the intelligent optimization algorithm is determined; iterative calculation is performed in combination with the intelligent optimization algorithm to obtain the optimal scheme allocation vector, and the newly added tasks at the current moment are allocated. The beneficial effect is that it can effectively avoid the node paralysis problem caused by unreasonable distributed resource allocation due to communication delay between nodes, improve the response speed of distributed computing to tasks, and improve the processing efficiency of the entire distributed computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] A distributed resource management method of the present application is further described in detail below in conjunction with the accompanying drawings.

[0033] Figure 1 A flowchart of a distributed resource management method provided in an embodiment of the present application;

[0034] Figure 2 A flowchart of the steps of a method for obtaining the node response balance corresponding to each particle provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of this application more clear, the following is a further detailed description of a distributed resource management method and system proposed in this application in conjunction with the accompanying drawings and implementation examples. It should be understood that the specific embodiments described here are only used to explain this application and are not used to limit this application.

[0036] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0037] See also Figure 1 , which shows a flow chart of a distributed resource management method provided by an embodiment of the present application, the method comprising the following steps:

[0038] Step 1, obtain the maximum computing speed of each node in the distributed processing system and the communication delay between any two nodes, as well as the task generation time, task generation node, task size, and remaining processing time of each task at the current moment.

[0039] This embodiment improves resource allocation in a distributed processing system for banking financial data. Therefore, there are multiple nodes in the distributed processing system, and the multiple nodes are connected through a network to form a communication network, wherein each node has both a communication function and a computing function.

[0040] First, obtain the static data of each node, including the maximum computing speed and communication delay, specifically:

[0041] According to the inherent attribute characteristics of each node, the maximum computing speed of each node is obtained, where the maximum computing speed indicates how many operations the node can perform at most in one second; secondly, the communication delay between any two nodes is measured through the ping command, where the communication delay indicates how long it takes to communicate from one node to another.

[0042] It should be noted that, assuming Indicates Node to The communication delay of each node is Indicates Node to The communication delay between two nodes does not change with the direction of information transmission when two nodes communicate. , secondly, , indicating the Each node does not need to communicate with itself, which is equivalent to a communication delay of 0; the maximum computing speed and communication delay are fixed data and will not change with the internal computing tasks of the distributed processing system.

[0043] Further dynamic data acquisition through distributed processing system includes:

[0044] Through the monitoring system deployed in the distributed processing system, the task generation time, task generation node, task size, and remaining processing time of each task at the current moment are obtained;

[0045] It should be noted that the task generation time indicates the moment when the task first appears in the distributed processing system, the task generation node is the node where the task first appears in the distributed processing system, the task size is an estimate of how many operations are required to fully process the task, and the remaining processing time represents an estimate of how long it will take to complete the calculation of the task; secondly, if a task has not yet begun to be processed by the assigned node, the remaining processing time is the ratio of the task size to the maximum computing speed of the node to which the task is assigned.

[0046] At this point, the maximum computing speed of each node and the communication delay between any two nodes are obtained, as well as the task generation time, task generation node, task size, and remaining processing time of each task at the current moment.

[0047] Step 2: Determine the task resolution time at which each new task is processed by any node at the current moment based on the queuing waiting time required for each new task to be processed by any node at the current moment, the communication delay between the task generation node of each new task and any node, and the time required for each new task to be calculated and processed by any node.

[0048] In the existing distributed resource allocation process, the communication delay between nodes and the law of task generation are not fully considered. When allocating new tasks, by monitoring the computing tasks accumulated on each node in the entire distributed processing system, the new computing tasks are allocated to the nodes with fewer accumulated computing tasks for calculation. However, when a node is beginning to generate a large number of computing tasks, the tasks of the node itself have not yet begun to accumulate. At this time, the node still belongs to the node with fewer accumulated computing tasks. The traditional distributed resource management method will allocate the tasks of other nodes to the node, and the subsequent new computing tasks of the node will be allocated to other nodes, which causes the tasks generated by the node to not meet the principle of proximity of node calculation.

[0049] At the same time, as a node generates a large number of computing tasks, the tasks of other nodes continue to occupy the task sorting table of the node, causing the subsequent new tasks of the node to be accepted by more distant nodes, increasing the overall communication delay of the distributed processing system. The node may therefore experience a decrease in response speed, and then the node may become paralyzed, resulting in a decrease in the task processing capacity of the entire distributed processing system.

[0050] Different nodes in the distributed processing system will generate multiple new computing tasks, and resources need to be allocated to the new tasks. When the new tasks are allocated to different nodes, the entire distributed processing system has different processing times for different new tasks. New tasks, New tasks added by A node is generated, and at this time The nodes are arranged in New tasks previously existed Tasks are queued for processing, but There are relatively few tasks accumulated on each node. The node generated New tasks are assigned to nodes to process, then The processing time of the newly added tasks includes: The node and The time it takes for nodes to communicate back and forth , No. Nodes processing The time spent on each task , No. The new task was The time it takes for each node to perform calculations .

[0051] Assume New tasks added from The node transmits to the The time spent in the process of each node is ,exist First The node will continue to process other tasks waiting to be processed. Nodes process existing The time spent on a task is , therefore, if Greater than or equal to , then The processing time of the newly added tasks does not include the Nodes process existing The time spent on each task , on the contrary, if Less than , No. The processing time of the newly added tasks does not include the processing time from The node transmits to the The time spent in the process of each node is .

[0052] Therefore, the The longer it takes to process each new task, the The longer it takes for a new task to be fully solved, the New tasks are assigned to The more unreasonable the first node is, the more likely it is to cause the The first new task is generated A node fails to respond for a long time, resulting in node paralysis.

[0053] All newly generated tasks of all nodes at the current moment are recorded as each newly added task at the current moment;

[0054] The accumulated remaining processing time of all tasks queued before each newly added task at any node at the current moment is used as the queue waiting time before each newly added task is processed by any node at the current moment;

[0055] The calculation formula for the task resolution time of each newly added task at the current moment being processed and completed by any of the nodes is: ,in, For the current moment The task size of the newly added task, For the The maximum computing speed of each node, For the current moment Task generation node for new tasks To the corresponding The communication delay of each node is For the current moment The node will process the completed The newly added tasks are transferred to Task generation node for new tasks The communication delay, For the current moment The new task was The waiting time before a node is processed. is the maximum value function, where and equal.

[0056] It should be noted that the waiting time in the queue indicates the current The new task will be When processing nodes, The length of time that a newly added task has to wait in the queue; secondly, the obtained task resolution time indicates the time taken for the corresponding newly added task to be completely resolved. The longer the task resolution time is, the faster the task resolution time is. New tasks are assigned to After processing the nodes, The longer it takes for a newly added task to be completely solved, the more The more new tasks are assigned to the Nodes for processing.

[0057] At this point, the task resolution time at which each newly added task at the current moment is processed and completed by any of the nodes is obtained.

[0058] Step 3: randomly assign nodes to all newly added tasks at the current moment, and obtain multiple solution allocation vectors as the initial population of the intelligent optimization algorithm; analyze the task solution time described in the solution allocation vector corresponding to each particle, and the speed of node generating tasks, and determine the solution efficiency sequence and generation speed sequence corresponding to each particle respectively; based on the solution efficiency sequence and generation speed sequence, determine the node response balance corresponding to each particle.

[0059] Furthermore, all new tasks should be assigned better distributed computing resources as much as possible to ensure that the task resolution time is short so that the new tasks can be quickly resolved; however, the urgency of new tasks is different. If a node generates new tasks quickly, it means that the node needs to calculate and process a large number of tasks. If a node generates tasks very quickly, it means that more computing resources are needed to process these newly generated tasks. Therefore, the distributed processing system should give priority to allocating more distributed computing resources to nodes with fast task generation speeds to ensure that these nodes can process new tasks in a timely manner and avoid slow responses or node paralysis due to task backlogs.

[0060] Based on the above analysis, the speed of task generation of each node is analyzed, specifically:

[0061] Get the number of tasks generated by each node in multiple moments before the current moment; calculate the task generation speed of each node at the current moment;

[0062] Preferably, in this embodiment, the number of tasks generated by each node within 10 seconds before the current moment is obtained; as other implementation methods, the implementer can set it according to the actual situation; secondly, assuming that the number of tasks generated by each node within 10 seconds before the current moment is The number of tasks generated by a node is , then at the current moment The task generation speed of each node The calculation method is: .

[0063] It should be noted that the faster the task generation speed is, the faster the task generation speed of the corresponding node is. At this time, better distributed computing resources should be allocated to the newly generated tasks of the node to avoid the node's slow task response, which may cause the node to crash.

[0064] Furthermore, for all new tasks in the distributed processing system, it is necessary to allocate all new tasks. In order to select the optimal task allocation scheme, it is necessary to evaluate the task allocation scheme. In the task allocation scheme, if the tasks generated by the nodes with faster task generation speed are solved in a shorter time, it means that the distributed computing resources allocated to the nodes with faster task generation speed by the task allocation scheme are better. After the task allocation scheme is executed, the response time of the new tasks of all nodes is closer, which can effectively avoid the paralysis of a node due to the slow response time, reflecting the better node response balance of the task allocation scheme.

[0065] Further, the step flow chart of the method for obtaining the node response balance corresponding to each particle provided in the embodiment of the present application is as follows: Figure 2 shown.

[0066] Based on this analysis, the time it takes to solve the tasks generated by each node in the task allocation scheme is analyzed, specifically:

[0067] Construct a multidimensional vector with the same number as all newly added tasks at the current moment, and record it as the solution allocation vector;

[0068] Number all nodes, randomly select a node number from all nodes as the first component of the solution allocation vector, randomly select another node from all nodes as the second component of the solution allocation vector, repeat this step to obtain a solution allocation vector; use the same method to use multiple solution allocation vectors as the initial population of the intelligent optimization algorithm;

[0069] Preferably, in this embodiment, 30 scheme allocation vectors are used as the initial population of the intelligent optimization algorithm.

[0070] It should be noted that the first The elements are , indicating that The new task was assigned to Nodes for processing.

[0071] It should be noted that when new tasks are allocated through a certain scheme allocation vector, multiple new tasks may be allocated to the same node. At this time, these new tasks need to queue up and wait for calculation. In this embodiment, the queuing method of the new tasks is: for multiple new tasks that are simultaneously allocated to the same node, they are sorted according to the task generation speed corresponding to the task generation node of the new tasks. The larger the task generation speed, the higher the priority of the corresponding new task in calculation; if the task generation speeds corresponding to the task generation nodes of the new tasks are equal, the multiple new tasks are sorted from first to last according to the node serial numbers corresponding to the task generation nodes from large to small; if the task generation nodes of multiple new tasks are the same, they are sorted according to the order of the task generation times of the new tasks.

[0072] The average of the task solving time of all newly added tasks corresponding to the same task generation node in the solution allocation vector corresponding to each particle and processed by the corresponding node is recorded as the average solving time;

[0073] It should be noted that when calculating the task resolution time for each newly added task in the solution allocation vector to be processed by the corresponding node, the queue waiting time is calculated as follows: Assuming that The new task was assigned to nodes to process, if there are other new tasks, they will be arranged in the first Before the new task is added, the waiting time in the queue is the The sum of the remaining processing time of all tasks before the newly added task.

[0074] It should be noted that, for ease of understanding, assuming that the number of new tasks is 7, the number of nodes is 5, and the allocation vector of a certain solution is , where the first element is Indicates that the first newly added task is assigned to nodes, if the task generation nodes of the second, sixth, and seventh new tasks are all nodes , then calculate the second newly added task to be The time it takes to complete the task processing by each node , calculate the 6th newly added task The time it takes to complete the task processing by each node , calculate the 7th newly added task The time it takes to complete the task processing by each node ,Will , and The mean of the nodes The smaller the average solution time, the smaller the node The faster the generated new tasks are solved, the faster the node The faster the task response speed, the less likely it is that a node will be paralyzed.

[0075] Furthermore, in order to prevent the node from being paralyzed due to the slow response of the task generated by the node, the task allocation scheme is evaluated based on the average solution time and the task generation speed to determine the node response balance, specifically:

[0076] After negative mapping, the average solution time of all task generation nodes in the solution allocation vector corresponding to each particle is normalized to form a solution efficiency sequence;

[0077] Preferably, in this embodiment, the negative mapping process is: calculate the cumulative sum of the maximum and minimum values ​​of the average solution time of all task generation nodes in the solution allocation vector corresponding to each particle; record the difference between the cumulative sum and the average solution time of each task generation node as the relative difference of each task generation node; normalize the relative differences of all task generation nodes in the solution allocation vector corresponding to each particle to form a solution efficiency sequence.

[0078] Normalize the task generation speeds of all nodes in the solution allocation vector corresponding to each particle to form a generation speed sequence;

[0079] Preferably, in this embodiment, the ratio of the relative difference of each task generation node to the sum of the relative differences of all task generation nodes is calculated, recorded as the first ratio, and the first ratio of all task generation nodes is combined into a solution efficiency sequence; correspondingly, the ratio of the task generation speed of each node to the sum of the task generation speeds of all nodes is calculated, recorded as the second ratio, and the second ratios of all nodes are combined into a generation speed sequence, and normalization processing is performed by this method. As other implementation methods, the implementer can adopt other methods of the prior art, such as Min-Max normalization, and this embodiment does not impose any special restrictions on this.

[0080] It should be noted that the task generation node and the node are nodes in the communication network; each node can generate tasks and also calculate and process tasks; The larger the element, the The faster the new tasks generated by the nodes can be solved, the The more nodes there are, the less likely they are to become paralyzed; The larger the element, the The faster the task generation speed of each node is, the more it is necessary to The newly added tasks generated by each node are allocated better distributed computing resources, making the The newly generated tasks generated by each node are quickly resolved.

[0081] Calculating the cross entropy between the solution efficiency sequence and the generation speed sequence corresponding to each particle as the node response balance corresponding to each particle;

[0082] It should be noted that the calculation of cross entropy is a well-known technology and will not be described in detail here. The smaller the node response balance is, the better the distributed computing resources allocated to the new tasks generated by the nodes with fast task generation speed are. When the task is allocated through the scheme allocation vector corresponding to the particle, the task allocation scheme can prevent a single node from responding too slowly, so that the response time of the new tasks generated by each node is balanced, and the node paralysis can be avoided.

[0083] At this point, the node response balance corresponding to each particle is obtained.

[0084] Step 4: Determine the absolute communication consumption of each new task in the solution allocation vector corresponding to each particle according to the time consumed by each new task in the solution allocation vector corresponding to each particle in the round-trip communication transmission between the task generation node and the node corresponding to the task being processed; determine the communication delay proportion corresponding to each particle based on the absolute communication consumption of all new tasks in the solution allocation vector corresponding to each particle and the task solution time; determine the fitness of each particle in the intelligent optimization algorithm based on the node response balance and the communication delay proportion; perform iterative calculation in combination with the intelligent optimization algorithm to obtain the optimal solution allocation vector and allocate the new tasks at the current moment.

[0085] Furthermore, the node balance can be used to screen out task allocation schemes that can balance the response time of new tasks generated by each node, thereby avoiding node paralysis. However, the communication delay in the task allocation scheme may not be taken into account. A certain task allocation scheme may make the response time of the entire distributed computing network to new tasks more balanced, but some new tasks will be assigned to nodes with larger communication delays for calculation, which will lead to a decrease in the overall response time of the entire distributed processing system to the new tasks.

[0086] Based on the above analysis, by analyzing the time consumed by each new task in the communication transmission process when each new task is allocated according to the scheme allocation vector corresponding to each particle, the absolute communication consumption is determined and constrained by the communication delay of the task allocation scheme, specifically:

[0087] The absolute communication cost of each newly added task in the solution allocation vector corresponding to each particle is calculated as follows: ,in, For the The solution allocation vector corresponding to the particle The absolute communication cost of each new task is For the The solution allocation vector corresponding to the particle The number of newly added tasks to be processed corresponds to the Node to Task generation node for new tasks The communication delay between For the The solution allocation vector corresponding to the particle Task generation node for new tasks To The number of newly added tasks to be processed corresponds to the The communication delay between nodes is The solution corresponding to each particle is assigned to the first The node processes The waiting time before adding a new task, is the maximum value function, where .

[0088] It should be noted that, for ease of understanding, it is assumed that the solution allocation vector corresponding to a certain particle is , where the third element corresponds to , indicating that the third newly added task is assigned to nodes, then the third new task corresponds to nodes, if the task generation node of the third newly added task is , then the third new task is generated from the task node Transfer to The communication delay of each node , No. The node will return the third newly added task after processing to the task generation node Communication delay , and the first The waiting time before a node processes the third newly added task is .

[0089] It should be noted that if The larger the The more new tasks are added, the more response time is consumed in communication. If it is less than 0, then remove the The time that a new task should wait in line for processing, and the absolute communication consumption of the actual communication process is 0; if greater than 0, The difference is the increased response time of the communication process. The larger the absolute communication consumption, the more the corresponding new tasks are assigned to nodes with larger communication delays for processing, resulting in a more significant decrease in the overall response time of the distributed processing system for the new tasks.

[0090] Further, based on the absolute communication consumption and the task resolution time, the communication delay ratio is determined to reflect the ratio of the time spent on communication for each newly added task to the time spent on the entire task processing, specifically:

[0091] The mean of the absolute communication consumption of all newly added tasks in the solution allocation vector corresponding to each particle is recorded as the first mean;

[0092] The average of the task solution time of all newly added tasks in the solution allocation vector corresponding to each particle and processed by the corresponding node is recorded as the second average;

[0093] The ratio of the first mean value to the second mean value is used as the communication delay proportion corresponding to each particle;

[0094] It should be noted that the greater the proportion of the communication delay, the greater the proportion of the communication delay in the overall task solution time when the task is allocated according to the scheme allocation vector corresponding to the corresponding particle, and the greater the response delay caused by the scheme allocation vector in the communication, the worse the effect of the task allocation scheme.

[0095] Further, based on the node response balance and the communication delay ratio, the fitness of each particle in the intelligent optimization algorithm is determined to evaluate the pros and cons of task allocation according to the scheme allocation vector corresponding to each particle, specifically:

[0096] The calculation formula for the fitness of each particle in the intelligent optimization algorithm is: ,in, The particle swarm optimization algorithm The fitness of a particle, For the The node response balance corresponding to each particle, For the The communication delay ratio corresponding to each particle is To preset a value greater than 0, avoid or One of the items is 0, so that the parameter item of the node response balance or the communication delay ratio is invalid, resulting in the denominator being 0. In this embodiment, The value is 0.01. As other implementation methods, the implementer can set it according to the actual situation.

[0097] It should be noted that the smaller the node response balance is, the more the task allocation scheme represented by the corresponding particle can balance the response time of the newly added tasks generated by each node, reflecting that the task allocation scheme is better; the smaller the proportion of communication delay is, the smaller the response delay generated by the task allocation scheme represented by the corresponding particle in communication is, reflecting that the task allocation scheme is better. Therefore, the greater the fitness of each particle is, it means that the task allocation scheme represented by the corresponding particle can ensure that the response time of the newly added tasks generated by each node is balanced, and the overall communication delay of the scheme is small, and the task allocation scheme represented by the corresponding particle is better.

[0098] Based on the initial population and the fitness, an intelligent optimization algorithm is iterated to obtain an optimal solution allocation vector and allocate all tasks at the current moment.

[0099] Preferably, in this embodiment, the intelligent optimization algorithm adopts a particle swarm optimization algorithm, wherein the particle swarm optimization algorithm is a well-known technology and will not be described in detail here; wherein, the inertia weight in the particle swarm optimization algorithm is 0.7, the acceleration factor is 1.5, and the maximum number of iterations is 100. Other implementers can set them according to actual conditions.

[0100] It should be noted that the optimal solution allocation vector The element represents the The newly added tasks are assigned to the corresponding nodes for processing, thereby achieving efficient management of distributed resources.

[0101] Based on the same inventive concept as the above method, an embodiment of the present application also provides a distributed resource management system, including a memory, a processor, and a computer program stored in the memory and running on the processor, and when the processor executes the computer program, the steps of any one of the above-mentioned distributed resource management methods are implemented.

[0102] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0103] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the present application. It should be pointed out that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application without departing from the content of the technical solution of the present application, shall fall within the protection scope of the technical solution of the present application.

Claims

1. A distributed resource management method, characterized in that: The method comprises the following steps: Obtain the maximum computing speed of each node in the distributed processing system and the communication delay between any two nodes, as well as the task generation time, task generation node, task size, and remaining processing time of each task at the current moment; record all newly generated tasks of all nodes at the current moment as each newly added task at the current moment; Determine the task resolution time at which each newly added task is processed by any node at the current moment based on the queuing waiting time required for each newly added task to be processed by any node at the current moment, the communication delay between the task generation node of each newly added task and any node, and the time required for each newly added task to be calculated and processed by any node; Randomly assign nodes to all newly added tasks at the current moment, and obtain multiple solution allocation vectors as the initial population of the intelligent optimization algorithm; determine the average solution time according to the task solution time when all newly added tasks corresponding to the same task generation node in the solution allocation vector are processed and completed by the corresponding node, and determine the solution efficiency sequence according to the average solution time; calculate the task generation speed of each node at the current moment according to the task generation time of each task and the number of tasks generated by each node before the current moment, and form a generation speed sequence according to the task generation speed of all nodes in the solution allocation vector; record the cross entropy between the solution efficiency sequence corresponding to the particle and the generation speed sequence as the node response balance corresponding to the particle; determine the absolute communication consumption of each newly added task in the solution allocation vector corresponding to each particle, where the first The solution allocation vector corresponding to the particle Absolute communication cost of each new task The calculation method is: ,in, For the The solution allocation vector corresponding to the particle The number of newly added tasks to be processed corresponds to the Node to Task generation node for new tasks The communication delay between For the The solution allocation vector corresponding to the particle Task generation node for new tasks To The number of newly added tasks to be processed corresponds to the The communication delay between nodes is The solution corresponding to each particle is assigned to the first The node processes The waiting time before adding a new task, is a maximum function; based on the absolute communication consumption of all newly added tasks in the solution allocation vector corresponding to each particle and the task solution time thereof, determining the communication delay proportion corresponding to each particle; Based on the node response balance and the communication delay ratio, determine the fitness of each particle in the intelligent optimization algorithm; perform iterative calculations in combination with the intelligent optimization algorithm to obtain the optimal solution allocation vector and allocate the newly added tasks at the current moment; The determining of the task resolution time at which each newly added task at the current moment is processed by any node includes: Calculate the queuing waiting time of each newly added task before it is processed by any node based on the remaining processing time required to calculate and process the remaining tasks before each newly added task is processed by any node; The current moment The new task was The time it takes to complete the task processing by each node The calculation formula is: ,in, For the current moment The task size of the newly added task, For the The maximum computing speed of each node, For the current moment Task generation node for new tasks To the corresponding The communication delay of each node is For the current moment The node will process the completed The newly added tasks are transferred to Task generation node for new tasks The communication delay, For the current moment The new task was The waiting time before a node is processed. is the maximum value function; The determining of the communication delay proportion corresponding to each particle includes: The mean of the absolute communication consumption of all newly added tasks in the solution allocation vector corresponding to each particle is recorded as the first mean; The average of the task solution time of all newly added tasks in the solution allocation vector corresponding to each particle and processed by the corresponding node is recorded as the second average; The ratio of the first mean value to the second mean value is used as the communication delay proportion corresponding to each particle; The calculation formula for the fitness of each particle in the intelligent optimization algorithm is: ,in, For the The fitness of a particle, For the The node response balance corresponding to each particle, For the The communication delay ratio corresponding to each particle is The default value is greater than 0.

2. A distributed resource management method as claimed in claim 1, characterized in that: The queue waiting time is the cumulative sum of the remaining processing time of all tasks arranged before each newly added task waiting to be processed at any node at the current moment.

3. A distributed resource management method as claimed in claim 1, characterized in that: The obtaining of multiple scheme allocation vectors includes: Construct a multidimensional vector with the same number as all newly added tasks at the current moment, and record it as the solution allocation vector; All nodes are numbered, and a node is randomly assigned to the newly added task corresponding to each dimension in the solution allocation vector to obtain a solution allocation vector; this step is repeated to obtain multiple solution allocation vectors.

4. A distributed resource management method as claimed in claim 1, characterized in that: The method for obtaining the solution efficiency sequence and generation speed sequence corresponding to each particle is: The average of the task solving time of all newly added tasks corresponding to the same task generation node in the solution allocation vector and processed by the corresponding node is recorded as the average solving time; the average solving time of all task generation nodes in the solution allocation vector is negatively mapped and normalized to form a solving efficiency sequence; Based on the task generation time of each task, count the number of tasks generated by each node in multiple moments before the current moment; Calculate the task generation speed of each node at the current moment; The task generation speeds of all nodes in the solution allocation vector are normalized to form a generation speed sequence.

5. A distributed resource management system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of a distributed resource management method as described in any one of claims 1 to 4 are implemented.

Images