GPU cluster NLP task parallelization system and method based on particle swarm optimization

Abstract

The invention provides a GPU cluster NLP task parallelization system and method based on a particle swarm optimization, and the method comprises the steps: carrying out the analysis of an input NLP model based on a task decomposition module, dividing the NLP model into a plurality of subtasks, and collecting and recording the GPU cluster resource state based on a GPU resource management module, a plurality of sub-tasks are mapped on a GPU cluster based on a task allocation module, a plurality of independent particle swarms are constructed, task allocation is optimized by using a particle swarm algorithm based on a particle swarm optimization module, task execution is scheduled according to an optimized task allocation scheme based on a task scheduling module, and task execution is completed. And based on a result processing module, collecting and integrating sub-task calculation results on each GPU, and outputting the sub-task calculation results. According to the method, the NLP training speed is remarkably increased, the resource utilization efficiency is effectively improved, and the method is suitable for being used in a compute-intensive NLP task and particularly has a remarkable effect in a scene where GPU resources are rich and highly parallel computing is needed.

Description

Technical Field

[0001] The present invention relates to the technical field of parallel computing of NLP tasks, and in particular to a system and method for parallelizing NLP tasks based on a particle swarm algorithm GPU cluster. Background Art

[0002] In recent years, natural language processing (NLP) has been widely used in machine translation, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, optical character recognition and other fields. However, due to the large amount of data and high computational complexity of NLP tasks, efficient hardware resources are often required to support model training. GPU clusters have become the preferred platform for NLP tasks due to their advantages in parallel computing. Existing task allocation methods and scheduling strategies are insufficient in the efficiency of GPU cluster resource utilization, which often leads to problems such as load imbalance and long training time. Particle swarm optimization (PSO), as an optimization algorithm based on swarm intelligence, has global search capabilities and can be used for task allocation and resource optimization. Therefore, it is very necessary to design a GPU cluster NLP task parallelization system and method based on particle swarm optimization. Summary of the invention

[0003] In order to overcome the deficiencies of the prior art, the object of the present invention is to provide a system and method for parallelizing NLP tasks in a GPU cluster based on a particle swarm algorithm.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] The present invention provides a GPU cluster NLP task parallelization system based on particle swarm algorithm, comprising:

[0006] Task decomposition module, which is used to analyze the input NLP model and divide it into several subtasks;

[0007] GPU resource management module, used to collect and record GPU cluster resource status;

[0008] The task allocation module is used to map several subtasks on the GPU cluster to build multiple independent particle swarms;

[0009] Particle swarm optimization module, used to optimize task allocation using particle swarm algorithm;

[0010] The task scheduling module is used to schedule task execution according to the optimized task allocation plan;

[0011] The result processing module is used to collect and integrate the subtask calculation results on each GPU and output them.

[0012] The present invention also provides a method for parallelizing NLP tasks in a GPU cluster based on a particle swarm algorithm, comprising:

[0013] Step 100: Analyze the input NLP model based on the task decomposition module and divide it into several subtasks;

[0014] Step 200: Collect and record GPU cluster resource status based on the GPU resource management module;

[0015] Step 300: Mapping a number of subtasks onto a GPU cluster based on a task allocation module to construct multiple independent particle swarms;

[0016] Step 400: Optimizing task allocation using a particle swarm algorithm based on a particle swarm optimization module;

[0017] Step 500: Scheduling task execution based on the optimized task allocation scheme based on the task scheduling module;

[0018] Step 600: Collect and integrate the subtask calculation results on each GPU based on the result processing module, and output them.

[0019] Preferably, in step 100, the input NLP model is analyzed based on the task decomposition module and divided into several subtasks, specifically:

[0020] Analyze the input NLP model based on the task decomposition module and divide it into multiple subtasks according to the analysis results;

[0021] Analyze the dependencies between subtasks, determine the order of subtask execution and dependent data flow, and establish a task dependency graph.

[0022] Preferably, in step 200, the GPU cluster resource status is collected and recorded based on the GPU resource management module, specifically:

[0023] Initialize the resources of the GPU cluster, collect information about each device in the GPU cluster, and record the resource status of each device.

[0024] Preferably, in step 300, a plurality of subtasks are mapped on a GPU cluster based on a task allocation module to construct a plurality of independent particle swarms, specifically:

[0025] According to the characteristics of subtasks and GPU cluster devices and the task dependency graph, several subtasks are mapped on the devices of the GPU cluster. A variety of task allocation strategies are generated according to different allocation processes. Each task allocation strategy represents an independent particle swarm. Each task allocation strategy contains multiple task allocation schemes, and each task allocation scheme is represented as a particle in the corresponding particle swarm.

[0026] In each particle group, the position and velocity of the particles are initialized, and the initial tasks are randomly assigned to complete the particle initialization.

[0027] Preferably, in step 400, the task allocation is optimized using a particle swarm algorithm based on a particle swarm optimization module, specifically:

[0028] Design fitness functions according to different needs;

[0029] Set up interaction mechanisms;

[0030] Perform iterative updates to obtain the task allocation solution for each task allocation strategy.

[0031] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0032] The present invention provides a system and method for parallelizing NLP tasks in a GPU cluster based on a particle swarm algorithm. The system includes a task decomposition module, a GPU resource management module, a task allocation module, a particle swarm optimization module, a task scheduling module, and a result processing module. The method includes analyzing an input NLP model based on the task decomposition module and dividing it into a plurality of subtasks, collecting and recording the GPU cluster resource status based on the GPU resource management module, mapping a plurality of subtasks on the GPU cluster based on the task allocation module, constructing a plurality of independent particle swarms, optimizing task allocation using a particle swarm algorithm based on the particle swarm optimization module, scheduling task execution according to an optimized task allocation scheme based on the task scheduling module, and collecting and integrating the subtask calculation results on each GPU based on the result processing module, and outputting them. The present invention includes the following advantages:

[0033] 1. Set up a multi-particle swarm structure: By decomposing the task allocation into multiple particle swarms, each particle swarm focuses on different optimization goals, and achieves the global optimal solution through the information interaction mechanism. The multi-particle swarm structure and task migration strategy can improve the load balancing of the GPU cluster and significantly reduce the training time. The multi-swarm particle swarm algorithm can better perform global search and optimization and avoid falling into the local optimum;

[0034] 2. Set adaptive dynamic scheduling: The system can adaptively adjust the allocation strategy according to the GPU load status, enhance adaptability and task response speed, enable the system to have good responsiveness when the task load changes, and optimize task execution efficiency;

[0035] 3. Set up a feedback learning mechanism: After the task is executed, the parameter settings of the particle swarm algorithm are optimized through analysis and self-learning, and the intelligence level of the system and the accuracy of task allocation are gradually improved;

[0036] 4. It significantly improves the NLP training speed and effectively improves resource utilization efficiency. It is suitable for use in computationally intensive NLP tasks, especially in scenarios with abundant GPU resources and requiring highly parallel computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0038] Figure 1 A flow chart of a method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The purpose of the present invention is to provide a system and method for parallelizing NLP tasks in a GPU cluster based on a particle swarm algorithm, which significantly improves the NLP training speed and effectively improves resource utilization efficiency. The system and method are suitable for use in computationally intensive NLP tasks, especially in scenarios with abundant GPU resources and requiring highly parallel computing.

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] The present invention provides a GPU cluster NLP task parallelization system based on particle swarm algorithm, comprising:

[0043] Task decomposition module, which is used to analyze the input NLP model and divide it into several subtasks;

[0044] GPU resource management module, used to collect and record GPU cluster resource status;

[0045] The task allocation module is used to map several subtasks on the GPU cluster to build multiple independent particle swarms;

[0046] Particle swarm optimization module, used to optimize task allocation using particle swarm algorithm;

[0047] The task scheduling module is used to schedule task execution according to the optimized task allocation plan;

[0048] The result processing module is used to collect and integrate the subtask calculation results on each GPU and output them.

[0049] like Figure 1 As shown, the present invention also provides a method for parallelizing NLP tasks in a GPU cluster based on a particle swarm algorithm, comprising:

[0050] Step 100: Analyze the input NLP model based on the task decomposition module and divide it into several subtasks;

[0051] Step 200: Collect and record GPU cluster resource status based on the GPU resource management module;

[0052] Step 300: Mapping a number of subtasks onto a GPU cluster based on a task allocation module to construct multiple independent particle swarms;

[0053] Step 400: Optimizing task allocation using a particle swarm algorithm based on a particle swarm optimization module;

[0054] Step 500: Scheduling task execution based on the optimized task allocation scheme based on the task scheduling module;

[0055] Step 600: Collect and integrate the subtask calculation results on each GPU based on the result processing module, and output them.

[0056] In step 100, the input NLP model is analyzed based on the task decomposition module and divided into several subtasks, specifically:

[0057] The input NLP model is analyzed based on the task decomposition module, and divided into multiple subtasks according to the analysis results (for example, according to the model's hierarchy or computing unit, each hierarchy or computing unit (such as convolutional layer, fully connected layer, etc.) represents a subtask) to facilitate parallel computing;

[0058] Analyze the dependencies between subtasks, determine the order of subtask execution and dependent data flow, and establish a task dependency graph so that the data flow and calculation order can be clarified during parallel execution;

[0059] In addition, the present invention also provides an embodiment, which can adjust the task granularity. According to the performance characteristics of different GPUs (such as computing power and memory capacity), the task granularity can be dynamically adjusted. Tasks with large computational complexity can be further subdivided so that they can be allocated to multiple GPUs for processing, thereby improving the overall parallel efficiency.

[0060] In step 200, the GPU cluster resource status is collected and recorded based on the GPU resource management module, specifically:

[0061] Initialize the resources of the GPU cluster, collect the information of each device in the GPU cluster (including computing power, memory capacity, bandwidth, current load and other parameters), and record the resource status of each device.

[0062] First, the particle swarm optimization algorithm is introduced:

[0063] Dr. Eberhart and Dr. Kennedy were deeply inspired by the social behavior of bird flocks foraging, and simulated a population-based random search method, which is an optimization technique based on global probability. This method can use the interaction between particles when searching for the best location in a complex exploration space. Therefore, the algorithm is also called Particle Swarm Optimization (PSO for short), and is also included in the group intelligence algorithm.

[0064] Among the heuristic optimization algorithms, the PSO algorithm is a typical one. The advantages of this algorithm are: easy operation, relatively simple parameters and principles, and rapid convergence. It also has strong global optimization capabilities. As a new type of evolutionary computing technology, this algorithm mainly relies on the design and implementation of cluster intelligence. At the same time, strong robustness is another outstanding advantage of the particle optimization algorithm, which enables the algorithm to effectively reduce the error rate caused by a single fault in the entire problem-solving process, and the probability of obtaining the global optimal solution is also increased accordingly. Therefore, the proposal of the particle swarm optimization algorithm has attracted the attention and extensive exploration of researchers, and has been widely used in scientific research and engineering applications. However, its shortcomings are also obvious. When searching for partial functions, multiple local extreme point values ​​can be obtained, which usually falls into the local optimal solution; global search is a unique feature of the particle swarm algorithm, and the basic particle swarm algorithm cannot accurately obtain the search effect of the particle on the global optimal point. The emergence of this situation makes the basic particle swarm algorithm unsuitable for calculating high-dimensional optimization problems with multiple local extreme points, so the basic particle swarm algorithm needs to be improved urgently.

[0065] Since the particle velocity v i and position x i Constantly change and update the speed at the next moment, define the values ​​of c1, c2 and w, define the population size range i, and the formula for updating the speed and position is:

[0066] Speed ​​conversion formula:

[0067] v i+1 =w×v i +c1×rand×(p_best i -xi )+c2×rand×(p_best i -x i )

[0068] Position transformation formula:

[0069] x i+1 =x i +v i

[0070] In the formula, w is the inertia factor. We can control the process of the algorithm by controlling the adaptive change of w to ensure the global and local search capabilities of the particles. c is the learning factor. The size of the learning factors c1 and c2 determines the impact of particle self-cognition and social cognition on the particle trajectory, reflecting the degree of information exchange between particles in the group. rand is a random number between (0, 1) to increase the randomness of flight. i and x i Respectively represent the speed and position of the i-th dimension, p_best i 、g_best i They respectively represent the historical optimal i-th dimension value of a particle and the global optimal i-th dimension value of the entire population.

[0071] The present invention improves the traditional multi-objective particle swarm optimization algorithm, including:

[0072] (1) Optimization of inertia weight

[0073] In the particle swarm optimization algorithm, w represents the inertia weight. The control of the particle swarm algorithm process is often controlled by w to ensure the global and local search capabilities of the particles. In order to enhance the global search capability, w can be assigned a larger value; on the contrary, in order to reduce the global search capability of the particles and enhance the local search capability of the particles, w is assigned a smaller value. The strength of the global and local search capabilities of the particles can be balanced by constantly changing the value of the inertia weight w. In the early stage of the search, a larger w value can be selected to ensure the global search capability of the particles; in the later stage of the search, it is mainly used to enhance the local search capability of the particles, and a smaller w value should be used at this time. Therefore, in the present invention, the value of w is dynamically updated to coordinate the search capability of the particles, as follows:

[0074]

[0075] In the formula, iter and iter-max represent the number of iterations and the maximum number of iterations, usually w max =0.9, w min = 0.4, the global and local search capabilities of the particles can be adjusted by dynamically changing w;

[0076] (2) Optimization of learning factors

[0077] The learning factor c can well reflect the degree of information exchange between particles in the group. c1 mainly emphasizes the self-learning ability of particles, that is, the self-awareness of particles; c2 mainly emphasizes the social learning ability of particles, that is, the social perception of particles; both will affect the trajectory of particles. In the initial optimization iteration, the self-learning ability of particles is relatively weak, contrary to the social learning ability, so the global search ability is strong and the search can be carried out quickly and easily. In the final stage of the optimization iteration, the self-learning ability of particles is strong, contrary to the social learning ability, and the global optimal solution can be obtained more accurately, which is helpful for global refinement. Usually, the value of c1 is related to the inertia weight w, and will decrease as w decreases, as follows:

[0078] c1=0.5×w 2 +3×w

[0079] The value of c2 is related to c1 and increases as c1 decreases, which is:

[0080] c2=4-c1

[0081] (3) Speed ​​constraint range optimization

[0082] Because the particle swarm algorithm is simple, portable, and has fast convergence speed, it has been widely used in solving many optimization problems. However, in the particle search process, the search speed directly affects the search for the optimal solution. max If v is too large, the algorithm search will often be not detailed enough and it is easy to skip the optimal solution. max If it is too small, the search will be too slow and it will be easy to fall into the local optimal solution. Therefore, the reasonable adjustment of the speed of the particle swarm flight is a very important part of the solution process.

[0083] The maximum and minimum values ​​of the speed update are determined by the displacement update, which are:

[0084] v max =x max -x min

[0085] v min =x min -x max

[0086] (4) External document storage strategy optimization

[0087] Aiming at the shortcomings of traditional particle swarm algorithm that it is easy to fall into local optimum and the search method is not precise enough, the selection of parameters in the algorithm and the population initialization limit are improved, and external documents commonly used in multi-objective optimization problems are cited to store the global optimal solution. In the process of solving multi-objective optimization problems, how to find suitable particles in the algorithm to guide the flight and save the optimal solution found by the particles from the beginning to the end of the whole process is the main problem that needs to be solved urgently at this stage. Therefore, by introducing external documents commonly used in multi-objective optimization problems to store Pareto optimal solutions, the external storage document determines how the global optimal choice is, and its scale is often set by oneself. Under normal circumstances, the scale of the external document is smaller than the population size. In the present invention, the "crowding distance" is used to update the particle state in the external document.

[0088] In multi-objective optimization problems, a set of Pareto optimal solutions are often obtained, which are stored according to the domination and non-domination relationship between particles. Usually, the obtained non-dominated solutions are stored in an external document. If a particle is generated during the iteration process and this particle can dominate another particle in the external document, the particle stored in the external document is replaced. The entire algorithm process of finding the global optimal solution is completed by the external document.

[0089] When the number of non-dominated solutions in the external document is greater than the size of the external document, how to update the non-dominated solutions in the external document. The size of the particle crowding distance reflects the distribution of particles in the spatial range of the solution. The larger the crowding distance, the smaller the number of particles distributed in the local range of the solution space. In order to obtain a final solution set with better distribution and convergence, the present invention uses the crowding distance metric to maintain and update the external document.

[0090] In the process of initializing the particle swarm, for particles that cannot meet the constraints, the following formula is usually used to find the particles x that do not meet the constraints: dis The farthest particle x that satisfies the constraint sat , and the particle x sat The position is assigned to particle x dis , and take it as the latest initial individual best historical position:

[0091]

[0092] In the formula, D represents the distance, dim represents the dimension of the particle position, and x i-dis and x i-sat They represent the values ​​of particles that do not satisfy the constraints and particles that satisfy the constraints in the i-th dimension respectively.

[0093] In step 300, a number of subtasks are mapped onto the GPU cluster based on the task allocation module to construct multiple independent particle swarms, specifically:

[0094] According to the characteristics of subtasks and GPU cluster devices and the task dependency graph, several subtasks are mapped to the devices of the GPU cluster, wherein a variety of task allocation strategies are generated according to different allocation processes (for example, a particle swarm focusing on load balancing, a particle swarm optimizing data transmission delay, etc.), each task allocation strategy represents an independent particle swarm, each task allocation strategy contains multiple task allocation schemes (for example, a subtask is allocated to a cluster device), and each task allocation scheme is represented as a particle in the corresponding particle swarm;

[0095] In each particle swarm, the particle positions (i.e., task allocation scheme) and velocities are initialized, initial tasks are randomly assigned, and particle initialization is completed to ensure that different particle swarms have differentiated initial solutions.

[0096] In step 400, the task allocation is optimized using a particle swarm algorithm based on a particle swarm optimization module, specifically:

[0097] Design the fitness function according to different requirements, specifically:

[0098] Design the fitness function according to specific needs, including:

[0099] Computational capacity fitness: measures the performance of the task allocation scheme in terms of computational efficiency, such as the expected time for task execution. Shorter task completion time will increase the fitness value.

[0100] Load balancing fitness: measures the load distribution of each GPU device to make the task distribution plan more even. The plan with a higher load balancing degree has a higher fitness value.

[0101] Data transmission fitness: evaluates the delay and overhead of data transmission in task allocation. The allocation scheme with lower data transmission delay has higher fitness;

[0102] Set the interaction mechanism, specifically:

[0103] Set the interaction mechanism of particle swarms, and exchange population information every certain number of iterations. Each particle swarm can exchange the current local best solution and diffuse the better solution among different swarms, making it easier for the overall algorithm to converge to the global optimal solution.

[0104] Perform iterative updates to obtain the task allocation scheme for each task allocation strategy, specifically:

[0105] In each iteration, the speed and position of each particle are updated, and finally the task allocation scheme for each task allocation strategy is obtained;

[0106] The overall optimization steps based on the improved particle swarm algorithm are summarized as follows:

[0107] S1: Initialization of population particles, inputting required parameters, including displacement, velocity, variable constraint range, population size, etc.;

[0108] S2: Initialize the global optimal solution G-best and the individual optimal solution P-best;

[0109] S3: Initialize the external document, set the maximum size M of particles that the external document can store, and make it empty, the current iteration number t = 0;

[0110] S4: Find all non-dominated solutions through the improved particle swarm algorithm and store them in an external document;

[0111] S5: The preset maximum scale M determines whether the particles stored in the external document exceed the range of M. If it exceeds the maximum scale M, a part of the solution with high density is deleted according to the crowding distance formula. If it exceeds the maximum scale limit, S6 operation is performed;

[0112] S6: Update the current global optimal position G-best and individual optimal position P-best;

[0113] S7: Update the current displacement x according to the speed and displacement formula i and speed v i ;

[0114] S8: Determine whether the termination condition of the algorithm is met or the maximum number of iterations has been reached. If both conditions are not met, return to S3; otherwise, the loop ends, the solution set in the external document (i.e., the task allocation plan for each task allocation strategy) is input, and the algorithm ends.

[0115] The present invention also provides an embodiment in which iterative optimization based on adaptive dynamic scheduling can be added, specifically:

[0116] Adaptive inertia weight adjustment: In each iteration, the system will adaptively adjust the inertia weight according to the current GPU cluster load status. If the system load is too high, the inertia weight will be reduced to enable the particle swarm to quickly focus on the current optimal solution. If the system load is low, the inertia weight will be increased to explore a larger solution space.

[0117] Task migration strategy: When it is detected that a GPU resource load is too high, the intelligent scheduling module will start the task migration strategy, reallocate some tasks to GPUs with lower loads, and simultaneously update the task allocation plan to avoid resource bottlenecks;

[0118] Dynamic task allocation update: According to the fitness feedback of each particle, the task allocation plan is adaptively updated, the fitness is recalculated in each iteration, and the plan with the highest fitness is used as the current optimal task allocation plan to ensure that the allocation plan is gradually optimized;

[0119] In step 500, the task scheduling module schedules the task execution according to the optimized task allocation scheme, specifically:

[0120] Task mapping and execution: According to the current optimized task allocation scheme, each subtask is mapped to the corresponding GPU device. Each GPU executes its assigned subtasks independently and in parallel, reducing the waiting time between subtasks.

[0121] Real-time load monitoring: During task execution, the system continuously monitors GPU resource utilization and task execution progress. When a drastic change in the load of a GPU is detected (for example, the execution time of a task exceeds expectations), the system will automatically reallocate and dynamically adjust the allocation of unfinished tasks to ensure efficient operation of the cluster.

[0122] Fault tolerance mechanism: If a GPU device fails, the system will automatically identify and migrate the affected tasks to other GPU devices to ensure the integrity and stability of task execution.

[0123] In step 600, the result processing module collects and integrates the subtask calculation results on each GPU and outputs them, specifically:

[0124] Result aggregation: After all subtasks are completed, the system automatically aggregates the calculation results of each GPU and reconstructs the output results of the complete NLP model;

[0125] Performance analysis: collect statistics on the performance data of each task execution, including task execution time, GPU resource utilization, data transmission efficiency and other parameters, analyze and generate performance reports;

[0126] Feedback learning mechanism: Based on the collected performance data, the system uses the NLP model for adaptive learning, takes the key parameters in the task allocation process as training data, optimizes the key parameter settings in the particle swarm algorithm through feedback learning, and provides more accurate allocation strategies for subsequent tasks.

[0127] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0128] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A GPU cluster NLP task parallelization system based on particle swarm algorithm, characterized by: include: Task decomposition module, which is used to analyze the input NLP model and divide it into several subtasks; GPU resource management module, used to collect and record GPU cluster resource status; The task allocation module is used to map several subtasks on the GPU cluster to build multiple independent particle swarms; Particle swarm optimization module, used to optimize task allocation using particle swarm algorithm; The task scheduling module is used to schedule task execution according to the optimized task allocation plan; The result processing module is used to collect and integrate the subtask calculation results on each GPU and output them.

2. A method for parallelizing NLP tasks on GPU cluster based on particle swarm algorithm, characterized in that: include: Step 100: Analyze the input NLP model based on the task decomposition module and divide it into several subtasks; Step 200: Collect and record GPU cluster resource status based on the GPU resource management module; Step 300: Mapping a number of subtasks onto a GPU cluster based on a task allocation module to construct multiple independent particle swarms; Step 400: Optimizing task allocation using a particle swarm algorithm based on a particle swarm optimization module; Step 500: Scheduling task execution based on the optimized task allocation scheme based on the task scheduling module; Step 600: Collect and integrate the subtask calculation results on each GPU based on the result processing module, and output them.

3. The method according to claim 2, characterized in that In step 100, the input NLP model is analyzed based on the task decomposition module and divided into several subtasks, specifically: Analyze the input NLP model based on the task decomposition module and divide it into multiple subtasks according to the analysis results; Analyze the dependencies between subtasks, determine the order of subtask execution and dependent data flow, and establish a task dependency graph.

4. The method according to claim 2, characterized in that: In step 200, the GPU cluster resource status is collected and recorded based on the GPU resource management module, specifically: Initialize the resources of the GPU cluster, collect information about each device in the GPU cluster, and record the resource status of each device.

5. The method according to claim 2, characterized in that: In step 300, a number of subtasks are mapped onto the GPU cluster based on the task allocation module to construct multiple independent particle swarms, specifically: According to the characteristics of subtasks and GPU cluster devices and the task dependency graph, several subtasks are mapped on the devices of the GPU cluster. A variety of task allocation strategies are generated according to different allocation processes. Each task allocation strategy represents an independent particle swarm. Each task allocation strategy contains multiple task allocation schemes, and each task allocation scheme is represented as a particle in the corresponding particle swarm. In each particle group, the position and velocity of the particles are initialized, and the initial tasks are randomly assigned to complete the particle initialization.

6. The method according to claim 2, characterized in that In step 400, the task allocation is optimized using a particle swarm algorithm based on a particle swarm optimization module, specifically: Design fitness functions according to different needs; Set up interaction mechanisms; Perform iterative updates to obtain the task allocation solution for each task allocation strategy.

Images