Edge calculation method and device for distributed large model pipeline parallel training

Abstract

The invention discloses an edge calculation method and device for distributed large model pipeline parallel training. The implementation process of the method covers several key steps, and aims at efficiently processing complex deep learning models and mass data. Firstly, a system receives a large model training task, and the task carries key information including the category of a large model, the detailed structure of each layer and the size of mini-batch. And then, the system carries out deep analysis according to the information, and generates a pipeline parallel training scheduling strategy specially aiming at the distributed large model, so that the training process is optimized, and efficient utilization of resources is ensured. Finally, the generated scheduling strategy is sent to edge computing nodes, and the nodes are guided to execute large model training tasks. Through the series of elaborately designed steps, the method not only can effectively realize the pipeline parallel training oriented to the distributed large model, but also gives full play to the advantages of edge calculation, thereby remarkably improving the overall training efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of network communication technology, and in particular to an edge computing method and device for distributed large model pipeline parallel training. Background Art

[0002] In recent years, large multimodal language models have attracted extensive attention from industry and academia due to their outstanding performance in understanding and generating multimodal data such as text, images, speech, and video. However, since these large AI models usually contain more than one billion ultra-large-scale parameters, training them in the cloud faces many challenges.

[0003] First, the massive data generated by the Internet of Things needs to pass through the access network, core network and the Internet to reach the remote cloud computing center, which causes large model training tasks to experience extremely long network delays and is difficult to meet the low-latency requirements of IoT applications. Secondly, the remote transmission of massive sensor data will incur high communication bandwidth costs. In addition, the way of transmitting IoT data to the cloud computing center for processing via the Internet is prone to data privacy and security issues. Finally, the ultra-large-scale parameter storage of billions of levels also poses serious challenges to the memory management of cloud computing nodes.

[0004] To solve these problems, edge computing has emerged as an emerging computing paradigm. In edge computing, the data generated by the Internet of Things is migrated to computing nodes at the edge of the network for processing. This local computing method effectively reduces data transmission delays and improves data privacy and security. However, the computing and storage capabilities of edge computing nodes are relatively limited, and a single node is difficult to undertake large model training tasks with hundreds of millions of parameters. In addition, existing edge computing is also faced with resource constraints. The computing power and storage space of edge nodes are usually insufficient to effectively process complex deep learning models and massive data, which leads to low training efficiency, thus affecting the overall performance and real-time performance of training. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the above-mentioned deficiencies in the prior art and propose an edge computing method and device for distributed large model pipeline parallel training. The method can effectively improve the computing power and storage space of edge nodes through reasonable allocation, so as to effectively process complex deep learning models and massive data, thereby improving the efficiency of model training.

[0006] In a first aspect, the present invention provides an edge computing method for distributed large model pipeline parallel training, the method comprising the following steps:

[0007] Step K1: receiving a large model training task; the large model training task carries task information, and the task information includes large model category, large model layer information, and mini-batch size;

[0008] Step K2: generating a pipeline parallel training scheduling strategy for a distributed large model according to the large model category, the large model layer information, and the mini-batch size;

[0009] The pipeline parallel training scheduling strategy of the distributed large model is formulated based on the candidate terminal subset sequence and the maximum number of divisible stages, combined with the goal of minimizing training delay;

[0010] Step K3: Send the pipeline parallel training scheduling strategy to the edge computing node so that the edge computing node executes the large model training task, thereby realizing edge computing for distributed large model pipeline parallel training.

[0011] Furthermore, the step K2 specifically includes the following steps:

[0012] Step K21: Determine a distributed large model training candidate terminal subset sequence;

[0013] Step K22: determining the maximum number of divisible stages of large model training according to the large model category, the large model layer information, the mini-batch size, and the distributed large model training candidate terminal subset sequence;

[0014] Step K23: Based on the goal of minimizing the training time delay and the maximum number of divisible stages of the large model training, generate the number of pipeline parallel computing stages, pipeline segmentation decisions, computing power allocation scheduling decisions and batch split number of the large model training task, that is, form a pipeline parallel training scheduling strategy for the distributed large model.

[0015] Furthermore, the step K21 specifically includes the following steps:

[0016] Divide the IoT terminal set into a finite number of subsets, and arrange the finite number of subsets in descending order according to computing power to obtain a subset vector;

[0017] Based on the network topology of the edge computing system, find the target transmission path corresponding to each subset vector;

[0018] The data transmission rate of any link in the target transmission path is the maximum data transmission rate from the source terminal to the target terminal of the link;

[0019] The subset vector corresponding to the target transmission path is used as a candidate terminal subset sequence for distributed large model training.

[0020] Furthermore, the step K22 specifically includes the following steps:

[0021] Step K221: Determine the number of layers of the large model based on the large model category and the large model layer information;

[0022] Step K222: Calculate the total computational effort C of the large model based on the number of layers of the large model and the size of the mini-batch. total ; and, training a candidate terminal subset sequence according to the distributed large model to calculate the total computing power;

[0023] The total computational effort of the large model is C total The calculation formula is as follows (1):

[0024]

[0025] In formula (1), B is the size of mini-batch;

[0026] L is the number of layers of the large model;

[0027] is the computational amount of a single training task at the i-th layer;

[0028] The total computing power P total The calculation formula is as follows (2):

[0029]

[0030] In formula (2),

[0031] N represents the number of computing nodes in the distributed large model training candidate terminal subset sequence;

[0032] P n Indicates the computing power of the nth computing node;

[0033] Step K223: Set the initial stage number Stage1 and calculate the average calculation amount C stage ;

[0034] Order average calculation amount C stage The calculation formula is as follows (3):

[0035] C stage =C total / Stage1;

[0036] Step K224: Calculate the amount C based on the order average stage and total computing power P total , calculate the computation time T of each large model training stage stage ;

[0037] The computation time T of the large model training phase stage The calculation formula is as follows (4):

[0038]

[0039] In formula (4),

[0040] T comm Indicates the communication delay between nodes; max() indicates the maximum value function;

[0041] Step K225: Based on the total calculation amount C of the large model total 、Total computing power P total , and the computation time T of the large model training phase stage , calculate the number of divisible stages S of the target targe ;

[0042] The calculation formula for the number of divisible stages of the target is as follows (5):

[0043]

[0044] In formula (5), int() represents the rounding function;

[0045] Step K226: Compare the initial number of stages S1 with the target number of divisible stages S targe :

[0046] If the initial stage number Stage1 is greater than the target divisible stage number S targe , then reduce the initial stage number, repeat steps S223 to S226 until the initial stage number Stage1 and the target divisible stage number S targe equal;

[0047] If the initial stage number Stage1 is less than the target divisible stage number S targe , then increase the initial stage number, and repeat steps S223 to S226 until the initial stage number Stage1 and the target divisible stage number S targe equal;

[0048] If the initial stage number Stage1 and the target divisible stage number S targe If they are equal, then the target divisible stage number S is set targe is the maximum number of divisible stages, which determines the maximum number of divisible stages for large model training.

[0049] Furthermore, the step K23 specifically includes the following steps:

[0050] Step K231: Set the current stage number of the large model pipeline parallel computing to the initialization value 1, set the current terminal subset sequence number to the initialization value 0, set the current distributed computing terminal set to the initialization value empty set, initialize the current batch split number to the initialization value 1, set the current mini-batch training delay to the initialization value infinity, and set the current pipeline split decision and the current computing power allocation scheduling decision to empty;

[0051] Step K232: Determine the candidate stage number and candidate calculation terminal vector;

[0052] The candidate stage number is the number of terminals between the first subset and the subset with the current terminal subset number plus 1 in the distributed large model training candidate subset sequence; the candidate calculation terminal vector includes all terminals from the first subset to the subset with the current terminal subset number plus 1;

[0053] Step K233: Determine whether the number of candidate stages is less than or equal to the maximum number of divisible stages:

[0054] If the number of candidate stages is greater than the maximum number of divisible stages, the process proceeds to step S238; if the number of candidate stages is less than or equal to the maximum number of divisible stages, the process proceeds to step S234;

[0055] Step K234: Pipeline segmentation is performed on the large model with the number of candidate stages as the number of segmentation, and the sub-models of each stage obtained by segmentation are matched to the terminals in the candidate computing terminal vector in pipeline order, so as to obtain a joint decision of pipeline segmentation and computing power allocation and scheduling;

[0056] Step K235: Determine the local optimal batch split number and the local optimal mini-batch training delay;

[0057] Step K236: Determine whether the local optimal mini-batch training delay is less than the current mini-batch training delay:

[0058] If the local optimal mini-batch training delay is less than the current mini-batch training delay, proceed to step K237; if the local optimal mini-batch training delay is not less than the current mini-batch training delay, proceed to step K238;

[0059] Step K237: Update the current stage number of the large model pipeline parallel calculation to the candidate stage number, update the current pipeline segmentation decision to the large model pipeline segmentation decision, increase the current terminal subset sequence number by 1, update the current distributed computing terminal set to the candidate computing terminal set, update the current computing power allocation scheduling decision to the computing power allocation scheduling decision, update the current batch split number to the local optimal batch split number, update the current mini-batch training delay to the local optimal mini-batch training delay, and then jump to step K232 again until the local optimal mini-batch training delay is not less than the optimal mini-batch training delay;

[0060] Step K238: Set the current number of stages of large model pipeline parallel computing as the optimal number of stages of large model pipeline parallel computing; and, set the current pipeline segmentation decision as the optimal pipeline segmentation decision; and, set the current distributed computing terminal set as the optimal distributed computing terminal set; and, set the current batch segmentation number as the optimal batch segmentation number.

[0061] Furthermore, in step K235, determining the local optimal batch split number and the local optimal mini-batch training delay specifically includes the following steps:

[0062] Step A1: Determine the splittable set of the mini-batch training set based on the mini-batch size;

[0063] Step A2: taking the element size in the divisible set as the batch split number, evenly splitting the mini-batch to obtain different batch split numbers; and evaluating the mini-batch training delay corresponding to the batch split number;

[0064] Step A3: Set the local optimal batch split number to the batch split number corresponding to the minimum mini-batch training delay, and set the local optimal mini-batch training delay to the minimum mini-batch training delay.

[0065] Furthermore, the step A1 specifically includes the following steps:

[0066] Step A11: Calculate the power M with base 2 according to the mini-batch size N;

[0067] M = ceiling(log2(N), 1);

[0068] ceiling() represents the rounding function, and its rounding base is 1;

[0069] log2(N) represents the logarithm of N with base 2;

[0070] Step A12: According to the power M, determine the divisible set of the mini-batch training set as {2 0 , 2 1 , ..., 2 M}.

[0071] Furthermore, in step K234, the joint decision of pipeline segmentation and computing power allocation scheduling is obtained by matching the micro-batch in the mini-batch and the terminal in the candidate computing terminal vector, and the obtaining process specifically includes the following steps:

[0072] Step B1: Split the mini-batch into m micro-batches;

[0073] The size of each micro-batch is equal, and the sum of the data size of m micro-batches is equal to the data size of the mini-batch;

[0074] Step B2: According to the scheduling rules, match the micro-batch in the mini-batch and the terminal in the candidate computing terminal vector;

[0075] The scheduling rules include:

[0076] Micro-batches are scheduled sequentially; or,

[0077] The training of each micro-batch is distributed to the IoT terminals corresponding to each stage according to the large model pipeline segmentation method and trained in sequence; or,

[0078] Backpropagation starts only after the forward pass of all micro-batches in all micro-batch sequences is completed; or,

[0079] After all micro-batches in the micro-batch sequence have completed backpropagation, pipeline refresh is performed to synchronize the gradients of the large model; or,

[0080] When a terminal in the distributed computer terminal set is idle, once it receives a micro-batch stage training task from an upstream terminal or there is a waiting micro-batch stage training task in the queue, it executes the stage training subtask in the order of the micro-batch in the mini-batch.

[0081] In a second aspect, the present invention provides an edge computing device for distributed large model pipeline parallel training, the device comprising:

[0082] A receiving unit, configured to receive a large model training task; the large model training task carries task information, wherein the task information includes a large model category, large model layer information, and a mini-batch size;

[0083] A generating unit, connected to the receiving unit, for generating a pipeline parallel training scheduling strategy for a distributed large model according to the large model category, the large model layer information, and the mini-batch size;

[0084] The pipeline parallel training scheduling strategy of the distributed large model is formulated based on the candidate terminal subset sequence and the maximum number of divisible stages, combined with the goal of minimizing training delay;

[0085] A sending unit is connected to the generating unit and is used to send the pipeline parallel training scheduling strategy to the edge computing node so that the edge computing node executes the large model training task, thereby realizing edge computing for distributed large model pipeline parallel training.

[0086] Furthermore, the generating unit comprises:

[0087] A first determination module is used to determine a distributed large model training candidate terminal subset sequence;

[0088] a second determination module, connected to the first determination module, for determining a maximum number of divisible stages of large model training according to the large model category, the large model layer information, and the mini-batch size, and based on the distributed large model training candidate terminal subset sequence;

[0089] A generation module is connected to the second determination module and is used to generate the number of pipeline parallel computing stages, pipeline segmentation decisions, computing power allocation scheduling decisions and batch segmentation numbers of the large model training task based on the goal of minimizing training time delay and the maximum number of divisible stages of the large model training, that is, to form a pipeline parallel training scheduling strategy for the distributed large model.

[0090] The present invention optimizes resource scheduling and load balancing strategies, reasonably allocates computing power and storage space, and effectively improves the performance of edge nodes, so as to better process complex deep learning models and massive data, thereby improving the efficiency of model training. The specific beneficial effects are as follows:

[0091] 1. Reduce computing and communication requirements:

[0092] The present invention significantly reduces the demand for computing power and network communication capabilities of a single node by optimizing the segmentation of large model pipelines and the coordinated allocation and scheduling of distributed computing power, thereby effectively improving the ability of the edge computing system to perform large model training at the edge of the network.

[0093] 2. Reduce network transmission delay:

[0094] The present invention avoids the frequent transmission of data and models in the network by directly performing large model training at the edge node, reducing the network latency of computationally intensive IoT applications. In addition, since data and large models do not need to leave the local network, the privacy and security of data and models are greatly improved.

[0095] 3. Improve model training efficiency:

[0096] The present invention effectively shortens the training time by preferentially allocating the segmented large model to terminal nodes with strong computing power and high communication capabilities for training. At the same time, by further dividing the mini-batch into multiple micro-batches for parallel scheduling, the overall training time is further reduced.

[0097] 4. Ensure the continuity of distributed training:

[0098] The present invention relies on the dual communication guarantees of cellular network communication of terminal nodes and device-to-device communication to ensure that during the distributed training process, an efficient parameter transfer rate is always maintained between nodes, thereby stably supporting the continuity of distributed large model training.

[0099] 5. Reduce algorithm complexity:

[0100] The present invention classifies edge computing nodes according to computing power and data transmission rate, and reasonably matches them to different training stages of the large model. At the same time, by adopting a phased iterative joint optimization strategy, the algorithm complexity is reduced, so that it can be quickly executed and optimized in an edge computing environment with limited computing power.

[0101] 6. Improve system flexibility and scalability:

[0102] The present invention is based on a modular pipeline design and distributed architecture, which can be flexibly adjusted according to actual needs, supports more edge nodes to participate in training tasks, and significantly improves the flexibility and scalability of the system, thereby better meeting the complex needs of large-scale deep learning model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 A schematic diagram of an edge computing method for distributed large model pipeline parallel training in an embodiment of the present invention;

[0104] Figure 2 This is a general flow chart of an edge computing method for distributed large model pipeline parallel training in an embodiment of the present invention;

[0105] Figure 3It is a schematic diagram of the structure of the edge computing system in an embodiment of the present invention;

[0106] Figure 4 A schematic diagram of the hierarchical structure of a large model in a large model training task in an embodiment of the present invention;

[0107] Figure 5 This is a flow chart of IoT terminal set diversity and subset sorting in an embodiment of the present invention;

[0108] Figure 6 A schematic diagram of the network topology of the terminals in the subset in the edge computing system in an embodiment of the present invention;

[0109] Figure 7 This is a flow chart of a distributed large model edge computing strategy in an embodiment of the present invention;

[0110] Figure 8 is a flow chart of a batch segmentation strategy in an embodiment of the present invention;

[0111] Fig. 9 It is a schematic diagram of large model pipeline parallel edge computing in an embodiment of the present invention;

[0112] Fig.10 Schematic diagram of an edge computing device for distributed large model pipeline parallel training in an embodiment of the present invention.

[0113] Among them, the figure numbers are: 10, receiving unit, 20, generating unit, 30, sending unit. DETAILED DESCRIPTION

[0114] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0115] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.

[0116] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.

[0117] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.

[0118] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0119] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.

[0120] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.

[0121] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.

[0122] Embodiment 1:

[0123] like Figure 1 As shown, this embodiment provides an edge computing method for parallel training of distributed large model pipelines. The edge computing method for parallel training of distributed large model pipelines is widely used in multiple scenarios, including real-time data processing and optimization in intelligent manufacturing, sensor data analysis of self-driving cars, traffic management and environmental monitoring of smart cities, intelligent improvement of Internet of Things devices, real-time monitoring and personalized treatment in the field of medical health, face recognition and behavior analysis of video surveillance and security systems, data processing in augmented reality and virtual reality applications, and real-time transaction monitoring and risk assessment in financial services. These application scenarios give full play to the advantages of edge computing, improve the intelligence level and data processing capabilities of various industries, and meet the growing demand for real-time and security. The method comprises the following steps:

[0124] Step K1: Receive a large model training task; the large model training task carries task information, which includes the large model category, large model layer information, and mini-batch size. The large model training task contains key information, including the category of the large model, the detailed structure of each layer, and the size of the mini-batch. These task information provide the necessary context for the training process, enabling the system to effectively schedule and allocate resources according to different types of large models and their hierarchical structures, while ensuring that the size of the mini-batch can be optimized when processing data to improve training efficiency and performance. Through the integration of this information, the system can better understand the task requirements and thus achieve efficient model training.

[0125] Step K2: Generate a pipeline parallel training scheduling strategy for distributed large models based on the large model category, the large model layer information, and the mini-batch size. The pipeline parallel training scheduling strategy for distributed large models is based on the candidate terminal subset sequence and the maximum number of divisible stages, and is formulated in combination with the goal of minimizing training delay. The candidate terminal subset sequence is a sequence formed by screening and sorting terminal devices, aiming to optimize network scheduling and resource allocation to improve user experience and system performance; the maximum number of divisible stages refers to the ability to split the overall task into the maximum number of independent processing stages in the algorithm or process to achieve parallel computing and improve resource utilization and efficiency; minimizing training delay involves reducing the time required for machine learning model training by adopting efficient algorithms, optimizing model design, and reasonably allocating computing resources, thereby speeding up the model training process and the deployment speed of practical applications. Through the combined effect of the above three, we are committed to improving system performance and response speed.

[0126] The step K2 specifically includes the following steps:

[0127] Step K21: Determine the distributed large model training candidate terminal subset sequence.

[0128] Step K21 specifically includes the following steps:

[0129] Divide the IoT terminal set into a finite number of subsets, and arrange the finite number of subsets in descending order according to computing power to obtain a subset vector;

[0130] Based on the network topology of the edge computing system, find the target transmission path corresponding to each subset vector;

[0131] The data transmission rate of any link in the target transmission path is the maximum data transmission rate from the source terminal to the target terminal of the link;

[0132] The subset vector corresponding to the target transmission path is used as a candidate terminal subset sequence for distributed large model training.

[0133] Step K22: Determine the maximum number of divisible stages of large model training based on the large model category, the large model layer information, the mini-batch size, and the distributed large model training candidate terminal subset sequence, that is, form a pipeline parallel training scheduling strategy for the distributed large model.

[0134] Step K22 specifically includes the following steps:

[0135] Step K221: Determine the number of layers of the large model based on the large model category and the large model layer information;

[0136] Step K222: Calculate the total computational effort C of the large model based on the number of layers of the large model and the size of the mini-batch. total ; and, training a candidate terminal subset sequence according to the distributed large model to calculate the total computing power;

[0137] The total computational effort of the large model is C total The calculation formula is as follows:

[0138]

[0139] Where B is the size of mini-batch;

[0140] L is the number of layers of the large model;

[0141] is the computational amount of a single training task at the i-th layer;

[0142] The total computing power P total The calculation formula is as follows:

[0143]

[0144] In the formula,

[0145] N represents the number of computing nodes in the distributed large model training candidate terminal subset sequence;

[0146] P n Indicates the computing power of the nth computing node;

[0147] Step K223: Set the initial stage number Stage1 and calculate the average calculation amount C stage ;

[0148] Order average calculation amount C stage The calculation formula is as follows:

[0149] C stage =C total / Stage1;

[0150] Step K224: Calculate the amount C based on the order average stage and total computing power P total , calculate the computation time T of each large model training stage stage ;

[0151] The computation time T of the large model training phase stage The calculation formula is as follows:

[0152]

[0153] In the formula,

[0154] Tcomm Indicates the communication delay between nodes; max() indicates the maximum value function;

[0155] Step K225: Based on the total calculation amount C of the large model total 、Total computing power P total , and the computation time T of the large model training phase stage , calculate the number of divisible stages S of the target targe ;

[0156] The calculation formula for the number of divisible stages of the target is as follows:

[0157]

[0158] In the formula, int() represents the rounding function;

[0159] Step K226: Compare the initial number of stages S1 with the target number of divisible stages S targe :

[0160] If the initial stage number Stage1 is greater than the target divisible stage number S targe , then reduce the initial stage number, repeat steps S223 to S226 until the initial stage number Stage1 and the target divisible stage number S targe equal;

[0161] If the initial stage number Stage1 is less than the target divisible stage number S targe , then increase the initial stage number, and repeat steps S223 to S226 until the initial stage number Stage1 and the target divisible stage number S targe equal;

[0162] If the initial stage number Stage1 and the target divisible stage number S targe If they are equal, then the target divisible stage number S is set targe is the maximum number of divisible stages, which determines the maximum number of divisible stages for large model training.

[0163] Step K23: Based on the goal of minimizing training time delay and the maximum number of divisible stages of the large model training, generate the number of pipeline parallel computing stages, pipeline segmentation decisions, computing power allocation scheduling decisions and batch segmentation numbers of the large model training task.

[0164] The step K23 specifically includes the following steps:

[0165] Step K231: Set the current stage number of the large model pipeline parallel computing to the initialization value 1, set the current terminal subset sequence number to the initialization value 0, set the current distributed computing terminal set to the initialization value empty set, initialize the current batch split number to the initialization value 1, set the current mini-batch training delay to the initialization value infinity, and set the current pipeline split decision and the current computing power allocation scheduling decision to empty.

[0166] Step K232: Determine the candidate stage number and candidate calculation terminal vector;

[0167] Among them, the candidate stage number is the number of terminals between the first subset in the distributed large model training candidate subset sequence and the subset with the current terminal subset number plus 1; the candidate calculation terminal vector includes all terminals from the first subset to the subset with the current terminal subset number plus 1.

[0168] Step K233: Determine whether the number of candidate stages is less than or equal to the maximum number of divisible stages:

[0169] If the number of candidate stages is greater than the maximum number of divisible stages, the process proceeds to step S238 ; if the number of candidate stages is less than or equal to the maximum number of divisible stages, the process proceeds to step S234 .

[0170] Step K234: perform pipeline segmentation on the large model with the number of candidate stages as the segmentation number, and match the sub-models of each stage obtained by the segmentation to the terminals in the candidate computing terminal vector in pipeline order, so as to obtain a joint decision on pipeline segmentation and computing power allocation and scheduling.

[0171] In step K234, the joint decision of pipeline segmentation and computing power allocation scheduling is obtained by matching the micro-batch in the mini-batch and the terminal in the candidate computing terminal vector, and the obtaining process specifically includes the following steps:

[0172] Step B1: Split the mini-batch into m micro-batches;

[0173] The size of each micro-batch is equal, and the sum of the data size of m micro-batches is equal to the data size of the mini-batch;

[0174] Step B2: According to the scheduling rules, match the micro-batch in the mini-batch and the terminal in the candidate computing terminal vector;

[0175] The dispatching rules include at least one of the following aspects:

[0176] 1) The micro-batch sequence is scheduled in order;

[0177] 2) The training of each micro-batch is distributed to the IoT terminals corresponding to each stage for training in sequence according to the large model pipeline segmentation method;

[0178] 3) Backward propagation starts only after the forward pass of all micro-batch training in all micro-batch sequences is completed;

[0179] 4) After all micro-batches in the micro-batch sequence complete backpropagation, pipeline refresh is performed to synchronize the gradients of the large model;

[0180] 5) When a terminal in the distributed computer terminal set is idle, once it receives a micro-batch stage training task from an upstream terminal or there is a waiting micro-batch stage training task in the queue, it executes the stage training subtask in the order of the micro-batch in the mini-batch.

[0181] Step K235: Determine the local optimal batch split number and the local optimal mini-batch training delay.

[0182] As a specific implementation, in step K235, determining the local optimal batch split number and the local optimal mini-batch training delay specifically includes the following steps:

[0183] Step A1: Determine the splittable set of mini-batch training set based on the mini-batch size.

[0184] The step A1 specifically includes the following steps:

[0185] Step A11: Calculate the power M with base 2 according to the mini-batch size N;

[0186] M = ceiling(log2(N), 1);

[0187] ceiling() represents the rounding function, and its rounding base is 1;

[0188] log2(N) represents the logarithm of N with base 2;

[0189] Step A12: According to the power M, determine the divisible set of the mini-batch training set as {2 0 , 2 1 , ..., 2 M}.

[0190] According to the mini-batch size, there are many ways to determine the splittable set of the mini-batch training set. In addition to using the logarithm of the number of samples N with base 2 for splitting, you can also use the uniform splitting method to evenly divide the training set into multiple mini-batches of equal size, or split according to sample characteristics to ensure that each mini-batch contains samples of different categories. In addition, methods such as dynamically adjusting the mini-batch size, randomly selecting samples to form mini-batches, adaptive splitting, and splitting based on memory constraints can also effectively determine the splittable set, thereby improving training efficiency and model performance.

[0191] Step A2: taking the element size in the divisible set as the batch split number, splitting the mini-batch evenly to obtain different batch split numbers; and evaluating the mini-batch training delay corresponding to the batch split number.

[0192] Step A3: Set the local optimal batch split number to the batch split number corresponding to the minimum mini-batch training delay, and set the local optimal mini-batch training delay to the minimum mini-batch training delay.

[0193] Step K236: Determine whether the local optimal mini-batch training delay is less than the current mini-batch training delay:

[0194] If the local optimal mini-batch training delay is less than the current mini-batch training delay, proceed to step K237; if the local optimal mini-batch training delay is not less than the current mini-batch training delay, proceed to step K238.

[0195] Step K237: Update the current number of stages of parallel computing of the large model pipeline to the candidate number of stages, update the current pipeline segmentation decision to the large model pipeline segmentation decision, increase the current terminal subset sequence number by 1, update the current distributed computing terminal set to the candidate computing terminal set, update the current computing power allocation scheduling decision to the computing power allocation scheduling decision, update the current batch split number to the local optimal batch split number, update the current mini-batch training delay to the local optimal mini-batch training delay, and then jump back to step K232 until the local optimal mini-batch training delay is not less than the optimal mini-batch training delay.

[0196] Step K238: Set the current number of stages of large model pipeline parallel computing as the optimal number of stages of large model pipeline parallel computing; and, set the current pipeline segmentation decision as the optimal pipeline segmentation decision; and, set the current distributed computing terminal set as the optimal distributed computing terminal set; and, set the current batch segmentation number as the optimal batch segmentation number.

[0197] Step K3: Send the pipeline parallel training scheduling strategy to the edge computing node so that the edge computing node executes the large model training task, thereby realizing edge computing for distributed large model pipeline parallel training.

[0198] Figure 2 This is a schematic diagram of the overall process of the edge computing method for distributed large model pipeline parallel training provided in this embodiment, combined with Figure 3 , including an edge computing system, the edge computing system includes an Internet of Things terminal set, a base station, and a computing decision controller, the Internet of Things terminal set includes one or more Internet of Things terminals with computing capabilities, the Internet of Things terminals can communicate with each other through a cellular network through the base station, or directly communicate in a device-to-device manner, when the computing decision controller receives a large model training task from an application, triggering an edge computing process for distributed large model pipeline parallel training, including the following steps:

[0199] S101: The computing decision controller receives a large model training task Task from the application, and obtains information about the task Task, including the large model category class of the task, and information about each layer of the large model L = {L1, L2, ..., L T}, mini-batch size W / S, where the variable class represents the type of the large model, for example, the visual large model, the language large model, the natural number T represents the number of layers of the large model, the natural number W represents the size of the mini-batch based on the computing power requirement, and the natural number S represents the size of the mini-batch in bits;

[0200] S102: Execute the distributed edge computing decision for the large model and determine the distributed large model training decision {K * ,M * ,τ *} and computing power allocation scheduling decision ε * , specifically including: determining the distributed large model training candidate terminal subset sequence u={u1,u2,……,u U}, determine the maximum number of divisible stages K of the large model, and further determine the optimal number of stages K of pipeline parallel computing of the large model training task Task based on the distributed large model edge computing strategy with minimum latency * , optimal pipeline segmentation decision τ* , optimal computing power allocation scheduling decision ε * , optimal batch split number M * , where the training decision includes the optimal number of pipeline parallel calculation stages K * , optimal batch split number M * , optimal pipeline segmentation decision τ * ;

[0201] S103: The calculation decision controller calculates the decision {K * ,M * ,τ * ,ε *} is sent to relevant nodes, and the relevant nodes execute the large model training task in the edge computing system according to the decision.

[0202] Figure 4 Schematic diagram of the hierarchical structure of a large model for a large model training task provided by an embodiment of the present invention. Figure 2 , Figure 3 and Figure 4 For example, a large artificial intelligence model can be divided into T layers according to its machine learning structure. The information L of the tth layer (1≤t≤T) T Contains the number and size of parameters of this layer; according to the inter-layer structure of the large model, the model can be divided into a maximum of K stages of pipeline parallel computing, that is, Figure 4 As shown in the figure, L1 and L2 must be executed on the same computing node, therefore, L1 and L2 are divided into the first stage and assigned to an independent computing node to perform parameter training of this stage; L3 and several adjacent consecutive layers must be processed on the same computing node, therefore, they are divided into the second stage and assigned to another independent computing node for execution, and so on. In the figure, stage K is the last stage of the division, indicating that the large model can be divided into K stages at most and assigned to K computing nodes for staged training; in the process of parallel training of the large model pipeline, for each round of training, the previous stage (for example, stage 1) must be trained first and the training results received before the next stage (for example, stage 2) can start the training of this stage.

[0203] A specific artificial intelligence model is used to illustrate how to perform hierarchical division and scheduling.

[0204] Assume that the structure of the large model is as follows:

[0205] Layer 1 (input layer): responsible for receiving image data, with fewer parameters and mainly processing input.

[0206] Layer 2 (Convolutional Layer 1): Contains 100 convolution kernels and 10,000 parameters, responsible for extracting basic features of the image.

[0207] Layer 3 (Activation Layer 1): uses ReLU activation function with very few parameters.

[0208] Layer 4 (Convolutional Layer 2): Contains 200 convolution kernels and 20,000 parameters, responsible for extracting more complex features.

[0209] Layer 5 (Pooling Layer 1): Performs pooling operations to reduce feature dimensions with very few parameters.

[0210] Layer 6 (fully connected layer): has 500 neurons and 100,000 parameters, and is responsible for classifying the extracted features.

[0211] According to the above structure, this embodiment performs the following stage division and scheduling:

[0212] (1) Maximum number of divisible stages K: Assume that through analysis, the large model can be divided into a maximum of three stages.

[0213] (2) Stage division:

[0214] Phase 1: Contains Layer 1 and Layer 2. These two layers can be executed in parallel on the same computing node because their computations are closely linked and data flows smoothly. Assigned to computing node A for training.

[0215] Phase 2: includes layers 3 and 4. The output of layer 3 requires the input of layer 4, so they are also assigned to the same computing node, here assigned to computing node B for training.

[0216] Phase 3: includes layers 5 and 6. These two layers also need to be processed on the same computing node and are assigned to computing node C.

[0217] (3) Training Scheduling:

[0218] During the training process, the training of stage 1 is first performed, and computing node A processes the image data through the first and second layers to obtain the output results.

[0219] Once Phase 1 is complete, compute node A passes the training results to compute node B, and Phase 2 can begin. Compute node B uses the output of Phase 1 to complete the training of layers 3 and 4.

[0220] After implementing stage 2, the results are sent to computing node C, and stage 3 begins training the 5th and 6th layers.

[0221] Finally, after completing all three stages of training, the entire large model completes a round of training. In this way, through layering and scheduling, the training process of the large model can be carried out more efficiently.

[0222] Figure 5is a flow chart of IoT terminal diversity and subset sorting provided by this embodiment, Figure 6 This is a network topology diagram of a subset of IoT terminals in the edge computing system. Figure 2 to Figure 6 , an example is given to illustrate the method of determining the sequence of candidate terminal subsets for distributed large model training:

[0223] S201: Divide the IoT terminal set into a finite number of subsets according to the computing capability level of the terminals in the IoT terminal set, and let u = {U1, U2, ..., U N} represents the IoT terminal set, which contains N terminals. represents the computing power level of the corresponding terminal. Assuming that the above terminals can be divided into U categories according to the computing power level, the IoT terminal set can be expressed as u = {u1, u2, ..., u U}, where u u ∈u represents the uth subset, and the terminals in this subset have the same computing capability level;

[0224] Further, u={u1,u2,……,u U The subsets in} are arranged in descending order of computing power, and the subset vector based on computing power is obtained as u = {u1, u2, ..., u U}, where the terminals in subset u (1≤u≤U) have the same computing power level, and the computing power of the terminals in subset u is greater than the computing power of the terminals in subset u+1. For example, assuming that the computing power of terminal 1 is 10 9 OPS (Operations per second), belongs to the u-th subset, the computing power of terminal 2 is 10 6 OPS, belongs to the u+1th subset.

[0225] S202: For each subset, find the best transmission path between the terminals in the subset based on the network topology of the edge computing system Among them, the natural number u represents the uth (1≤u≤U) subset in u, let the set u u ={U1,U2,……,U N} represents the terminal of the subset, the natural number N represents the number of terminals in the subset, combined with Figure 5 , let the variable Indicates the terminal i to terminal j in the subset (U i , U j ∈U u ) device-to-device (D2D) communication mode (that is, data is transmitted directly from U i Sent to U via D2D link j ) under the data transmission rate, let the variable Terminal i to terminal j (U i , U j ∈U u ) cellular network communication mode (that is, data is transmitted via U i —Base Station—U j The data transmission rate under the path transmission) is set to R i , j Indicates link The data transmission rate of the optimal transmission path P u Satisfy: 1) Link The data transmission rate is the maximum value of the data transmission rate under D2D and the data transmission rate under cellular communication mode, that is, 2) Transmission path P u The data transmission rates of the links in the path are arranged in descending order, that is, R 1,2 ≥R2,3≥……≥R i , j ≥R j , k ≥……; 3) The data transmission rates of the links in the path have the same data transmission rate level, for example, R i，j =80Mbps,R j，k =75Mbps,R k，N =100Kbps, then the link have the same data rate level, and the link With the transmission path P u Any link in the network is not at the same data transmission rate level;

[0226] S203: For each subset, update the terminals in the subset according to the optimal transmission path, that is, remove the destination terminals of the links in the transmission path whose data transmission rate level is lower than the data transmission rate level of other links in the path from the subset, for example, due to the link With the transmission path P u If any link in the network is not at the same data transmission rate level, the terminal U N From U u Remove from the list and get the updated subset vector u′={u1,u2,……,u i ,u j ,u k ,u N′};

[0227] S204: Use the updated subset vector as the candidate subset sequence for distributed large model training, that is, the candidate subset sequence is u′={u′1,u′2,……,u′ u ,……,u U′}.

[0228] Figure 7 It is a distributed large model edge computing decision flow chart. Figure 8 is the batch segmentation flowchart, Fig. 9 This is a schematic diagram of large model pipeline parallel edge computing. Figure 2 to Figure 9 , an example is given to illustrate the edge computing decision-making process for distributed large model pipeline parallel training:

[0229] S301: Variable initialization, specifically including: initializing the optimal number of stages of parallel computing of the large model pipeline to 1, that is, K * =1, initialize the best terminal subset number to 0, that is, u * = 0, initialize the distributed computing terminal vector to empty, that is, Initialize the optimal batch split number to 1, that is, M * = 1, initialize the optimal mini-batch training delay to infinity, that is, D batch =∞, set the optimal pipeline segmentation decision and the optimal computing power allocation scheduling decision to null, that is,

[0230] S302: Determine the candidate stage number K′ and the candidate computing terminal vector u″, specifically including: calculating the number of terminals between the first subset and the subset with the best terminal subset number plus 1 in the candidate subset sequence of the distributed large model training, marking it as the candidate stage number, and putting all the terminals from the first subset to the subset with the best terminal subset number plus 1 into the candidate computing terminal vector, that is, calculating (u′1, u′2, ..., u′ u*+1 ), assuming that the number is N, then the number of candidate stages is set to K′=N, and (u′1, u′2, …, u′ u*+1 ) are put into the candidate calculation terminal vector u″, that is, u″=(U1,U2,……,U N );

[0231] S303: Determine whether the number of candidate stages is less than or equal to the maximum number of divisible stages, that is, determine whether K′≤K holds: if so, go to step S304; if not, jump to step S308;

[0232] S304: The large model is pipeline segmented with the candidate stage number K′ as the segmentation number, and the sub-models of each stage obtained by the segmentation are matched to the terminals in the candidate computing terminal vector u″ in pipeline order, forming a joint decision of pipeline segmentation and computing power allocation scheduling (τ * ,ε *), the decision satisfies: 1) any stage of the large model only matches one terminal in the candidate computing terminal vector u″, for example, stage 1 matches the first terminal in u″, and stage K′ matches the last terminal in u″; 2) pipeline segmentation decision τ * Describes the relationship between the various layers and stages of machine learning in a large model, for example, τ * =({L1,L2},{L3,L4,L5},……,{L T}) indicates that the L1 and L2 layers in the large model are in the first stage, the L3, L4, and L5 layers are in the second stage, and the last layer L T Split to the last stage K′; 3) The computing power allocation scheduling decision ε' describes which terminal is trained at each stage of the large model and the training order, for example, ε' = ({Stage1, U1}, {Stage2, U2}, ..., {Stage k′ , U k′}), indicating that the first stage is the calculation of the terminal U1 assigned to u″, the second stage is the calculation of the terminal U2 assigned to u″, and the last stage K' is the last terminal U in u″ k′ The calculation order is Stage1→Stage2→…→Stage k′ ; 4) Except for the first stage, the amount of large model parameter training in other stages is proportional to the computing power of the computing terminal, that is, W2 / f2~W3 / f3~…~W k′ / f k′ , where variable W i (2≤i≤k′) represents the amount of model parameter training in the i-th stage (measured in terms of computational requirements), and the variable f i Indicates the computing power of the terminal that performs the calculation at this stage;

[0233] S305: Determine the local optimal batch split number M′ and the local optimal mini-batch training delay D batch′ ;

[0234] S306: Determine whether the local optimal mini-batch training delay is less than the optimal mini-batch training delay, that is, determine D batch′ <D batch Is it true: If yes, go to step S307; if no, jump to step S308;

[0235] S307: Update the optimal number of stages for parallel computing of the large model pipeline to the candidate number of stages, that is, set K * = K′, update the optimal pipeline segmentation decision to the large model pipeline segmentation decision, that is, set τ * = K′, add 1 to the best terminal subset number, that is, set U * =U* +1, update the distributed computing terminal vector to the candidate computing terminal vector, that is, set U * =u″, update the optimal computing power allocation scheduling decision to the computing power allocation scheduling decision, that is, set ε * =ε′, update the optimal batch split number to the local optimal batch split number, that is, set M * =M′, update the optimal mini-batch training delay to the local optimal mini-batch training delay, that is, set D batch′ =D batch , return to step S302;

[0236] S308: Return the optimal number of stages K for parallel computing of the large model pipeline * , optimal pipeline segmentation decision τ * , optimal computing power allocation scheduling decision ε * , optimal batch split number M * .

[0237] Among them, step S305 determines the local optimal batch split number M′ and the local optimal mini-batch training delay D batch′ , combined with Figure 7 to Figure 9 , further comprising:

[0238] S401: Determine the divisible set of the small batch training set based on the mini-batch size, specifically: mark the size of a mini-batch of the large model training task as a power of 2 with a base of 2 M , where the variable M is the index, further, the integer sequence M 集合 = {1, 2, ..., M} is marked as a separable set of mini-batch training sets;

[0239] S402: Taking the element size in the divisible set of the mini-batch training set as the batch division number, the mini-batch is evenly divided, and the training delay of the mini-batch under the batch division number is evaluated to form a mini-batch training delay set corresponding to the divisible set of the mini-batch training set. The mini-batch segmentation generates a micro-batch sequence, and the sum of the data volume of the micro-batch sequence is equal to the data volume of the mini-batch. For example, Fig. 9 As shown, when the batch split number is m(m∈M 集合 ), a mini-batch is divided into m(m∈M 集合) micro-batches, each micro-batch is of equal size, and the sum of the data size of these m micro-batches is equal to the data size of the mini-batch, such as Figure 8 As shown in , if xy represents a micro-batch, where x represents the sequence number of the mini-batch in the mini-batch set in the entire large model training task, and y represents that the micro-batch is the yth micro-batch in the mini-batch, for example, micro-batch 1-m represents the mth micro-batch in the first mini-batch in the large model training task, the scheduling rules of the micro-batch sequence under the large model pipeline segmentation and computing power allocation scheduling decision (τ', ε') should satisfy: 1) The micro-batch sequence is scheduled in sequence, for example, Fig. 9 As shown, terminal U1 (stage 1) / terminal U2 (stage 2) / … / terminal U k′ (Stage k′) The order of scheduling the first mini-batch is: micro-batch 1-1, micro-batch 1-2, ..., micro-batch 1-m; 2) The training of each micro-batch is allocated to the IoT terminals corresponding to each stage according to the large model pipeline segmentation method and trained in sequence, for example, Figure 8 As shown in Figure 2, for the 1-mth micro-batch, the training order is: terminal U1 (stage 1) → terminal U2 (stage 2) → … → terminal U k′ (stage k′); 3) Backward propagation starts only after the forward pass of all micro-batch training in the micro-batch sequence is completed, for example, Figure 8 As shown, micro-batch 1-1 to 1-m starts back propagation after the forward transfer from stage 1 to stage k′ is completed; 4) After all micro-batches in the micro-batch sequence complete the back propagation, the pipeline is refreshed to synchronize the gradient of the large model, as shown in Figure 8 As shown in Figure 2, only when terminal U1 completes the back propagation of the last micro-batch 1-m can the pipeline refresh be initiated to synchronize terminals U1 to U k′5) When a terminal in the distributed computing terminal set is idle, once a micro-batch stage training task is received from an upstream terminal or there is a waiting micro-batch stage training task in the queue, the stage training subtask is executed in the order of micro-batches in the mini-batch, for example, Figure 8 As shown, when terminal U k′ After executing the forward transfer of micro-batch1-(m-1), the terminal is idle. When receiving the k′ -1, or, when it is found that the training task of stage k′ of the 1-mth micro-batch is already in the waiting queue, the training of stage k′ of the micro-batch is performed; 6) The terminals in the distributed computing terminal vector u″ perform model training according to the computing capacity of the terminals and transmit the training results to the terminals corresponding to the next stage according to the data transmission rate of the terminals, that is, let f i Indicates terminal U i ∈u″’s computing power, R i,j Indicates link The data transmission rate of terminal U i To calculate the rate f i Process each micro-batch processed at this terminal at a rate R i,j Transmit the calculation results to terminal U j ;

[0240] S403: setting the local optimal batch split number to the batch split number corresponding to the minimum delay in the mini-batch training delay set, and setting the local optimal mini-batch training delay to the minimum delay, that is, if the mini-batch training delay set The number of batch splits corresponding to the minimum delay in is m', that is, m′∈M 集合 , where min() represents the minimum function, then, set M′=m′,

[0241] This embodiment optimizes resource scheduling and load balancing strategies and reasonably allocates computing power and storage space, thereby effectively improving the performance of edge nodes, so as to better process complex deep learning models and massive data, thereby improving the efficiency of model training.

[0242] Embodiment 2:

[0243] like Fig.10As shown, the present invention provides an edge computing device for distributed large model pipeline parallel training, the device comprising:

[0244] A receiving unit 10 is used to receive a large model training task; the large model training task carries task information, and the task information includes a large model category, large model layer information, and a mini-batch size;

[0245] A generating unit 20, connected to the receiving unit 10, for generating a pipeline parallel training scheduling strategy for a distributed large model according to the large model category, the large model layer information, and the mini-batch size;

[0246] The sending unit 30 is connected to the generating unit 20, and is used to send the pipeline parallel training scheduling strategy to the edge computing node, so that the edge computing node executes the large model training task, thereby realizing edge computing for distributed large model pipeline parallel training.

[0247] As a specific implementation, the generating unit 20 includes:

[0248] A first determination module is used to determine a distributed large model training candidate terminal subset sequence;

[0249] a second determination module, connected to the first determination module, for determining a maximum number of divisible stages of large model training according to the large model category, the large model layer information, and the mini-batch size, and based on the distributed large model training candidate terminal subset sequence;

[0250] A generation module is connected to the second determination module and is used to generate the number of pipeline parallel computing stages, pipeline segmentation decisions, computing power allocation scheduling decisions and batch segmentation numbers of the large model training task based on the goal of minimizing training time delay and the maximum number of divisible stages of the large model training, that is, to form a pipeline parallel training scheduling strategy for the distributed large model.

[0251] The device in this embodiment can execute the method in embodiment 1.

[0252] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. An edge computing method for distributed large model pipeline parallel training, characterized in that: The method comprises the following steps: Step K1: receiving a large model training task; the large model training task carries task information, and the task information includes large model category, large model layer information, and mini-batch size; Step K2: generating a pipeline parallel training scheduling strategy for a distributed large model according to the large model category, the large model layer information, and the mini-batch size; The pipeline parallel training scheduling strategy of the distributed large model is formulated based on the candidate terminal subset sequence and the maximum number of divisible stages, combined with the goal of minimizing training delay; Step K3: Send the pipeline parallel training scheduling strategy to the edge computing node so that the edge computing node executes the large model training task, thereby realizing edge computing for distributed large model pipeline parallel training.

2. The edge computing method for distributed large model pipeline parallel training according to claim 1 is characterized in that: The step K2 specifically includes the following steps: Step K21: Determine a distributed large model training candidate terminal subset sequence; Step K22: determining the maximum number of divisible stages of large model training according to the large model category, the large model layer information, the mini-batch size, and the distributed large model training candidate terminal subset sequence; Step K23: Based on the goal of minimizing the training time delay and the maximum number of divisible stages of the large model training, generate the number of pipeline parallel computing stages, pipeline segmentation decisions, computing power allocation scheduling decisions and batch split number of the large model training task, that is, form a pipeline parallel training scheduling strategy for the distributed large model.

3. The edge computing method for distributed large model pipeline parallel training according to claim 2 is characterized in that: The step K21 specifically includes the following steps: Divide the IoT terminal set into a finite number of subsets, and arrange the finite number of subsets in descending order according to computing power to obtain a subset vector; Based on the network topology of the edge computing system, find the target transmission path corresponding to each subset vector; The data transmission rate of any link in the target transmission path is the maximum data transmission rate from the source terminal to the target terminal of the link; The subset vector corresponding to the target transmission path is used as a candidate terminal subset sequence for distributed large model training.

4. The edge computing method for distributed large model pipeline parallel training according to claim 2 is characterized in that: The step K22 specifically includes the following steps: Step K221: Determine the number of layers of the large model based on the large model category and the large model layer information; Step K222: Calculate the total computational effort C of the large model based on the number of layers of the large model and the size of the mini-batch. total ; as well as , training a candidate terminal subset sequence according to the distributed large model, and calculating the total computing power; The total computational effort of the large model is C total The calculation formula is as follows (1): In formula (1), B is the size of mini-batch; L is the number of layers of the large model; is the computational amount of a single training task at the i-th layer; The total computing power P total The calculation formula is as follows (2): In formula (2), N represents the number of computing nodes in the distributed large model training candidate terminal subset sequence; P n Indicates the computing power of the nth computing node; Step K223: Set the initial stage number Stage1 and calculate the average calculation amount C stage ; Order average calculation amount C stage The calculation formula is as follows (3): C stage =C total / Stage1; Step K224: Calculate the amount C based on the order average stage and total computing power P total , calculate the computation time T of each large model training stage stage ; The computation time T of the large model training phase stage The calculation formula is as follows (4): In formula (4), T comm Indicates the communication delay between nodes; max() indicates the maximum value function; Step K225: Based on the total calculation amount C of the large model total 、Total computing power P total , and the computation time T of the large model training phase stage , calculate the number of divisible stages S of the target targe ; The calculation formula for the number of divisible stages of the target is as follows (5): In formula (5), int() represents the rounding function; Step K226: Compare the initial number of stages S1 with the target number of divisible stages S targe : If the initial stage number Stage1 is greater than the target divisible stage number S targe , then reduce the initial stage number, repeat steps S223 to S226 until the initial stage number Stage1 and the target divisible stage number S targe equal; If the initial stage number Stage1 is less than the target divisible stage number S targe , then increase the initial stage number, and repeat steps S223 to S226 until the initial stage number Stage1 and the target divisible stage number S targe equal; If the initial stage number Stage1 and the target divisible stage number S targe If they are equal, then the target divisible stage number S is set targe is the maximum number of divisible stages, which determines the maximum number of divisible stages for large model training.

5. The edge computing method for distributed large model pipeline parallel training according to any one of claims 2 to 4, characterized in that: The step K23 specifically includes the following steps: Step K231: Set the current stage number of the large model pipeline parallel computing to the initialization value 1, set the current terminal subset sequence number to the initialization value 0, set the current distributed computing terminal set to the initialization value empty set, initialize the current batch split number to the initialization value 1, set the current mini-batch training delay to the initialization value infinity, and set the current pipeline split decision and the current computing power allocation scheduling decision to empty; Step K232: Determine the candidate stage number and candidate calculation terminal vector; The candidate stage number is the number of terminals between the first subset and the subset with the current terminal subset number plus 1 in the distributed large model training candidate subset sequence; the candidate calculation terminal vector includes all terminals from the first subset to the subset with the current terminal subset number plus 1; Step K233: Determine whether the number of candidate stages is less than or equal to the maximum number of divisible stages: If the number of candidate stages is greater than the maximum number of divisible stages, the process proceeds to step S238; if the number of candidate stages is less than or equal to the maximum number of divisible stages, the process proceeds to step S234; Step K234: Pipeline segmentation is performed on the large model with the number of candidate stages as the number of segmentation, and the sub-models of each stage obtained by segmentation are matched to the terminals in the candidate computing terminal vector in pipeline order, so as to obtain a joint decision of pipeline segmentation and computing power allocation and scheduling; Step K235: Determine the local optimal batch split number and the local optimal mini-batch training delay; Step K236: Determine whether the local optimal mini-batch training delay is less than the current mini-batch training delay: If the local optimal mini-batch training delay is less than the current mini-batch training delay, proceed to step K237; if the local optimal mini-batch training delay is not less than the current mini-batch training delay, proceed to step K238; Step K237: Update the current stage number of the large model pipeline parallel calculation to the candidate stage number, update the current pipeline segmentation decision to the large model pipeline segmentation decision, increase the current terminal subset sequence number by 1, update the current distributed computing terminal set to the candidate computing terminal set, update the current computing power allocation scheduling decision to the computing power allocation scheduling decision, update the current batch split number to the local optimal batch split number, update the current mini-batch training delay to the local optimal mini-batch training delay, and then jump to step K232 again until the local optimal mini-batch training delay is not less than the optimal mini-batch training delay; Step K238: Set the current number of stages of large model pipeline parallel computing as the optimal number of stages of large model pipeline parallel computing; and, set the current pipeline segmentation decision as the optimal pipeline segmentation decision; and, set the current distributed computing terminal set as the optimal distributed computing terminal set; and, set the current batch segmentation number as the optimal batch segmentation number.

6. The edge computing method for distributed large model pipeline parallel training according to claim 5 is characterized in that: In the step K235, determining the local optimal batch split number and the local optimal mini-batch training delay specifically includes the following steps: Step A1: Determine the splittable set of the mini-batch training set based on the mini-batch size; Step A2: taking the element size in the divisible set as the batch split number, evenly splitting the mini-batch to obtain different batch split numbers; and evaluating the mini-batch training delay corresponding to the batch split number; Step A3: Set the local optimal batch split number to the batch split number corresponding to the minimum mini-batch training delay, and set the local optimal mini-batch training delay to the minimum mini-batch training delay.

7. The edge computing method for distributed large model pipeline parallel training according to claim 6 is characterized in that: The step A1 specifically includes the following steps: Step A11: Calculate the power M with base 2 according to the mini-batch size N; M = ceiling(log2(N), 1); ceiling() represents the upward rounding function, and its rounding base is 1; log2(N) represents the logarithm of N with base 2; Step A12: According to the power M, determine the divisible set of the mini-batch training set as {2 0 , 2 1 , ..., 2 M }.

8. The edge computing method for distributed large model pipeline parallel training according to claim 5 is characterized in that: In step K234, the joint decision of pipeline segmentation and computing power allocation scheduling is obtained by matching the micro-batch in the mini-batch and the terminal in the candidate computing terminal vector, and the obtaining process specifically includes the following steps: Step B1: Split the mini-batch into m micro-batches; The size of each micro-batch is equal, and the sum of the data size of m micro-batches is equal to the data size of the mini-batch; Step B2: According to the scheduling rules, match the micro-batch in the mini-batch and the terminal in the candidate computing terminal vector; The scheduling rules include: Micro-batches are scheduled sequentially; or, The training of each micro-batch is distributed to the IoT terminals corresponding to each stage according to the large model pipeline segmentation method and trained in sequence; or, Backpropagation starts only after the forward pass of all micro-batches in all micro-batch sequences is completed; or, After all micro-batches in the micro-batch sequence have completed backpropagation, pipeline refresh is performed to synchronize the gradients of the large model; or, When a terminal in the distributed computer terminal set is idle, once it receives a micro-batch stage training task from an upstream terminal or there is a waiting micro-batch stage training task in the queue, it executes the stage training subtask in the order of the micro-batch in the mini-batch.

9. An edge computing device for distributed large model pipeline parallel training, characterized in that: include: A receiving unit, used to receive large model training tasks; The large model training task carries task information, and the task information includes large model category, large model layer information, and mini-batch size; A generating unit, connected to the receiving unit, for generating a pipeline parallel training scheduling strategy for a distributed large model according to the large model category, the large model layer information, and the mini-batch size; The pipeline parallel training scheduling strategy of the distributed large model is formulated based on the candidate terminal subset sequence and the maximum number of divisible stages, combined with the goal of minimizing training delay; A sending unit is connected to the generating unit and is used to send the pipeline parallel training scheduling strategy to the edge computing node so that the edge computing node executes the large model training task, thereby realizing edge computing for distributed large model pipeline parallel training.

10. The edge computing device for distributed large model pipeline parallel training according to claim 9, characterized in that: The generating unit comprises: A first determination module is used to determine a distributed large model training candidate terminal subset sequence; a second determination module, connected to the first determination module, for determining a maximum number of divisible stages of large model training according to the large model category, the large model layer information, and the mini-batch size, and based on the distributed large model training candidate terminal subset sequence; A generation module is connected to the second determination module and is used to generate the number of pipeline parallel computing stages, pipeline segmentation decisions, computing power allocation scheduling decisions and batch segmentation numbers of the large model training task based on the goal of minimizing training time delay and the maximum number of divisible stages of the large model training, that is, to form a pipeline parallel training scheduling strategy for the distributed large model.

Images