A Distributed Transformer Large Model Training Method for Mobile Devices

Abstract

This invention discloses a distributed Transformer large-scale model training method for mobile devices. During the distributed training of the Transformer large-scale model, for each mobile device participating in the training, the computing resources of each heterogeneous processor are collected. Based on this, different numbers of self-attention heads in the Transformer are allocated to the heterogeneous processors for parallel computation, thereby accelerating the computation of the self-attention mechanism in the Transformer large-scale model on mobile devices. The fault-tolerant recovery process pre-handles predictable faults caused by the dynamic nature of mobile devices during distributed training, enabling distributed training to complete fault-tolerant recovery without interruption due to faults. This invention fully utilizes the dynamic nature of mobile devices and the computing resources of their multiple processors to achieve efficient and robust distributed training of the Transformer model on mobile devices.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology on mobile devices, and specifically to a distributed Transformer large model training method for mobile devices. Background Technology

[0002] Edge computing offers low latency, high security, and high customization for deep learning applications. With the widespread adoption of large Transformer-based models, users increasingly demand domain-specific knowledge from neural network models, making customization a critical need. Therefore, training neural networks on mobile devices is essential for enabling customized deep learning applications. Federated learning is a typical distributed training architecture for mobile devices, where each mobile device independently trains a complete neural network model, with edge servers aggregating and distributing the neural network weights. However, with the release of large Transformer-based models, the parameter scale of neural network models has reached hundreds of billions. Due to memory limitations, a single device can no longer train neural network models with tens of billions of parameters, thus restricting the federated learning architecture. To address the memory constraints of mobile devices, a feasible solution is to divide the neural network model into several sub-models and deploy them across multiple mobile devices for distributed collaborative training.

[0003] Because distributed training on mobile devices uses wireless networks for communication, network disconnections and device crashes may occur during training. Considering the dynamic nature of mobile devices, situations such as device battery depletion or premature device termination may also occur, all of which can interrupt distributed training on mobile devices. When these situations occur, fault-tolerant recovery strategies can be used to resume training after an interruption. However, the faults caused by the dynamic nature of mobile devices are predictable, and current fault-tolerant recovery strategies cannot leverage this dynamic nature to reduce the time overhead of fault-tolerant recovery.

[0004] Furthermore, although the self-attention mechanism in large Transformer models is capable of parallel computation, current distributed training methods cannot fully utilize the computing resources of mobile devices to accelerate its computation. Unlike GPUs on servers, which have superior parallel computing capabilities compared to CPUs, GPUs on mobile devices have similar or even weaker computing capabilities than CPUs. This allows mobile devices to utilize multi-processor parallel computing to accelerate the computation of the self-attention mechanism. However, on the one hand, current neural network computing frameworks supporting mobile devices can only perform computations on one processor at a time, meaning that neural network computations on mobile devices are often completed on the CPU, resulting in underutilization of mobile device computing resources. On the other hand, it is necessary to distribute the attention heads in the self-attention mechanism across different processors based on their heterogeneous computing power to minimize the parallel computation time of Transformer.

[0005] Therefore, how to fully utilize the computing resources and dynamism of mobile devices to improve the efficiency and robustness of distributed training of large Transformer models on mobile devices is an urgent research topic. Summary of the Invention

[0006] The present invention mainly addresses the aforementioned technical problems existing in the prior art, and provides a distributed Transformer large model training method for mobile devices. This method achieves efficient and robust distributed training of Transformer large models on mobile devices through a multi-processor scheduling module and an active fault-tolerant recovery module within the device.

[0007] The present invention mainly solves the above technical problems through the following technical solutions: A distributed Transformer large model training method for mobile devices. There are N mobile devices, including 1 central mobile device and N - 1 collaborative mobile devices. All mobile devices are connected through a network. The training method is as follows: The Transformer large model is divided into N sub-models and deployed to N mobile devices respectively for distributed collaborative training; if the mobile device is a multi-processor mobile device, this mobile device distributes the attention heads to each heterogeneous processor for calculation; in the forward propagation process of collaborative training, when 1 ≤ i < N, the i-th mobile device transmits the intermediate output obtained by local forward propagation calculation to the (i + 1)-th mobile device. If i = N, the i-th mobile device calculates the loss and starts to perform backpropagation, sending the gradient to the (i - 1)-th mobile device; in the backpropagation process of collaborative training, when 1 < i ≤ N, the i-th mobile device transmits the gradient obtained by backpropagation calculation to the (i - 1)-th mobile device. If i = 1, the i-th mobile device proceeds to train the next data batch; when a certain collaborative mobile device needs to exit the distributed training, a suitable mobile device is selected according to the fault tolerance and recovery method to replace the mobile device to be exited and continue training.

[0008] The central mobile device has the original training data, is responsible for managing the entire distributed training process, and is also responsible for calculating the sub-model allocated to it locally, while the collaborative mobile devices are responsible for calculating the sub-model allocated to them locally.

[0009] Preferably, the specific operation of distributing the attention heads to each heterogeneous processor for calculation is as follows:

[0010] There are K attention heads in the current mobile device. Before the start of distributed training, search all processors available for neural network calculation on the current mobile device, with a total of M. Measure the time taken for each processor to execute k attention head calculations, denoted as T k_j , where 1 ≤ j ≤ M, 1 ≤ k ≤ K. If a certain heterogeneous processor supports multiple neural network calculation libraries, select the shortest calculation time among all calculation libraries as T k_j ; for example, if the GPU of a mobile phone supports using two calculation libraries, OpenCL or Vulcan, to calculate neural networks, then select the one with the relatively shorter calculation time between OpenCL and Vulcan here;

[0011] Initialize the lower bound l to 0 and the upper bound r to the time T required for all heterogeneous processors to calculate K attention heads K_jThe minimum value in the middle; in each iteration, calculate the intermediate value mid=(l+r) / 2, and then check if there is an allocation scheme. Under this allocation scheme, the total attention head calculation time is less than or equal to (mid+ε)×110%. Each processor executes the attention head calculation in parallel, so the total execution time is equal to the execution time of the slowest processor + the time for the processor to copy data to the CPU. Therefore, the time requirement is appropriately relaxed. ε is the calculation time deviation threshold, which is determined manually according to the model; define the allocation scheme S={(j,O j The specific checking method is as follows: (j=1,…,M;k=1,…K)

[0012] Initialize the current allocation scheme S'={}, and for the j-th processor, find |T k_j The minimum value of -mid| is denoted as O(n). j That is, to allocate k self-attention heads to the j-th processor, and to set (j, O) j Insert S'; if |T k_j If the minimum value of -mid| exceeds the set threshold ε, then set O. j =0 indicates that the processor is calculating too fast or too slow; if all O(0) values ​​are equal to 0, it means .... j If the sum of O's is greater than or equal to K, then the allocation scheme S' is feasible. Update the original allocation scheme, let S = S', and set the upper bound r to mid-σ; if all O's are greater than or equal to K, then the allocation scheme S' is feasible. j If the sum of σ and σ is less than K, the allocation scheme is not feasible. The lower bound is updated to mid + σ, and then an allocation scheme is searched again. σ is a relatively small quantity to avoid infinite loops, and is generally set to 0.1% of the upper bound l. When l > r, the iteration ends.

[0013] In parallel computing, the total computation time depends on the slowest processor. Therefore, in this scheme, it is necessary to ensure that the computation time of each processor is as close as possible to mid. Thus, too fast or too slow is not a reasonable allocation scheme.

[0014] Preferably, when the collaborative mobile device needs to exit distributed training due to a dynamic event, the mobile device d qThe central mobile device is notified α time in advance; after receiving the notification, the central mobile device searches for available mobile devices in the network for training via broadcast; if there are no available mobile devices in the network, the training process is restored using a conventional passive fault-tolerant recovery algorithm. Conventional passive fault-tolerant recovery algorithms include algorithms based on weight backup and model redistribution, for details see Li P, Koyuncu E, Seferoglu H. Respipe: Resilient model-distributeddnn training at edge networks[C] / / ICASSP 2021-2021 IEEEInternationalConference on Acoustics, Speech and Signal Processing (ICASSP).IEEE, 2021: 3660-3664., or ChenY, Yang Q, He S, et al. Ftpipehd: A fault-tolerant pipeline-parallel distributed training approach for heterogeneousedge devices[J]. IEEE Transactions on Mobile Computing, 2023, 23(4): 3200-3212., or Ye S, Zeng L, Chu X, et al. Asteroid: Resource-Efficient HybridPipeline Parallelism for Collaborative DNN Training on Heterogeneous EdgeDevices[C] / / Proceedings of the 30th Annual International Conference on MobileComputing and Networking. 2024: 312-326. If there are idle mobile devices, all idle mobile devices send their computing power representation vector and remaining battery percentage to the central mobile device. The computing power representation vector of the u-th idle mobile device is h. u The remaining battery percentage of the device is b. u h u The computation time of the Transformer module is used to characterize the computing power of a mobile device, and h is defined. u ={t u,1 ,t u,2 ,…,tu,L}, t u,n This represents the time required to compute an n-layer Transformer; b u It is a decimal number between 0 and 1, with a larger value indicating more remaining battery power on the mobile device; based on h u and b u The central equipment uses the Device Compatibility (DC) criterion to evaluate the compatibility of idle mobile devices, defined as follows:

[0015] ,

[0016] DC u For the adaptability of the u-th idle device, p represents the percentage of training remaining processes, equal to [B r +(TT cur [)*B]*B / T, where T represents the total number of training rounds, B represents the total number of data batches, and T cur Indicates the current training round, B r This represents the remaining batches of data in the current training round. η is a small constant greater than 0 to prevent the denominator from being 0. The normalized computing power of the device is equal to (H u -H min ) / (H max -H min ), here H u By h u H is obtained by summing all the elements in H. max and H min These are all H u The maximum and minimum values ​​in; The normalized electrical quantity of the device is equal to (b u -b min ) / (b max -b min ), b max It is all b u The maximum value in, b min It is all b u The minimum value in the range; based on the device compatibility criteria, the central mobile device selects the most suitable (DC) from the local area network. u Maximum value) Mobile device d s Replace the mobile device to exit. q The above process is performed synchronously with the collaborative training and will not interrupt the training; subsequently, the training is briefly interrupted, d q To d sThe weights of the Transformer sub-model are passed; after the weight transmission is complete, the central mobile device broadcasts a device change message to all participating devices, and finally, distributed training resumes normally. The device adaptability criterion considers both the computing resources and remaining battery power of the mobile device when selecting it to ensure training stability. At the beginning of training, the value of p is relatively large, and the device adaptability criterion focuses more on the remaining battery power of the mobile device; as training progresses, the value of p gradually decreases, and the device adaptability criterion pays more attention to the computing power of the mobile device.

[0017] Preferably, the α time for notifying the central mobile device in advance is greater than the time required for the central device to select the most suitable replacement device from the local area network devices.

[0018] Preferably, the dynamic events of the mobile device include running out of power and actively leaving the local area network.

[0019] Preferably, the mobile device is a smart terminal with computing capabilities, including mobile phones, watches, microcontrollers, cameras, laptops, and desktop computers.

[0020] Preferably, the processor of the mobile device is a chip with computing capabilities, including CPU, GPU, and NPU. Homogeneous processors are a special case of heterogeneous processors, and our solution is also applicable to homogeneous processors, so the allocation result may be an average allocation.

[0021] The substantial effects of this invention are: (1) The multiprocessor scheduling method can allocate the attention head in the large language model based on Transformer to multiple processors for parallel computation according to the computing power of each heterogeneous processor of the edge device, thereby accelerating the computation of the large language model on the edge device; (2) The active fault tolerance recovery method enables collaborative training to cope with training interruptions caused by dynamic events of mobile devices in advance, reducing the time overhead caused by fault tolerance recovery and improving the robustness of the collaborative training method. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a distributed Transformer large model training method for mobile devices according to the present invention;

[0023] Figure 2 , Figure 3 and Figure 4 This is a schematic diagram of the computation of the multiprocessor scheduling module within the device in a distributed Transformer large model training method for mobile devices according to the present invention.

[0024] Figure 5This is a schematic diagram of the active fault-tolerant recovery module in a distributed Transformer large model training method for mobile devices according to the present invention. Detailed Implementation

[0025] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0026] Example: A distributed Transformer large model training method for mobile devices, consisting of three mobile devices, such as... Figure 1 As shown, device 1 is the central mobile device, possessing the raw data to be trained; devices 2 and 3 are collaborating mobile devices. All three mobile devices are connected to the same router, identified by their IP addresses, and communicate via wireless network and HTTP requests. Each mobile device is equipped with an application implementing this invention, using MNN as the neural network computation framework. MNN is a computation framework that supports neural network training on mobile devices.

[0027] A central mobile device divides an eight-layer Transformer model into three sub-models, which are then deployed on three separate mobile devices. Device 1 computes the sub-models for layers 1-3, Device 2 computes the sub-models for layers 4-6, and Device 3 computes the sub-models for layers 7-8. Raw data is input to Device 1, where the sub-models perform forward propagation. The computed feature data is sent to Device 2 for further forward propagation, and the data labels needed to calculate the loss are also sent to Device 3. Device 3 then performs forward propagation. After forward propagation, Device 3 calculates the loss using the loss function and performs backpropagation to update the model weights. The gradient data is then sent to Device 2 for further backpropagation, and finally, Device 1 performs backpropagation, thus achieving distributed collaborative training.

[0028] The workflow of the in-device multiprocessor scheduling module on each mobile device is as follows: Figure 2 and Figure 3 As shown. The process of distributing attention heads to computation on various heterogeneous processors is as follows: Figure 2 As shown, specifically:

[0029] The current mobile device has K attention heads. Before distributed training begins, all processors available for neural network computation on the current mobile device are searched, with a total number of M. The time taken for each processor to execute a number of attention head computations is measured and denoted as T. k_j Where 1≤j≤M, 1≤k≤K, if a heterogeneous processor supports multiple neural network computing libraries, then the shortest computation time among all libraries is selected as T. k_j ;

[0030] Initialize the lower bound l to 0, and the upper bound r to the time T required to compute K attention heads across all heterogeneous processors. K_j The minimum value in the middle; in each iteration, calculate the intermediate value mid = (l + r) / 2, and then check if there is an allocation scheme where the total attention head computation time is close to mid. The checking method is as follows: Figure 3 As shown, the details are as follows:

[0031] For the j-th processor, find |T k_j The minimum value of -mid| is denoted as O(n). j That is, in this allocation scheme, the i-th processor is allocated k self-attention heads; if |T k_j If the minimum value of -mid| exceeds the set threshold ε, then set O. j =0 indicates that the processor is calculating too fast or too slow; if all O(0) values ​​are equal to 0, it means .... j If the sum of all O's is greater than or equal to K, then the allocation scheme is feasible. Update the original allocation scheme and set the upper bound r to mid-σ; if all O's are greater than or equal to K, then the allocation scheme is feasible. j If the sum of σ is less than K, the allocation scheme is not feasible. Update the lower bound to mid+σ, and then search for an allocation scheme again. σ is a relatively small quantity to avoid infinite loops. When l>r, the iteration ends.

[0032] Figure 4 An example of allocating six attention heads between the CPU and GPU is given. Taking device 1 as an example, assuming the self-attention mechanism of the large language model trained in the embodiment has 6 attention heads, before training begins, device 1 identifies the local processors available for neural network computation as CPU and GPU. Then, the time required for the local CPU and GPU to compute k attention heads is measured, where k = 1, 2, ..., 6. After the measurement, the module initializes its lower bound to 0 and its upper bound to the minimum time required for the CPU and GPU to compute the 6 attention heads. Then, using a binary search-based attention head allocation algorithm, an optimal allocation scheme is obtained, allocating 4 attention heads to the GPU and the remaining 2 to the CPU. The GPU and CPU perform parallel computation of the attention heads, thus accelerating the attention head computation.

[0033] The active fault-tolerant recovery mechanism of the hybrid fault-tolerant recovery module of each edge device is as follows: Figure 5As shown. When device 3 needs to exit training, it sends a notification to device 1 in advance. Device 1 then broadcasts on the local network to search for available devices. If no available devices are found, it reverts to the passive fault-tolerant recovery algorithm. If available devices are found, it collects the remaining battery percentage *b* and computing power representation vector *h* of all available devices, calculates the remaining progress *p* of the collaborative training, calculates the device suitability of each available device based on this, and selects the device with the highest suitability value as the replacement device. After finding a replacement device, training is temporarily interrupted. Device 1 broadcasts the list of collaborative training devices to all collaborative edge devices, and device 3 sends the weights of its local sub-model to the replacement device. After the replacement device initializes its corresponding sub-model and loads the weights sent from device 3, collaborative training resumes normally, and device 3 can exit training.

[0034] The computing power representation vector of the u-th idle mobile device is h. u The remaining battery percentage of the device is b. u h u The computation time of the Transformer module is used to characterize the computing power of a mobile device, and h is defined. u ={t u,1 ,t u,2 ,…,t u,L}, t u,n This represents the time required to compute an n-layer Transformer; b u It is a decimal number between 0 and 1, with a larger value indicating more remaining battery power on the mobile device; based on h u and b u The central equipment uses the Device Compatibility (DC) criterion to evaluate the compatibility of idle mobile devices, defined as follows:

[0035]

[0036] p represents the percentage of training processes remaining, equal to [B r +(TT cur [)*B]*B / T, where T represents the total number of training rounds, B represents the total number of data batches, and T cur Indicates the current training round, B r This represents the remaining batches of data in the current training round. η is a small constant greater than 0 to prevent the denominator from being 0. The normalized computing power of the device is equal to (H u -H min ) / (H max -H min ), here H u By h u H is obtained by summing all the elements in H.max and H min These are all H u The maximum and minimum values ​​in; The normalized electrical quantity of the device is equal to (b u -b min ) / (b max -b min According to the equipment compatibility criteria, the central mobile device selects the most suitable mobile device d from the local area network. s Replace the mobile device to exit. q .

[0037] This efficient and robust distributed Transformer large model training method for mobile devices boasts advantages such as low latency and high robustness. To verify the advantages of this invention, practical experiments were conducted on a distributed collaborative training system consisting of a Redmi K50, a Redmi 10X Pro, and a Xiaomi 10 Lite. The training time for ten data batches (with a batch size of 4 considering the limited memory of mobile devices) of two large Transformer models, BERT-Base and GPT-2-Medium, was measured. Experimental results showed that with the in-device multiprocessor scheduling module, the training times for these two models on three devices were 120.49 seconds and 676.06 seconds, respectively. In contrast, without the in-device multiprocessor scheduling module, the training times for these two models on three devices were 205.19 seconds and 1211.727 seconds, respectively.

[0038] The present invention also compared the fault recovery time overhead of active fault recovery and passive fault recovery algorithms on the distributed collaborative training system. By simulating the active exit event of device 2 during BERT-Base training, the execution time of different processes in the fault recovery process was measured, as shown in Table 1.

[0039]

[0040] Table 1

[0041] As can be seen from Table 1, since the device search and device replacement processes of active fault tolerance recovery are performed synchronously with training, the time for these two steps is not included in the total time cost. Therefore, the total time cost of active fault tolerance recovery is significantly lower than that of passive fault tolerance recovery, highlighting the high efficiency of this invention in fault tolerance recovery.

[0042] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0043] Although this document makes frequent use of terms such as mobile device, processor, and fault tolerance recovery, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of the invention; interpreting them as any additional limitation would contradict the spirit of the invention.

Claims

1. A distributed Transformer large model training method for mobile devices, characterized in that, There are N mobile devices, including 1 central mobile device and N - 1 collaborative mobile devices. All mobile devices are connected through a network. The training method is as follows: The Transformer large model is divided into N sub-models and deployed to N mobile devices respectively for distributed collaborative training; If the mobile device is a multi-processor mobile device, this mobile device distributes attention heads to each heterogeneous processor for calculation; In the forward propagation process of collaborative training, when 1 ≤ i < N, the i-th mobile device transmits the intermediate output obtained by local forward propagation calculation to the (i + 1)-th mobile device. If i = N, the i-th mobile device calculates the loss and starts to execute backpropagation, sending the gradient to the (i - 1)-th mobile device; In the backpropagation process of collaborative training, when 1 < i ≤ N, the i-th mobile device transmits the gradient obtained by backpropagation calculation to the (i - 1)-th mobile device. If i = 1, the i-th mobile device proceeds with the training of the next data batch; When a certain collaborative mobile device needs to withdraw from distributed training, a suitable mobile device is selected according to the fault tolerance and recovery method to replace the mobile device to be withdrawn and continue training; The specific operation of distributing attention heads to each heterogeneous processor for calculation is as follows: The current mobile device has K attention heads. Before distributed training begins, all processors available for neural network computation on the current mobile device are searched, with a total number of M. The time taken for each processor to execute the computation of k attention heads is measured and denoted as T. k_j Where 1≤j≤M, 1≤k≤K, if a heterogeneous processor supports multiple neural network computing libraries, then the shortest computation time among all libraries is selected as T. k_j ; Initialize the lower bound l to 0, and the upper bound r to the time T required to compute K attention heads across all heterogeneous processors. K_j The minimum value in the middle; in each iteration, calculate the median value mid=(l+r) / 2, and then check if there is an allocation scheme. Under this allocation scheme, the total attention head calculation time is less than or equal to (mid+ε)×110%, where ε is the calculation time deviation threshold; define the allocation scheme S={(j,O j The specific checking method is as follows: (j=1,…,M;k=1,…K) Initialize the current allocation scheme S'={}, and for the j-th processor, find |T k_j The minimum value of -mid| is denoted as O(n). j That is, to allocate k self-attention heads to the j-th processor, and to set (j, O) j Insert S'; if |T k_j If the minimum value of -mid| exceeds the set threshold ε, then set O. j =0; if all O j If the sum of O's is greater than or equal to K, then the allocation scheme S' is feasible. Update the original allocation scheme, let S = S', and set the upper bound r to mid-σ; if all O's are greater than or equal to K, then the allocation scheme S' is feasible. j If the sum of σ is less than K, the allocation scheme is not feasible. Update the lower bound to mid+σ, and then search for an allocation scheme again. σ is a relatively small quantity to avoid infinite loops. When l>r, the iteration ends.

2. The distributed Transformer large model training method for mobile devices according to claim 1, characterized in that, When a collaborative mobile device needs to exit distributed training due to a dynamic event, the mobile device d q The central mobile device is notified α time in advance; after receiving the notification, the central mobile device searches for available mobile devices in the network to participate in training via broadcast; if there are no available mobile devices in the network, the training process is recovered using a conventional passive fault-tolerant recovery algorithm. If there are idle mobile devices, all idle mobile devices will send their computing power representation vector and remaining battery percentage to the central mobile device. The computing power representation vector of the u-th idle mobile device is h. u The remaining battery percentage of the device is b. u h u The computation time of the Transformer module is used to characterize the computing power of a mobile device, and h is defined. u ={t u,1 ,t u,2 ,…,t u,L }, t u,n This represents the time required to compute n layers of Transformers; b u It is a decimal number between 0 and 1, with a larger value indicating more remaining battery power on the mobile device; based on h u and b u The central equipment uses equipment adaptability criteria to evaluate the adaptability of idle mobile devices, defined as follows: ， DC u For the adaptability of the u-th idle device, p represents the percentage of training remaining processes, equal to [B r +(TT cur )*B]*B / T, where T represents the total number of training rounds, B represents the total number of data batches, and T cur Indicates the current training round, B r This represents the remaining batches of data in the current training round, where η is a small constant greater than 0. The normalized computing power of the device is equal to (H u -H min ) / (H max -H min ), here H u By h u H is obtained by summing all the elements in H. max and H min These are all H u The maximum and minimum values ​​in; The normalized electrical quantity of the device is equal to (b u -b min ) / (b max -b min ), b max It is all b u The maximum value in, b min It is all b u The minimum value in; The central mobile device selects the DC from the local area network. u The mobile device with the largest value d s Replace the mobile device to exit. q ; then Training was briefly interrupted, d q To d s Pass the weights of the Transformer sub-model; After the weight transfer is completed, the central mobile device broadcasts a message about device replacement to all devices participating in training, and finally the distributed training resumes normal.

3. The distributed Transformer large model training method for mobile devices according to claim 2, characterized in that, The α time for notifying the central mobile device in advance is greater than the time required for the central device to select the most suitable replacement device from the LAN devices.

4. The distributed Transformer large model training method for mobile devices according to claim 2, characterized in that, The dynamic events of the mobile devices include power exhaustion and actively withdrawing from the local area network.

5. The distributed Transformer large model training method for mobile devices according to claim 1, characterized in that, The mobile devices are intelligent terminals with computing capabilities, including mobile phones, watches, microcontrollers, cameras, laptops, and desktops.

6. The distributed Transformer large model training method for mobile devices according to claim 1, characterized in that, The processors of the mobile devices are chips with computing capabilities, including CPUs, GPUs, and NPUs.

Images