Efficient robust distributed Transform large model training method for mobile equipment

Abstract

The invention discloses an efficient robust distributed Transform large model training method for a mobile device. In the distributed training process of the Transform large model, for each piece of mobile equipment participating in training, computing resources of each heterogeneous processor are collected, and on the basis of the computing resources, different numbers of self-attention heads in the Transform are distributed to the heterogeneous processors for parallel computing, so that computing acceleration of a self-attention mechanism in the Transform large model on the mobile equipment is realized; in the fault-tolerant recovery process, predictable faults generated due to the dynamic nature of the mobile equipment in the distributed training process are processed in advance, so that the fault-tolerant recovery is completed under the condition that the distributed training is not interrupted due to the faults. According to the method, the dynamic nature of the mobile equipment and the computing resources of a plurality of processors are fully utilized, and efficient and robust distributed training of the Transform model on the mobile equipment is realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology on mobile device side, and in particular to an efficient and robust distributed Transformer large model training method for mobile devices. Background Art

[0002] Edge computing can bring low latency, high security and high customization to deep learning applications. With the widespread application of large models based on Transformer, users hope that neural network models can have knowledge in specific fields, and the demand for customized neural network models is more obvious. Therefore, training neural networks on mobile devices is of great significance to the customization of deep learning applications. Federated learning is a typical distributed training architecture for mobile devices. Each mobile device independently trains a complete neural network model, and the edge server aggregates and distributes the neural network weights. However, with the release of large models based on Transformer, the parameter scale of neural network models has reached hundreds of billions. Due to memory limitations, a single device can no longer complete the training of neural network models with a scale of more than 10 billion parameters, which limits the federated learning architecture to a certain extent. In order to solve the problem of limited memory on mobile devices, a feasible solution is to split the neural network model into several sub-models and deploy them on multiple mobile devices for distributed collaborative training.

[0003] Since mobile devices use wireless networks to communicate with each other during distributed training, network disconnection and device downtime may occur during training. Considering the dynamic nature of mobile devices, the device may also run out of power or exit prematurely, which will cause the interruption of distributed training on mobile devices. When these situations occur, a fault-tolerant recovery strategy can be used to recover after the training is interrupted. However, the failures caused by the dynamic nature of mobile devices can be predicted in advance, and the current fault-tolerant recovery strategy cannot use the dynamic nature of mobile devices to reduce the time overhead of fault-tolerant recovery.

[0004] In addition, although the self-attention mechanism in the Transformer large model has the characteristic of parallel computing, the current distributed training method cannot fully utilize the computing resources of mobile devices to accelerate the calculation of the self-attention mechanism. Unlike the GPU on the server, which has excellent parallel computing capabilities compared to the CPU, the GPU on the mobile device has a computing power similar to or even weaker than the CPU, which makes it possible to use multi-processor parallel computing on mobile devices to accelerate the calculation of the self-attention mechanism. However, on the one hand, the neural network computing framework that currently supports mobile devices can only perform calculations on one processor at the same time. The calculation of the neural network on the mobile device is often completed on the CPU, and the computing resources of the mobile device are not fully utilized; on the other hand, it is necessary to allocate different numbers of attention heads in the self-attention mechanism to each processor according to the heterogeneous computing power of the processor to minimize the parallel computing time of the Transformer.

[0005] Therefore, for the distributed training of large Transformer models on mobile devices, how to make full use of the computing resources and dynamics of mobile devices to improve the efficiency and robustness of distributed training is a task that needs to be studied urgently. Summary of the invention

[0006] The present invention mainly solves the above-mentioned technical problems existing in the prior art, and provides an efficient and robust distributed Transformer large model training method for mobile devices, which realizes efficient and robust distributed training of Transformer large models on mobile devices through a multi-processor scheduling module and an active fault-tolerant recovery module in the device.

[0007] The present invention mainly solves the above technical problems through the following technical solutions: An efficient and robust distributed Transformer large model training method for mobile devices. There are N mobile devices, including 1 central mobile device and N - 1 collaborative mobile devices. All mobile devices are connected through a network. The training method is as follows: The Transformer large model is divided into N sub-models and deployed to N mobile devices respectively for distributed collaborative training. If the mobile device is a multi-processor mobile device, this mobile device distributes the attention heads to each heterogeneous processor for calculation. During the forward propagation process of collaborative training, when 1 ≤ i < N, the i-th mobile device transmits the intermediate output obtained from local forward propagation calculation to the (i + 1)-th mobile device. If i = N, the i-th mobile device calculates the loss and starts to execute backpropagation, sending the gradient to the (i - 1)-th mobile device. During the backpropagation process of collaborative training, when 1 < i ≤ N, the i-th mobile device transmits the gradient obtained from backpropagation calculation to the (i - 1)-th mobile device. If i = 1, the i-th mobile device proceeds with the training of the next data batch. When a certain collaborative mobile device needs to exit the distributed training, a suitable mobile device is selected according to the fault tolerance and recovery method to replace the mobile device to be exited and continue the training.

[0008] The central mobile device has the original training data, is responsible for managing the entire distributed training process, and is also responsible for calculating the sub-model allocated to it locally, while the collaborative mobile devices are responsible for calculating the sub-model allocated to them locally.

[0009] Preferably, the specific method of distributing the attention heads to each heterogeneous processor for calculation is as follows: There are K attention heads in the current mobile device. Before the start of distributed training, search all processors available for neural network calculation on the current mobile device, with a total of M. Measure the time taken for each processor to execute k attention head calculations, denoted as T k_j , where 1 ≤ j ≤ M, 1 ≤ k ≤ K. If a certain heterogeneous processor supports multiple neural network calculation libraries, select the shortest calculation time among all calculation libraries as T k_j ; for example, if the GPU of a mobile phone supports using two calculation libraries, OpenCL or Vulcan, to calculate neural networks, then select the one with the relatively shorter calculation time between OpenCL and Vulcan here; Initialize the lower bound l to 0 and the upper bound r to the time T required for all heterogeneous processors to calculate K attention heads K_jIn each iteration, calculate the intermediate value mid=(l+r) / 2, and then check whether there is a distribution scheme under which the total attention head calculation time is less than or equal to (mid+ε)×110%. Each processor performs the calculation of the attention head in parallel, so the total execution time is equal to the execution time of the slowest processor + the time for the processor to copy data to the CPU. Therefore, the time requirement is appropriately relaxed. ε is the calculation time deviation threshold, which is determined manually according to the model. Define the distribution scheme S={(j,O j )|j=1,…,M;k=1,…K}, the inspection method is as follows: Initialize the current allocation scheme S'={}, for the jth processor, find |T k_j -mid|, and the corresponding number of self-attention heads k is recorded as O j , that is, assign k self-attention heads to the jth processor and convert (j,O j ) insert S'; if |T k_j -mid| exceeds the set threshold ε, then set O j = 0, indicating that the processor calculates too fast or too slow; if all O j If the sum of is greater than or equal to K, it means that the allocation plan S' is feasible, update the original allocation plan, let S = S', and set the upper bound r to mid-σ; if all O j If the sum of is less than K, the allocation scheme is not feasible. Update the lower bound to mid+σ, and then search for the allocation scheme again. σ is a relatively small amount to avoid infinite loops, generally set to 0.1% of the upper bound l. When l>r, end the iteration.

[0010] In parallel computing, the total computing time depends on the slowest processor, so in this solution, it is necessary to ensure that the computing time of each processor is as close to mid as possible. Therefore, too fast or too slow is not a reasonable allocation solution.

[0011] Preferably, when the coordinated mobile device needs to exit the distributed training due to a dynamic event, the mobile device d qNotify the central mobile device α time in advance; after receiving the notification, the central mobile device searches for idle mobile devices that can participate in training in the network through broadcasting; if there are no idle mobile devices in the network, use the conventional passive fault-tolerant recovery algorithm to perform fault-tolerant recovery on the training process. The conventional passive fault-tolerant recovery algorithm includes algorithms based on weight backup and model redistribution. For details, see Li P, Koyuncu E, SeferogluH. Respipe: Resilient model-distributeddnn training at edge networks[C] / / ICASSP 2021-2021 IEEE InternationalConference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021:3660-3664., or Chen Y, Yang Q, He S, etal. Ftpipehd: A fault-tolerant pipeline-parallel distributed training approach for heterogeneous edge devices[J].IEEE Transactions on Mobile Computing, 2023, 23(4): 3200-3212., or Ye S, ZengL, Chu X, et al. Asteroid: Resource-EfficientHybrid Pipeline Parallelism forCollaborative DNN Training on Heterogeneous Edge Devices[C] / / Proceedings ofthe 30th Annual International Conference on Mobile Computing and Networking.2024: 312-326.; If there are idle mobile devices, all idle mobile devices send the computing power representation vector and the percentage of remaining power of the device to the central mobile device. The computing power representation vector of the uth idle mobile device is h u , the remaining power percentage of the device is b u ;h u The computing time of the Transformer module is used to characterize the computing power of the mobile device. We define h u ={t u,1 ,t u,2 ,…,tu,L}, t u,n represents the time required to calculate n-layer Transformer; b u It is a decimal between 0 and 1. The larger the value, the more remaining power the mobile device has. u and b u The central device uses the Device Compatibility (DC) criterion to evaluate the compatibility of idle mobile devices, defining: , DC u is the adaptability of the uth idle device, p represents the percentage of the remaining training process, which is equal to [B r +(TT cur )*B]*B / T, T represents the total training rounds, B represents the total data batches, T cur Indicates the current training round, B r Represents the remaining data batches of the current training round. η is a small constant greater than 0 to prevent the denominator from being 0. Represents the normalized computing power of the device, equal to (H u -H min ) / (H max -H min ), where H u By h u The sum of all elements in H is max and H min All H u The maximum and minimum values ​​in ; Represents the normalized power of the device, equal to (b u -b min ) / (b max -b min ), b max Yes all b u The maximum value in b min Yes all b u According to the device compatibility criteria, the central mobile device selects the most suitable (DC u Maximum value) mobile device d s Replace the mobile device you want to exit q ; The above process is carried out synchronously with the collaborative training and will not interrupt the training; then, the training is briefly interrupted, d q Towards d sThe weights of the Transformer sub-model are transferred; after the weight transfer is completed, the central mobile device broadcasts the message of device replacement to all devices participating in the training, and finally the distributed training returns to normal; the device adaptability criterion considers the computing resources and remaining power of the mobile device when selecting the mobile device to ensure the stability of the training. At the beginning of the training, the value of p is large, and the device adaptability criterion will focus more on the remaining power of the mobile device; as the training progresses, the value of p gradually decreases, and the device adaptability criterion will pay more attention to the computing power of the mobile device.

[0012] Preferably, the α time for notifying the central mobile device in advance is greater than the time required for the central device to select the most suitable replacement device from the local area network devices.

[0013] Preferably, the dynamic events of the mobile device include power exhaustion and active exit from the local area network.

[0014] Preferably, the mobile device is an intelligent terminal with computing capability, including a mobile phone, a watch, a single-chip microcomputer, a camera, a laptop computer, a desktop computer, etc.

[0015] Preferably, the processor of the mobile device is a chip with computing capability, including CPU, GPU and NPU, etc. Homogeneous processors are special cases of heterogeneous processors, and our solution is also applicable to homogeneous processors, and the allocation result given may be average allocation.

[0016] The substantial effects brought by the present invention are: (1) The multi-processor scheduling method can distribute the attention heads in the Transformer-based large language model to multiple processors for parallel computing according to the computing power of each heterogeneous processor of the edge device, thereby accelerating the computing of the large language model on the edge device; (2) The active fault-tolerant recovery method enables collaborative training to respond in advance to training interruptions caused by dynamic events of mobile devices, reduces the time overhead caused by fault-tolerant recovery, and improves the robustness of the collaborative training method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of an efficient and robust distributed Transformer large model training method for mobile devices of the present invention; Figure 2 , Figure 3 and Figure 4 It is a schematic diagram of calculation of a multi-processor scheduling module in a device in an efficient and robust distributed Transformer large model training method for mobile devices of the present invention; Figure 5 It is a schematic diagram of an active fault-tolerant recovery module in an efficient and robust distributed Transformer large model training method for mobile devices of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0019] Embodiment: An efficient and robust distributed Transformer large model training method for mobile devices, comprising three mobile devices, such as Figure 1 As shown, device 1 is a central mobile device, which has the original data to be trained; device 2 and device 3 are collaborative mobile devices. The three mobile devices are connected to the same router, using IP addresses as identifiers, and communicate through wireless networks and HTTP requests. Each mobile device is equipped with an application that implements the present invention, using MNN as a neural network computing framework, which is a computing framework that supports neural network training on mobile devices.

[0020] The central mobile device divides an eight-layer Transformer model into three sub-models and deploys them on three mobile devices respectively. Device 1 is responsible for calculating the sub-models of layers 1-3, device 2 is responsible for calculating the sub-models of layers 4-6, and device 3 is responsible for calculating the sub-models of layers 7-8. After the original data is input into device 1, the sub-model on device 1 performs forward propagation, and the calculated feature data is sent to device 2 for further forward propagation. At the same time, the data label required for calculating the loss is also sent to device 3. Device 3 then performs forward propagation. After the forward propagation of device 3 is completed, the corresponding loss is calculated using the loss function and back-propagation is performed to update the model weights. The gradient data is sent to device 2 for further back-propagation, and then device 1 performs back-propagation to achieve distributed collaborative training.

[0021] The workflow of the device multi-processor scheduling module on each mobile device is as follows Figure 2 and Figure 3 The process of allocating attention heads to various heterogeneous processors for calculation is shown in Figure 2 As shown, specifically: There are K attention heads in the current mobile device. Before the distributed training starts, all processors that can be used for neural network calculations are searched on the current mobile device. The total number is M. The time it takes for each processor to perform several attention head calculations is measured, recorded as T k_j , where 1≤j≤M, 1≤k≤K. If a heterogeneous processor supports multiple neural network computing libraries, the shortest computing time among all computing libraries is selected as T k_j ; Initialize the lower bound l to 0 and the upper bound r to the time T required to compute K attention heads on all heterogeneous processors. K_jIn each iteration, calculate the intermediate value mid=(l+r) / 2, and then check whether there is a distribution scheme under which the total attention head calculation time is close to mid. The checking method is as follows: Figure 3 As shown, the details are as follows: For the jth processor, find |T k_j -mid|, and the corresponding number of self-attention heads k is recorded as O j , that is, in this allocation scheme, the i-th processor allocates k self-attention heads; if |T k_j -mid| exceeds the set threshold ε, then set O j = 0, indicating that the processor calculates too fast or too slow; if all O j If the sum of is greater than or equal to K, it means that the allocation plan is feasible, update the original allocation plan, and set the upper bound r to mid-σ; if all O j If the sum of is less than K, the allocation scheme is infeasible, update the lower bound to mid+σ, and then search for the allocation scheme again; σ is a relatively small amount to avoid infinite loops; when l>r, end the iteration.

[0022] Figure 4 An example of allocating six attention heads between the CPU and GPU is given. Taking device 1 as an example, assuming that the self-attention mechanism of the large language model trained in the embodiment has 6 attention heads, before the training starts, device 1 finds that the local processors that can be used for neural network calculations are CPU and GPU, and then measures the time required for the local CPU and GPU to calculate k attention heads, where k=1,2,…,6. After the measurement, the module initializes the lower bound to 0 and the upper bound to the minimum value of the time required to calculate the 6 attention heads in the CPU and GPU. Then, through the attention head allocation algorithm based on binary search, an optimal allocation scheme is obtained, and 4 attention heads are allocated to the GPU, and the remaining 2 attention heads are allocated to the CPU. The GPU and CPU perform parallel calculations on the attention heads to achieve accelerated attention head calculations.

[0023] The active fault-tolerant recovery mechanism of the hybrid fault-tolerant recovery module of each edge device is as follows: Figure 5As shown. When device 3 needs to exit training, device 3 sends a notification to device 1 in advance. Device 1 then broadcasts in the local area network to find out whether there are available devices. If there are no available devices, it falls back to the passive fault-tolerant recovery algorithm. If there are available devices, the remaining power percentage b and computing power characterization vector h of all available devices are collected, and the remaining process p of collaborative training is calculated. Based on this, the device adaptability of each available device is calculated, and the device with the highest device suitability value is selected as the replacement device. After finding the replacement device, the training is temporarily interrupted, and device 1 broadcasts the list of devices for collaborative training to all collaborative edge devices, and device 3 sends the weights of the local sub-model to the replacement device. After the replacement device initializes the corresponding sub-model, it loads the weights sent from device 3. After the weights are loaded, the collaborative training returns to normal, and device 3 can exit training.

[0024] The computing power representation vector of the uth idle mobile device is h u , the remaining power percentage of the device is b u ;h u The computing time of the Transformer module is used to characterize the computing power of the mobile device. We define h u ={t u,1 ,t u,2 ,…,t u,L}, t u,n represents the time required to calculate n-layer Transformer; b u It is a decimal between 0 and 1. The larger the value, the more remaining power the mobile device has. u and b u The central device uses the Device Compatibility (DC) criterion to evaluate the compatibility of idle mobile devices, defining: , p represents the percentage of the remaining training process, which is equal to [B r +(TT cur )*B]*B / T, T represents the total training rounds, B represents the total data batches, T cur Indicates the current training round, B r Represents the remaining data batches of the current training round. η is a small constant greater than 0 to prevent the denominator from being 0. Represents the normalized computing power of the device, equal to (H u -H min ) / (H max -H min ), where H u By h u The sum of all elements in H is max and H minAll H u The maximum and minimum values ​​in ; Represents the normalized power of the device, equal to (b u -b min ) / (b max -b min ) According to the device compatibility criteria, the central mobile device selects the most suitable mobile device from the local area network d s Replace the mobile device you want to exit q .

[0025] This efficient and robust distributed Transformer large model training method for mobile devices has the advantages of low latency and high robustness. In order to verify the advantages of the present invention, the present invention conducted actual experiments on a distributed collaborative training system consisting of three mobile phones, Redmi K50, Redmi 10X Pro and Xiaomi 10 Lite, and measured the time required to train ten data batches on the two Transformer large models BERT-Base and GPT-2-Medium (considering the limited memory of mobile devices, the data batch size is set to 4). The experimental test shows that after using the multi-processor scheduling module in the device, the time required for the three devices to collaboratively train the two models is 120.49 seconds and 676.06 seconds respectively. In contrast, when the multi-processor scheduling module in the device is not used, the time required for the three devices to collaboratively train the two models is 205.19 seconds and 1211.727 seconds respectively.

[0026] The present invention also compares the fault-tolerance recovery time overhead of active fault-tolerance recovery and passive fault-tolerance recovery algorithms on the distributed collaborative training system. By simulating the active exit event of device 2 during BERT-Base training, the execution time of different processes in the fault-tolerance recovery process is measured, as shown in Table 1.

[0027] Table 1 As can be seen from Table 1, since the device search and device replacement process of active fault-tolerant recovery overhead is performed synchronously with training, the time of these two steps is not included in the total time overhead. It can be seen that the total time overhead of active fault-tolerant recovery is much lower than the total time overhead of passive fault-tolerant recovery, highlighting the high efficiency of the present invention in fault-tolerant recovery.

[0028] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0029] Although the terms such as mobile device, processor, and fault-tolerant recovery are used more frequently in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

Claims

1. An efficient and robust distributed Transformer large model training method for mobile devices, characterized in that: There are N mobile devices, including 1 central mobile device and N - 1 collaborative mobile devices. All mobile devices are connected through a network. The training method is as follows: The Transformer large model is divided into N sub-models and deployed to N mobile devices respectively for distributed collaborative training; If the mobile device is a multi-processor mobile device, this mobile device distributes the attention heads to each heterogeneous processor for calculation; During the forward propagation process of collaborative training, when 1 ≤ i < N, the i-th mobile device transmits the intermediate output obtained from local forward propagation calculation to the (i + 1)-th mobile device. If i = N, the i-th mobile device calculates the loss and starts to perform backpropagation, sending the gradient to the (i - 1)-th mobile device; During the backpropagation process of collaborative training, when 1 < i ≤ N, the i-th mobile device transmits the gradient obtained from backpropagation calculation to the (i - 1)-th mobile device. If i = 1, the i-th mobile device proceeds to train the next data batch; When a certain collaborative mobile device needs to exit the distributed training, a suitable mobile device is selected according to the fault tolerance and recovery method to replace the mobile device to be exited and continue the training.

2. According to claim 1, an efficient and robust distributed Transformer large model training method for mobile devices is characterized in that: The specific operation of distributing the attention heads to each heterogeneous processor for calculation is as follows: There are K attention heads in the current mobile device. Before the distributed training starts, all processors that can be used for neural network calculations are searched on the current mobile device. The total number is M. The time it takes for each processor to perform k attention head calculations is measured, recorded as T k_j , where 1≤j≤M, 1≤k≤K. If a heterogeneous processor supports multiple neural network computing libraries, the shortest computing time among all computing libraries is selected as T k_j ; Initialize the lower bound l to 0 and the upper bound r to the time T required to compute K attention heads on all heterogeneous processors. K_j In each iteration, calculate the intermediate value mid=(l+r) / 2, and then check whether there is an allocation scheme under which the total attention head calculation time is less than or equal to (mid+ε)×110%, where ε is the calculation time deviation threshold; define the allocation scheme S={(j,O j )|j=1,…,M;k=1,…K}, the inspection method is as follows: Initialize the current allocation scheme S'={}, for the jth processor, find |T k_j -mid|, and the corresponding number of self-attention heads k is recorded as O j , that is, assign k self-attention heads to the jth processor and convert (j,O j ) insert S'; if |T k_j -mid| exceeds the set threshold ε, then set O j =0; if all O j If the sum of is greater than or equal to K, it means that the allocation plan S' is feasible, update the original allocation plan, let S = S', and set the upper bound r to mid-σ; if all O j If the sum of is less than K, the allocation scheme is infeasible, update the lower bound to mid+σ, and then search for the allocation scheme again; σ is a relatively small amount to avoid infinite loops; when l>r, end the iteration.

3. The efficient and robust distributed Transformer large model training method for mobile devices according to claim 1 or 2, characterized in that: When a coordinated mobile device needs to exit distributed training due to a dynamic event, the mobile device d q Notify the central mobile device α time in advance; after receiving the notification, the central mobile device searches for idle mobile devices that can participate in training in the network through broadcasting; if there are no idle mobile devices in the network, use the conventional passive fault-tolerant recovery algorithm to perform fault-tolerant recovery on the training process; If there are idle mobile devices, all idle mobile devices send the computing power characterization vector and the remaining power percentage of the device to the central mobile device. The computing power characterization vector of the uth idle mobile device is h u , the remaining power percentage of the device is b u ;h u The computing time of the Transformer module is used to characterize the computing power of the mobile device. We define h u ={t u,1 ,t u,2 ,…,t u,L }, t u,n Indicates the time required to calculate n layers of Transformer; b u It is a decimal between 0 and 1. The larger the value, the more remaining power the mobile device has. u and b u , the central device uses the device suitability criteria to evaluate the suitability of idle mobile devices, defining: ， DC u is the adaptability of the uth idle device, p represents the percentage of the remaining training process, which is equal to [B r +(TT cur )*B]*B / T, T represents the total training rounds, B represents the total data batches, T cur Indicates the current training round, B r Represents the remaining data batches of the current training round, η is a small constant greater than 0; Represents the normalized computing power of the device, equal to (H u -H min ) / (H max -H min ), where H u By h u The sum of all elements in H is max and H min All H u The maximum and minimum values ​​in ; Represents the normalized power of the device, equal to (b u -b min ) / (b max -b min ), b max Yes all b u The maximum value in b min Yes all b u The minimum value in ; The central mobile device selects a DC from the LAN u The mobile device with the largest value d s Replace the mobile device you want to exit q ; then , training was briefly interrupted, d q Towards d s Pass the weights of the Transformer sub-model; After the weight transmission is completed, the central mobile device broadcasts a message about device replacement to all devices participating in the training, and finally the distributed training resumes normally.

4. According to claim 3, an efficient and robust distributed Transformer large model training method for mobile devices is characterized in that: The α time for notifying the central mobile device in advance is greater than the time required for the central device to select the most suitable replacement device from the LAN devices.

5. According to claim 3, an efficient and robust distributed Transformer large model training method for mobile devices is characterized in that: The dynamic events of the mobile device include power exhaustion and actively exiting the local area network.

6. According to claim 1, an efficient and robust distributed Transformer large model training method for mobile devices is characterized in that: The mobile device is an intelligent terminal with computing capabilities, including mobile phones, watches, single-chip microcomputers, cameras, laptops, desktop computers, etc.

7. According to claim 2, an efficient and robust distributed Transformer large model training method for mobile devices is characterized in that: The processor of the mobile device is a chip with computing capabilities, including CPUs, GPUs, NPUs, etc.

Images