Industrial internet of things edge-end collaborative model splitting and resource scheduling joint optimization method and system

Abstract

The application provides an industrial Internet of Things edge-end cooperative model splitting and resource scheduling joint optimization method and system, which comprises the following steps: dividing a training sample batch into m sub-batches, dividing an encoder of a to-be-trained model into S segments, each segment comprising a plurality of encoder blocks, and distributing the segments to k computing devices in a device cluster; sequentially sending embedded features of training samples in the sub-batches to the first computing device; receiving the kth intermediate data sent by the kth computing device and sending the kth intermediate data to an edge server, wherein the n+1th intermediate data is calculated by the n+1th computing device based on the nth intermediate data and the segment of the n+1th computing device; receiving the decoder gradient sent by the edge server and sending the decoder gradient to the kth computing device; and receiving the updated encoder parameters sent by the k computing devices respectively and sending the updated encoder parameters to the edge server. The application can improve the training efficiency of a large language model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for joint optimization of model decomposition and resource scheduling in industrial Internet of Things (IoT) edge-device collaboration. Background Technology

[0002] With the rapid development of AI-generated content (AIGC) technology in mobile IoT, AI multi-agent systems based on Large Language Models (LLM) are becoming a key driver of intelligent applications in mobile edge computing networks. By leveraging the semantic understanding capabilities of LLM, multi-agent systems can quickly utilize raw data and knowledge, promoting autonomous learning and decision-making in distributed IoT and supporting various scenarios, such as drone inspections of substations and power system operation optimization. Therefore, future mobile edge computing networks are expected to optimize the quality of distributed LLM services by providing massive amounts of data, idle computing resources, and autonomous continuous learning across data domains.

[0003] Optimizing LLM requires a large amount of high-quality data, which raises the risk of data privacy breaches. Therefore, many fields are unable to conduct effective collaborative learning due to data privacy protection constraints. To address this issue, federated learning methods for LLM training have been proposed and adopted. By training local models to form distributed knowledge, federated learning reduces the risk of user data leakage and enhances the generalization ability of edge-side models by embedding global knowledge. Nevertheless, the practical deployment of federated learning-based LLM training tasks still faces significant challenges. First, LLM training requires substantial computational and memory resources, exceeding the capabilities of a single device. Second, cloud-edge-device collaborative distributed LLM training requires frequent communication, resulting in excessive communication overhead and low resource utilization. These challenges contribute to the relatively low training efficiency of existing large language models. Summary of the Invention

[0004] This invention provides a method and system for joint optimization of model decomposition and resource scheduling in industrial IoT edge-device collaboration, which addresses the shortcomings of low training efficiency of large language models in the prior art and improves the training efficiency of large language models.

[0005] This invention provides a joint optimization method for model decomposition and resource scheduling in industrial IoT edge-device collaboration, comprising: Obtain training sample batches, divide the training sample batches into m sub-batches, divide the encoder of the model to be trained into S segments, each segment including multiple encoder blocks, and allocate the segments to k computing devices in the device cluster. The embedded features of the training samples in the sub-batch are sequentially sent to the first of the k computing devices, which are arranged according to the connection order of the encoder in the model to be trained. Receive the kth intermediate data sent by the kth computing device, and send the kth intermediate data to the edge server, wherein the (n+1)th intermediate data is calculated by the (n+1)th computing device based on the nth intermediate data and the segment of the (n+1)th computing device; Receive the decoder gradient sent by the edge server and send the decoder gradient to the k-th computing device; The updated encoder parameters are received from the k computing devices respectively, and then sent to the edge server.

[0006] According to the present invention, a joint optimization method for model splitting and resource scheduling for edge-to-edge collaboration in industrial IoT includes, before dividing the training sample batch into m sub-batches, the following steps are taken: Obtain the values ​​of the parameters to be optimized, including the number of sub-batches m, the number of segments S, and the encoders in each of the k computing devices; The value of the parameter to be optimized is obtained by optimizing the parameter to minimize the maximum training latency of each of the device clusters. The parameter to be optimized includes the number of sub-batches m, the number of segments S, and the number of encoder blocks in each of the k computing devices. .

[0007] According to the present invention, a joint optimization method for model splitting and resource scheduling for edge-device collaboration in industrial IoT is provided. The training latency includes a first training latency and a second training latency. The first training latency includes the computation latency of devices within the device cluster and the transmission latency between devices. The second training latency includes the transmission latency between the device cluster and the edge server.

[0008] According to the present invention, a joint optimization method for model decomposition and resource scheduling of industrial IoT edge-device collaboration is provided, wherein the parameters to be optimized further include the transmission bandwidth between the edge server and the transmission channel configuration between the edge server.

[0009] According to the present invention, a joint optimization method for model decomposition and resource scheduling in industrial IoT edge-device collaboration is provided, wherein the optimization of the parameters to be optimized aims to minimize the maximum value among the training delays of multiple device clusters, and includes: The maximum value of the total training latency for each of the device clusters across multiple training rounds is obtained. The optimization of the parameters to be optimized is then performed with the goal of minimizing the maximum value of the total training latency.

[0010] According to the present invention, a joint optimization method for model decomposition and resource scheduling in industrial IoT edge-device collaboration is provided, wherein the optimization of the parameters to be optimized is performed with the objective of minimizing the maximum training latency among multiple device clusters, including: Using Lyapunov optimization theory, the optimization problem, which aims to minimize the maximum training latency of each of the device clusters, is transformed into a virtual stable queue for solution.

[0011] This invention also provides a model training system based on joint optimization of task splitting and orchestration and resource scheduling, comprising: The task splitting and resource scheduling module is used to obtain training sample batches, divide the training sample batches into m sub-batches, divide the encoder of the model to be trained into S segments, each segment including multiple encoder blocks, and allocate the segments to k computing devices in the device cluster. The number of encoder blocks allocated to each of the k computing devices is [number missing]. ; The forward inference module is used to sequentially send the embedded features of the training samples in the sub-batch to the first computing device among the k computing devices, and the k computing devices are arranged according to the connection order of the encoder in the model to be trained; The transmission module is used to receive the kth intermediate data sent by the kth computing device and send the kth intermediate data to the edge server, wherein the (n+1)th intermediate data is calculated by the (n+1)th computing device based on the nth intermediate data and the segment of the (n+1)th computing device; A receiving module is used to receive the decoder gradient sent by the edge server and send the decoder gradient to the k-th computing device; The parameter update module is used to receive the updated encoder parameters sent by the k computing devices respectively, and send the updated encoder parameters to the edge server.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-mentioned methods for joint optimization of model decomposition and resource scheduling for edge-to-edge collaboration in the Industrial Internet of Things.

[0013] The present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the above-mentioned methods for joint optimization of model decomposition and resource scheduling for edge-to-edge collaboration in the Industrial Internet of Things.

[0014] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for joint optimization of model decomposition and resource scheduling for edge-to-edge collaboration in the Industrial Internet of Things.

[0015] The present invention provides a joint optimization method and system for model splitting and resource scheduling in industrial IoT edge-device collaboration. This method divides the training sample batches in each device cluster into multiple sub-batches and divides the encoder of the model to be trained into S segments, each segment including multiple encoder blocks. These encoder blocks are then allocated to k computing devices in the device cluster. Within the device cluster, each computing device outputs its intermediate data to the next computing device based on the intermediate data output by the previous computing device and its locally allocated encoder blocks. After obtaining the intermediate data output by the last computing device, the intermediate data is sent to an edge server, which performs decoder inference and gradient calculation. Upon receiving the decoder gradient from the edge server, the decoder gradient is backpropagated among the k computing devices. Each computing device updates the parameters of its allocated encoder. Finally, the updated encoder parameters of the k computing devices are sent to the edge server to achieve a global model update. In this process, the encoder part of the large language model is divided into segments of different sizes and deployed on computing devices for pipelined parallel training, while the decoder is deployed on edge server nodes to process generative AI tasks. Then, global model updates are performed through federated aggregation, realizing efficient collaboration between edge nodes in the distributed training system and improving the training efficiency of the large language model. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the joint optimization method for model decomposition and resource scheduling in industrial IoT edge-to-edge collaboration provided by the present invention.

[0018] Figure 2 This is a schematic diagram of the deployment of the LLM module of the distributed mobile edge IoT in the joint optimization method of model decomposition and resource scheduling for edge-to-edge collaboration of industrial IoT provided by the present invention.

[0019] Figure 3 This invention provides a federated learning workflow for splitting a communication round in a heterogeneous network within the joint optimization method for model decomposition and resource scheduling in the industrial IoT edge-to-edge collaboration.

[0020] Figure 4 This is a schematic diagram of the parallel model training pipeline in the joint optimization method of model decomposition and resource scheduling for edge-to-edge collaboration in the Industrial Internet of Things provided by this invention.

[0021] Figure 5 This is a schematic diagram of the experimental results of the joint optimization method for model decomposition and resource scheduling in the industrial IoT edge-device collaboration provided by this invention. Figure 1 .

[0022] Figure 6 This is a schematic diagram of the experimental results of the joint optimization method for model decomposition and resource scheduling in the industrial IoT edge-device collaboration provided by this invention. Figure 2 .

[0023] Figure 7 This is a schematic diagram of the experimental results of the joint optimization method for model decomposition and resource scheduling in the industrial IoT edge-device collaboration provided by this invention. Figure 3 .

[0024] Figure 8 This is a schematic diagram of the experimental results of the joint optimization method for model decomposition and resource scheduling in the industrial IoT edge-device collaboration provided by this invention. Figure 4 .

[0025] Figure 9 This is a schematic diagram of the structure of the industrial IoT edge-to-edge collaborative model decomposition and resource scheduling joint optimization system provided by the present invention.

[0026] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The following is combined with Figures 1-8 The present invention describes the joint optimization method for model decomposition and resource scheduling in industrial IoT edge-device collaboration, such as... Figure 1 As shown, the method includes the following steps: S110. Obtain training sample batches, divide the training sample batches into m sub-batches, divide the encoder of the model to be trained into S segments, each segment including multiple encoder blocks, and allocate the segments to k computing devices in the device cluster. S120. The embedded features of the training samples in the sub-batch are sent sequentially to the first computing device among the k computing devices. The k computing devices are arranged according to the connection order of the encoder in the model to be trained. S130. Receive the kth intermediate data sent by the kth computing device, and send the kth intermediate data to the edge server. The (n+1)th intermediate data is calculated by the (n+1)th computing device based on the nth intermediate data and the segment of the (n+1)th computing device. S140: Receive the decoder gradient sent by the edge server and send the decoder gradient to the kth computing device; S150: Receive the updated encoder parameters sent by k computing devices respectively, and send the updated encoder parameters to the edge server.

[0029] The present invention provides a joint optimization method for model splitting and resource scheduling in industrial IoT edge-device collaboration. This method divides the training sample batches in each device cluster into multiple sub-batches and divides the encoder of the model to be trained into S segments, each segment including multiple encoder blocks. These encoder blocks are then allocated to k computing devices in the device cluster. Within the device cluster, each computing device outputs its intermediate data to the next computing device based on the intermediate data output by the previous computing device and its locally allocated encoder blocks. After obtaining the intermediate data output by the last computing device, the intermediate data is sent to an edge server, which performs decoder inference and gradient calculation. Upon receiving the decoder gradient from the edge server, the decoder gradient is backpropagated among the k computing devices. Each computing device updates the parameters of its allocated encoder. Finally, the updated encoder parameters of the k computing devices are sent to the edge server to achieve a global model update. In this process, the encoder part of the large language model is divided into segments of different sizes and deployed on computing devices for pipelined parallel training, while the decoder is deployed on edge server nodes to process generative AI tasks. Then, global model updates are performed through federated aggregation, realizing efficient collaboration between edge nodes in the distributed training system and improving the training efficiency of the large language model.

[0030] In federated learning, considering data privacy, training sample data from different domains are not directly shared. In the method provided in this application, a device cluster is used to process training sample data within a domain. The original training sample data is forward inferred in the device cluster through the encoder part of the large model and then sent to the edge server, where the decoder part of the large model is deployed.

[0031] In the method provided in this application, such as Figure 2As shown on the left, an edge server node consists of N base stations, forming an edge cloud network. Within the network, devices are organized into different clusters. Each device cluster includes K end nodes. One designated end node is the Control Unit (CU), responsible for executing the method provided in this application, managing data storage and coordination within the device cluster, and the remaining end nodes are computing devices. The Transformer-based LLM is modular and can be deployed as follows: Figure 2 As shown on the right, edge nodes (i.e., edge servers) are responsible for initializing the complete global model, training the decoder module, and aggregating the local models. Within each cluster, the encoder module is pre-trained using pipelined parallelism. The CU stores local training data and executes the embedding layers, while encoder segments are adaptively allocated to devices based on their capabilities. These edge nodes then collaborate to train the encoder module in a distributed, parallel manner.

[0032] To achieve efficient collaborative training in heterogeneous networks, this application provides a hybrid parallel-based split federated learning framework (CollaPipe), which organizes the learning process into two levels: end-to-end (Device-to-Device, D2D) collaboration within the cluster and end-to-edge (Device-to-Edge, D2E) collaboration within the network. These two collaborations are explained below.

[0033] (1) D2D collaboration like Figure 2 As shown on the left, each cluster consists of K end nodes and one CU, interconnected via D2D communication. The LLM encoder is divided into S sequence model segments, each segment consisting of a set of Transformer Encoder Blocks (TEBs). The number of TEBs allocated to end node k is determined by... The set of end nodes is represented as... During training, end nodes collaborate to perform pipelined parallel learning by exchanging intermediate activations, labels, and gradients, thereby enabling efficient distributed training across different end nodes.

[0034] (2) D2E collaboration Units (CUs) from N clusters are responsible for sending local encoder parameters to micro base stations (BSs) for federated learning and determining the segment granularity and matching strategy for pipeline parallelism within the cluster. In each round of communication, the CUs collect and connect the parameters of the segments to assemble the complete encoder, and then upload the parameters to the BS for further integration and federated aggregation. This represents the set of available radio frequency bands. Orthogonal Frequency Division Multiple Access (OFDMA) technology is used to transmit local encoder parameters from the cluster to the base station in parallel. In each round of communication, the BS determines the cluster selection strategy, selects J CUs, and allocates available channels to them. After receiving updated encoder parameters from different clusters, the BS performs subsequent training on the decoder module and performs joint aggregation to update the parameters of the global LLM.

[0035] The method provided in this application targets an LLM based on an encoder-decoder architecture, with the Transformer as its backbone. The entire LLM is considered to consist of two main modules: the encoder and the decoder. The model is divided into S segments using a TEB (Transformer-Based Module) approach for collaborative deployment and training. This indicates parameterization of the i-th model segment in cluster n, where p represents the dimension of the model segment parameter vector. The encoder of the n-th cluster is... Parameterization, where e is the dimension of the parameter vector. Let... This represents the initial local dataset for the nth CU, where each CU is in... The upper layer executes the embedding layer, which is responsible for selecting S end nodes to execute the encoder sequentially.

[0036] In real-world mobile edge computing scenarios, the computing power and storage space of end nodes vary significantly. Therefore, it is necessary to design effective TEB-level allocation, system power control, and wireless resource scheduling strategies to ensure the efficiency of the edge learning system and the quality of the transmission model. Assume that all end nodes in each cluster have different computing capacities, and different clusters share a single CU interacting with the BS. The BS typically possesses substantial computing and communication resources. Inspired by the GPipe algorithm, a hybrid parallel training strategy based on micro-batch scheduling is designed, and a federated learning algorithm called CollaPipe is proposed. The goal of CollaPipe is to find the optimal LLM model parameters that minimize training loss. .set up Let represent the local loss function of cluster n, then the learning objective is as follows:

[0037] in, This represents the complete global LLM parameters. These are the local decoder parameters for the nth cluster. The symbols in formulas (3) and (4) are... This represents the matrix concat operation.

[0038] In each training iteration round of each cluster, the following steps are executed sequentially.

[0039] (1) Determination of key hyperparameters First, each cluster's CU executes the embedding layer based on its local data. Then, the output from the embedding layer is sent to the next end node. Therefore, the LLM encoder needs to be divided into several segments. These segments are executed sequentially from the CU to the last selected Kth end node using a micro-batch-based pipelined parallel algorithm. Therefore, the number of micro-batches, denoted by m, must also be determined. Let b represent the total batch size. The micro-batch size in the pipelined parallelism is determined by... It is represented and defined as:

[0040] The pipelined parallel learning process involves communication and scheduling between end nodes. Rules can be set to ensure that the D2D transmission latency plus the computation latency of each micro-batch is consistent, so that the working time of each stage does not overlap, thus ensuring the effective operation of the algorithm (details will be introduced later).

[0041] (2) Segmented scheduling Because each end node has limited computational, memory, and radio resources, and its available idle resources fluctuate over time, the CU needs to determine the number of TEBs trained on each end node to ensure efficient operation of the split pipeline parallel learning.

[0042] (3) Local forward propagation of LLM encoder Within each cluster n, the forward propagation of the LLM encoder is performed during the t-th FL communication round using a pipelined scheduling method. Each end node k assigned to this task receives input from its previous end node. Receive intermediate data It performs local computations through multiple Transformer layers, continuously generating new intermediate data. and pass them to the next end node. The last end node in the sequence uploads the intermediate data of the last segment of the encoder to the server, where the forward propagation of the decoder is performed. It is worth noting that in the method provided in this application, not all computing devices in a cluster are assigned to the training task; the number k is only used to indicate the task dependency order between devices.

[0043] (4) Global training of LLM decoder The base station edge node receives intermediate data from N clusters. Then, the results are directly fed back to the decoder module. Based on the predicted output, the loss value is calculated, the loss function value is calculated with respect to the decoder weight gradient, and the decoder model parameters are updated accordingly.

[0044] (5) Local backpropagation of LLM encoder The last end node in each cluster receives the gradient activated by the decoder from the server. Subsequently, cluster n performs backpropagation in the reverse order of the end nodes during forward propagation.

[0045] (6) Global model aggregation and update After backpropagation, each segment and decoder uses stochastic gradient descent (SGD) to update the model parameters. Each end node updates its model parameters. The parameters are sent to the cluster head (CU) of its cluster, where they are concatenated at CU n. The encoder parameters are then uploaded to the server, where an FL aggregation operation is performed to update the global full model. , means as follows:

[0046] in Let represent the aggregation gradient in round t.

[0047] Following the chain rule of SGD, the server calculates the average gradient of N local full models, as follows:

[0048] in

[0049] The training process for a single cluster and server described above is as follows: Figure 3 It is displayed in the middle.

[0050] Table 1 details the pseudocode of CollaPipe. Initially, pipeline parallelism is implemented within each cluster (lines 5 to 20). Subsequently, FL (Federated Learning) is executed to facilitate collaborative training across clusters (lines 21 to 24).

[0051] Table 1

[0052] Since different computing devices have different computing capabilities, how to partition the encoder and which encoder blocks to allocate to which computing devices will affect the inference computation time of the training samples, and thus the training latency of the training data. The method provided in this application adaptively partitions and allocates the encoder based on the actual situation of the computing devices in each cluster. Specifically, in one possible implementation, before dividing the training sample batch into m sub-batches, the following steps are included: Obtain the values ​​of the parameters to be optimized, including the number of sub-batches m, the number of segments S, and the encoders in each of the k computing devices; The values ​​of the parameters to be optimized are obtained by optimizing the parameters to minimize the maximum training latency of each device cluster. The parameters to be optimized include the number of sub-batches m, the number of segments S, and the number of encoder blocks in each of the k computing devices. .

[0053] In practical applications, the transmission between computing devices and the data transmission between the CU and the edge server nodes also require a certain amount of time. Therefore, in one possible implementation, the training latency caused by data transmission is also optimized. That is, the training latency includes a first training latency and a second training latency. The first training latency includes the computing latency of devices within the device cluster and the transmission latency between devices; the second training latency includes the transmission latency between the device and the edge server.

[0054] Transmission latency is related to transmission channel configuration and transmission bandwidth. Therefore, in one possible implementation, the parameters to be optimized further include the transmission bandwidth between the CU and the edge server, as well as the transmission channel configuration between the CU and the edge server.

[0055] Using the determined first and second training delays, the communication model in the method provided in this application is constructed as follows.

[0056] The CollaPipe framework operates in heterogeneous networks, considering D2D communication scenarios using Orthogonal Frequency Division Multiple Access (OFDMA) technology in wireless edge systems. The communication process can be broadly divided into the following two communication methods.

[0057] (1) End-to-Edge (D2E) Communication For D2E collaboration, the CU uplink rate of the nth cluster is:

[0058] in, This represents the bandwidth allocated to the nth CU. That is the transmit power of CU n. This represents the channel gain between CU n and BS. It is about The expectation. It is the noise power spectral density. This indicates interference caused by CUs located far from the service area.

[0059] The method provided in this application assumes that the base station is allocated sufficient bandwidth, and therefore ignores downlink latency in the construction of the communication model.

[0060] Based on the transmission rate, the transmission delay between the BS and each CU in the t-th round of communication is derived as:

[0061] in, and These are the sizes of the intermediate data and the encoder model, respectively.

[0062] The energy consumed by the nth CU in transmitting the cascade encoder parameters for the tth communication round is calculated as follows:

[0063] in, This represents the channel allocation parameter of the CU of the nth cluster, i.e., CU n, indicating which channel the CU of the nth cluster is allocated to.

[0064] (2) End-to-end (D2D) communication The goal of D2D collaboration is to identify the selected end nodes. The sequence is used to instantiate encoder segments. In heterogeneous wireless communication systems, end nodes communicate with each other within a cluster using the D2D protocol. For ease of analysis, it is assumed that the D2D communication speed is stable, using... This indicates the process from endpoint k to the next endpoint within the cluster. Data transmission delay can be calculated using the following formula:

[0065] in This represents the size of the intermediate data in the s-th encoder segment during the forward propagation, while It is the ( )th step in the backward propagation process The gradient of the intermediate data of the encoder segment. This represents the transmit power of k computing devices.

[0066] Therefore, the energy consumption of scheduling end node k in the t-th communication round (i.e., the t-th training round, in which the global model is updated based on a batch of training samples in each training round) is expressed by the following formula:

[0067] Considering the differences in computing power among end nodes, each end node has a relative computing capability. The method provided in this application borrows the classic concept of pipelined parallel computing to achieve parallel training between end nodes, such as... Figure 4 As shown, the computation and communication overhead of a microbatch is considered as a forward window and a backward window, so that each microbatch can perform pipelined computation independently without clock overlap.

[0068] The entire training process under the CollaPipe framework includes ( There are ) windows. Therefore, during encoder training in the t-th communication round, the cross-device training time of the scheduling end node K in cluster n can be expressed as:

[0069] in, and These represent the total number of floating-point operations (FLOPs) required for forward and backward propagation of each window in the l-th TEB, respectively; This represents the number of floating-point operations required for training each window in the l-th TEB.

[0070] In equation (15), This represents the unit computation delay of a single end node. This represents the unit delay of D2D communication. Since the communication between the S end nodes and the BS is not included in the pipeline training model, the end node scheduling time of one window can be omitted, i.e., the last term of formula (15). .

[0071] Within a cluster, the end nodes are responsible for performing the forward and backward propagation computations of the adapted encoder segments. Therefore, the energy consumption of the K end nodes within the nth cluster used for encoder training is:

[0072] The upper bound of model divergence is derived to quantify the impact of model splitting and signal transmission interference on LLM training. Based on this, the optimization problem is constructed.

[0073] The derived divergence bound allows us to predetermine the number of encoder segments S and the microbatch size for co-training. Furthermore, D2E collaboration involves long-distance communication, therefore the proposed framework also considers signal interference in the wireless environment. Consequently, the analysis of the model divergence bound focuses on three parts: interference error, encoder segment number S, and cluster size N.

[0074] For ease of analysis, several assumptions are made regarding the loss function, model weights, and gradients, as described below.

[0075] Assumption 1: Loss Function It is non-convex, differentiable, and beta-smooth, that is:

[0076] Assumption 2 (Bounded Gradient): Local Gradient The variance of all entities is a constant. The upper bound, namely:

[0077] Assumption 3: There exists a constant This makes for The Polyak-Lojasiewicz (PL) inequality holds, that is:

[0078] The first three assumptions are typically used in literature on stochastic optimization and joint learning to facilitate convergence analysis in non-convex settings. The following section presents the assumptions regarding pipeline parallelism.

[0079] Assumption 4: In pipelined parallel training, the sum of computation and communication delays between all end nodes tends to be consistent.

[0080] Assumption 4 can be achieved by adjusting the model segmentation strategy (the number of TEBs in each segment). and micro-batch This is achieved by adjusting the size (etc.). This avoids excessive latency in some end nodes causing overall pipeline computation blockage, which would affect the model's convergence speed and performance.

[0081] Assumption 5 (Stationary Backpropagation): The expected value remains unchanged during training.

[0082] Assumption 5 applies to TEB-based model splitting training in CollaPipe, and can optimize the weights of LLM encoder segments while keeping the weights of other segments unchanged. This ensures the statistics of backpropagation. It remains unchanged.

[0083] Lemma 1: Given the objective of formula (1), The convergence behavior is about The results of the encoder and decoder optimizations are as follows:

[0084] In the formula, L / S represents the average number of TEBs per encoder segment.

[0085] Proof: First, extend formula (1) to

[0086] Then, the following derivations were made respectively. and The derivative of, thus obtaining

[0087] Where (a) represents the expression for any vector Given any positive integer n, use the inequality relationship , here set .

[0088] The encoder is further divided into segments, and the parameters of each segment are represented as follows:

[0089] Therefore, we can obtain

[0090] In the formula, (a) the equality holds based on formulas (7) and (9); inequality (b) utilizes inequality relationships. , here set , This represents the average number of TEBs in the encoder segment, and therefore can be expressed by the equation. Remove letters It is worth noting that the proof only assumes that the number of TEBs in each segment is the same. The segmented scheduling module in the CollaPipe framework can allocate different numbers of TEBs based on different device capacities.

[0091] Combining formulas (22), (24), and (21), the expression for the upper bound can be derived:

[0092] Where (a) utilizes inequality relationships , here set .

[0093] Q.E.D.

[0094] Next, a new lemma is proposed to characterize the influence of the communication environment on convergence behavior.

[0095] Lemma 2 (Disturbance Effect): Let... Embed real and corrupted content respectively, and These represent the corresponding real and damaged transmitted signals, respectively, and their relationship can be expressed as:

[0096] In Lemma 2, each CU sends a signal. It is the beamforming vector and embedded symbols The synthesized mean square error of the transmitted signal depends on the transmit power of the transmitter (CU n). The specific measurement method is as follows:

[0097] Wherein, PL and SF represent path loss and shadow fading between CU and BS, respectively. It is the line-of-sight (LOS) path contribution. It is the non-line-of-sight (NLOS) path for each ray z and cluster n, where It refers to the amount of light.

[0098] Based on Lemmas 1 and 2 and Assumptions 1 to 5, the following theorems are derived.

[0099] Theorem 1: Given any bandwidth allocation and end-node-segment matching strategy, when At that time, the optimal upper bound of the error after T rounds of communication is:

[0100] in

[0101] Proof: Before proving, let's review the symbolic representation. This represents the transmission gradient after interference. This represents the updated model parameters after global aggregation. According to Lemma 1, given the decoder parameters, an upper bound on convergence related to the encoder segment is obtained. Therefore, in each communication round t of FL, the global LLM update is as follows:

[0102] in

[0103] Based on the gradient error caused by channel interference in Lemma 2, we can obtain

[0104] To aid the derivation, a virtual model is defined. It assumes that there is no interference in the transmission signal between the mobile device and the BS, meaning that there are almost no errors before and after data transmission:

[0105] use Smoothness can be obtained recursively.

[0106] The convergence upper bound of a complete LLM can be decoupled into two modules: the encoder and the decoder. Next, we analyze the convergence problem of the encoder part under federated split learning. According to Lemma 1, the encoder gradient variance is divided into S segments:

[0107] The overall gradient variance is

[0108] Write formula (35) as Take its form and obtain its expectation:

[0109] Inequality (a) satisfies assumption 5, which is obtained by substituting the PL condition into the second term; inequality (b) is obtained by combining like terms based on formula (38).

[0110] Subtracting equations (34) and (31) yields the difference between the actual parameter update and the virtual queue:

[0111] Based on assumption 1, taking the expected value of the above function, we will... The quadratic term of smoothness is transformed into:

[0112] (a) follows assumption 1 and lemma 2.

[0113] Similarly, (41) can be rewritten as Take its expected value in the form of:

[0114] Combining formulas (39) and (42), we can obtain

[0115] make

[0116] Then from arrive To expand (43), that is

[0117] According to (44), Control the convergence degree of FL in each round. As long as The system can then guarantee convergence. Let The convergence condition of the system can be deduced as follows:

[0118] Q.E.D.

[0119] Note 1: The parallel granularity of the pipeline has a dual impact on FL convergence behavior; increasing the number of encoder segments does not necessarily promote faster convergence.

[0120] The number of model segments reflects the granularity of pipeline parallelism and directly affects training latency and convergence behavior. As shown in Theorem 1, the parameter S appears in the quadratic form in the task-related terms:

[0121] This indicates that in a distributed cluster, an excessively large S amplifies the local model divergence caused by local gradient computation. Furthermore, S indirectly affects the latency per round, thus limiting the number of global updates T within the time budget. Increasing S enhances parallelism and improves end-node resource utilization, but it also increases communication overhead between segments, leading to diminishing returns in the convergence process. These findings suggest that S should not be optimized in isolation. Instead, it should be tuned in conjunction with other system-level parameters, such as the micro-batch size m, to achieve a balance between parallel efficiency and convergence performance.

[0122] Note 2: Communication interference can lead to the accumulation of delays during the convergence process.

[0123] In the interference-related terms, the transmit power of the CU in each cluster. This directly affects signal distortion during the encoder parameter uploading process, with specific error terms manifesting as follows:

[0124] This formula indicates an increase This results in lower transmission errors. This improves the accuracy of joint aggregation and accelerates global convergence. Furthermore, the function... It is about The monotonically decreasing convex function indicates that increasing power under a high-power mechanism... The returns are diminishing. This observation directly guides the design of the power control module in the CollaPipe framework when designing dynamic optimization algorithms. CollaPipe aims to allocate higher power levels to clusters with better channel gain or greater impact on the global model in each round of communication, in order to more accurately transmit local encoder model parameters. Simultaneously, energy constraints... This is to ensure that the improvement in convergence performance does not come at the cost of excessive energy consumption, thereby balancing convergence performance and resource efficiency in wireless networks.

[0125] Based on the above analysis, the latency of each round of communication is divided into two stages: (1) the encoder is trained in each cluster through pipelined parallelism; (2) model parameter transmission under the edge FL framework. Therefore, the total latency of the t-th round of communication used for LLM training is given by the following formula:

[0126] According to Theorem 1, the convergence of CollaPipe is influenced by model segmentation, pipelined parallelism micro-batching, and power control strategies. To obtain a computationally efficient LLM collaborative training framework, a dynamic segmentation scheduling and resource allocation protocol was developed. Guided by Theorem 1, this protocol explicitly combines model splitting, pipelined parallelism, and power control mechanisms to minimize the average end-to-end training latency, while adhering to constraints on energy consumption and network resource usage. Let the scheduling protocol content be... The stochastic optimization problem can be represented as:

[0127] in This represents the memory budget on end node k, while This represents the storage space required for one TEB training session. It is the equilibrium boundary for a single round of training, according to the proof of Theorem 1. Represented as:

[0128] variable The value range is constrained by the value of C1, ensuring that K end nodes in a cluster can train all TEBs; C2 indicates that the number of encoding segments cannot exceed the number of end nodes in the cluster; C3 and C5 indicate that all CUs train LLM collaboratively through FL, and a local encoder will be sent to the server in each communication round. Constrained by C6; C7 stipulates that each deployment segment's end node must guarantee sufficient memory to fine-tune its corresponding segment; C8 and C9 are the energy consumption of the end node and CU in each round of communication, respectively; long-term constraint C10 is adopted to optimize learning performance by balancing the number of segments and the power consumption of the CU.

[0129] Various optimization problem-solving methods can be used to solve the original optimization problem P0. In one possible implementation of the method provided in this application, the original problem P0 is transformed into a time delay minimization problem with system stability constraints using Lyapunov optimization theory. To address its inherent non-convexity, a dynamic optimization algorithm based on problem decoupling and block coordinate descent is proposed.

[0130] Since the optimization problem model optimizes the average time, the P0 problem can be transformed into a virtual stable queue using Lyapunov analysis. The following formula represents the virtual queue for each CU related to the delay and convergence constraints in the long-term state. :

[0131] in It is a virtual delay queue, representing the degree of failure accumulation that meets the convergence and delay requirements. This represents the convergence bound of the t-th round of communication. It is the maximum bound constraint.

[0132] In one possible implementation, to ensure system optimality in the long run, the optimization parameters are solved with the goal of minimizing the maximum training latency among multiple device clusters, including: The maximum value of the total training latency across multiple training rounds for each device cluster is obtained. The goal is to minimize the maximum value of the total training latency, and the parameters to be optimized are then solved.

[0133] In this implementation, stability constraints are added to the P0 problem, and it is rewritten as the following optimization problem:

[0134] To solve P1, we will then characterize the Lyapunov drift penalty function to transform the long-term stochastic problem P1 into a static problem P2 in each round of communication.

[0135] Definition 1: Given The Lyapunov drift penalty function is defined as follows:

[0136] in It is conditional Lyapunov drift. The Lyapunov function is defined.

[0137] Minimizing the drift penalty function helps stabilize the virtual queue. This ensures that the system satisfies the average convergence rate stability constraint C10'. This, in turn, minimizes the FL system delay while also adhering to the long-term segmented scheduling and resource allocation constraints C1~C9. The parameter V is a control factor that balances delay minimization and long-term convergence constraints. Therefore, an adaptive dynamic segmented scheduling and resource allocation algorithm is proposed to minimize the Lyapunov drift penalty function. Using the conclusion of Lemma 1, P1 can be rewritten as:

[0138] To solve for P2, two auxiliary variables are introduced. and The first type This represents the model partitioning and segment scheduling of each cluster executing TEB-level pipelined parallel processes:

[0139] variable This indicates that the model splitting result and segmented scheduling strategy are used in the t-th round of communication. The allocated latency for D2D collaboration within a cluster.

[0140] The second type is the auxiliary variable of D2E co-FL. :

[0141] Therefore, P2 can be rewritten as:

[0142] Using the objective function of P3 and , and The independence between them decomposes the joint optimization problem into the following subproblems.

[0143] Optimal auxiliary variable

[0144] By optimizing the number of segments S, mini-batch m, and the number of TEBs on each end node in end-to-end collaboration. To minimize .

[0145] minimize The first subproblem is represented as:

[0146] Problem Sub-1 involves integer and continuous variables, and both the objective function and constraints are non-convex. Alternating Optimization (AO) is employed to reduce the difficulty of solving the problem. Since S and Since they are strongly coupled and all variables are integers, we can first determine the first set of decision variables: m, and then determine the second set of decision variables: And S.

[0147] First, fix S and Then optimize m, thus transforming the problem into:

[0148] The optimal solution can be obtained by taking the derivative of m.

[0149] Next, fix m and optimize. And S, therefore the problem will be transformed into:

[0150] The optimal solution is obtained by transforming C2 into a linear constraint C2' through integer programming.

[0151] This method can be applied to discrete variables separately. The algorithm is optimized using integer variables S and m. Table 2 summarizes the solution steps for problem Sub-1 in Algorithm 2.

[0152] Table 2

[0153] Optimal auxiliary variable

[0154] The second subproblem of D2E communication is expressed as follows:

[0155] Similarly, the AO method is used to solve problem Sub-2, where the first set of decision variables is The second group is .

[0156] First, we need to solve the channel matching problem. (Fixed) Transform problem Sub-2 into a weighted matching problem:

[0157] In problem (59), due to resource constraints, some CUs are not allocated channels. Therefore, an improved Hungarian algorithm is used to solve this bipartite graph matching problem.

[0158] Next, we will address the power control issue. (Fixed) The objective function involves fractional and logarithmic functions. Using the Dinkelbach transform and Successive Convex Approximation (SCA) method, problem Sub-2 is transformed into:

[0159] A first-order Taylor approximation is used to approximate C8', and the relationship between C8 and C8' is calculated. We construct a convex approximation constraint using the gradient of the function. In the i-th iteration, given the current power... The approximate constraint is

[0160] in

[0161] Similarly, linearization constraint C11, at point At Taylor expansion, then The linear approximation constraint is:

[0162] in, and They are in Taylor's expansion after the location The first and second terms, that is,

[0163] Therefore, convex optimization tools such as CVXPY can be used to solve the problem (60).

[0164] Then, based on the current solution and ,calculate The value of is updated, and the queue in equation (48) is updated. Algorithm 3 in Table 3 summarizes the solution steps for problem Sub-2.

[0165] Table 3

[0166] Given the optimal and In this case, the optimal scheduling protocol can be solved using the coordinate block descent method. Algorithm 4 in Table 4 summarizes the Dynamic Online Algorithm for Segmented Scheduling and Resource Allocation (DSSRA).

[0167] Table 4

[0168] The complexity of the optimization algorithm proposed in this application is analyzed below. In Algorithm 2, optimizing m requires a time complexity of O(n). The closed-form solution. Optimization of S requires constrained integer programming, resulting in a worst-case time complexity of O(n log n). However, when This is acceptable. Therefore, the overall complexity of each iteration of the alternating solution is approximately... In Algorithm 3, the complexity of the Hungarian algorithm used for channel matching is O(n). The SCA method used for power allocation requires calculating the gradient in each iteration, with a complexity of O(n log n). Algorithm 4 calls Algorithms 2 and 3 once per round and updates the virtual queue. Since the complexity of updating the virtual queue is O(n log n), the algorithm is O(n log n). Therefore, the total time complexity of each round is O(n). The overall space complexity is This includes end-node scheduling, channel allocation matrix, virtual queues, temporary gradients, and delays.

[0169] This application proposes an edge-to-edge collaborative training framework based on adaptive splitting of large model encoders and federated aggregation of multiple encoders. Utilizing the ideas of split learning and federated learning, it maximizes the use of computing and communication resources in the edge network, enabling the efficient operation of the large model distributed training system. A pipelined parallel optimization strategy under block-based model splitting is proposed, constructing a pipelined parallel computing model between end nodes, and improving encoder training efficiency by dynamically deciding the micro-batch size and the number of model splits. A distributed hybrid training acceleration optimization model for long-term, multi-iteration training is constructed, proposing a joint optimization problem of model segment scheduling and communication resource allocation with end-to-end and edge-to-end collaborative optimization. A dynamic model segment scheduling and edge-to-edge communication resource allocation (DSSRA) algorithm based on Lyapunov theory is designed to solve for the optimal segment-device matching and channel allocation scheme.

[0170] To verify the effectiveness of the method provided in this application, experimental validation examples are presented, demonstrating large-scale model training on three types of natural language understanding: machine translation, named entity recognition, and sentence classification. For fair comparison, data distribution and training settings are kept consistent across different baselines. The experimental validation examples are described below.

[0171] (1) Model Experiments were conducted using the classic Transformer model and the BERT model, which are widely used in NLP tasks.

[0172] • Transformer: The classic Transformer model consists of two main components: an encoder and a decoder, each composed of six Transformer blocks. The decoder and encoder have similar structures, but their sublayers differ, and the interactions between decoder blocks are more complex.

[0173] • BERT: BERT consists of 12 Transformer coding blocks and is a large language model with an encoder-only architecture.

[0174] (2) Benchmarking To evaluate the performance of CollaPipe, three sets of experiments were conducted: (a) training a classic Transformer model on the Multi30K dataset for machine translation tasks, (b) pre-training a BERT model on a Chinese Named Entity Recognition (NER) dataset, and (c) fine-tuning a BERT model on a 15-class Chinese dataset to evaluate the accuracy of sentence classification tasks. Three baseline methods were used for comparison. • Federated Learning (VanillaFL): A standard FL method that uses a simple weighted average operation to aggregate models without performing any model partitioning or parallel optimization.

[0175] • Pipeline Parallelism: A batch-based pipeline parallelism method that does not perform model splitting optimization or apply FL aggregation.

[0176] • TITANIC: A framework for automatic model splitting that includes end-node selection strategies and uses a P2P communication protocol.

[0177] To further evaluate the effectiveness of the proposed dynamic online scheduling algorithm, it is compared with the following additional scheduling strategies: • Random scheduling: In each round of communication, the BS randomly selects J CUs for communication, and each CU randomly determines the training order of its associated end nodes.

[0178] • Loss-driven scheduling: BS selects JCUs based on the training loss of the local model in each cluster.

[0179] • Delay-driven scheduling: The BS selects J CUs based on the local training latency reported by each cluster.

[0180] (3) Environmental configuration The experiment considers the following default settings: One channel, Each cluster includes [number] clusters. Each segment has several end nodes. To estimate the theoretical computation and scheduling time for each segment, a predefined formula is used to calculate FLOPs, while the Transformers and Fvcore libraries are used to pre-calculate the FLOPs for each segment, thereby reducing hardware-level differences in computation time. Table 5 shows further settings for the CollaPipe and DSSRA algorithms.

[0181] Table 5

[0182] First, the Transformer model is pre-trained and compared with a baseline. The learning performance of different methods is as follows: Figure 5 As shown.

[0183] Figure 5 Figure 'a' in the diagram shows that, under the same batch size and channel conditions, there is no significant difference in training loss across different learning frameworks. However, from... Figure 5 As shown in section b, the proposed CollaPipe framework is better suited for machine translation tasks, achieving a 19.5% improvement in BLEU score compared to the baseline. Pre-training the BERT model on the Named Entity Recognition (NER) task also demonstrates improved learning performance. Figure 5The proposed framework achieves faster reduction of training loss in the early stages of training, demonstrating improved convergence speed and learning efficiency in the initial phase. Based on a pre-trained model, the BERT model was fine-tuned for single-sentence classification tasks. The training process converges much faster, typically within 15 training rounds. Figure 5 As shown in c in the figure. Compared with the resource scheduling strategies in existing joint learning systems, this method does not show a significant advantage in learning performance. However, its performance is consistently about 1.82% and 0.15% higher than PipeLine and TITANIC, respectively. These results indicate that under the condition of independent and identically distributed (IID) data, the proposed learning framework effectively compensates for the training loss caused by encoder partitioning through adaptive end-node scheduling and resource management. Figure 5 As shown in a, c, and e, this method maintains a relatively stable training loss. Furthermore, CollaPipe was observed to achieve more significant performance improvements in machine translation and sentence classification tasks, suggesting that these improvements may stem from differences in block-level or token-level behavior introduced by model partitioning.

[0184] Figure 6 The cumulative computation time during Transformer training is displayed. Among all benchmark methods, CollaPipe consistently maintains the lowest computational latency. Figure 8 Compared to pipelined methods, VanillaFL reduces model training latency by 29.98%. In contrast, other methods introduce additional scheduling latency due to the need for coordination between end nodes. Among all evaluated schemes, CollaPipe achieves the best performance, reducing training latency by 18.94% compared to TITANIC, 15.09% compared to VanillaFL, and 40.55% compared to PipeLine. However, when the encoder segment scheduling module is removed from CollaPipe, latency increases by approximately 22.45%, significantly reducing training efficiency. The light blue left-hand diagonal bar in the figure represents the CollaPipe framework without the segment scheduling module (CollaPipe w / o SS). Compared to the full framework, the cumulative computational latency increases significantly, indicating that the proposed algorithm can effectively reduce computational latency within the cluster. These results demonstrate that CollaPipe's coordinated scheduling mechanism minimizes system latency and improves model training efficiency in federated learning environments.

[0185] While FL does not introduce additional D2D communication latency, it places high demands on the storage capacity of mobile end nodes and the availability of high-quality user data. Table 6 compares the memory usage and data requirements of different methods. By dynamically adjusting the memory usage of each end node based on the number of encoding blocks allocated during model partitioning, the CollaPipe framework offers greater flexibility, making it well-suited for resource-constrained edge environments. Furthermore, since the training dataset is centrally stored in the control unit (CU), participating end nodes only contribute computational resources without needing to share local data. This significantly reduces the number of users providing data and alleviates the end node management overhead in mobile edge networks.

[0186] Table 6

[0187] Figure 8 The comparison between the proposed DSSRA algorithm and baseline schemes is shown. The proposed DSSRA algorithm achieves lower system latency, and its advantages become increasingly apparent with the increase of communication rounds. Specifically, compared with the loss-driven scheduling strategy, DSSRA reduces BERT training latency by 46.44% when the penalty parameter V=0.01. Compared with pure delay-driven scheduling, the latency is reduced by 15.48%, and compared with random scheduling, the latency is reduced by 7.12%. This improvement is attributed to the proposed dynamic optimization algorithm, which comprehensively considers the additional communication and computational overhead caused by frequent interactions at the Transformer block level. By effectively balancing intra-cluster and inter-cluster coordination, the DSSRA algorithm achieves efficient cooperation in heterogeneous networks.

[0188] The following describes the joint optimization system for model decomposition and resource scheduling in edge-to-edge collaboration of the Industrial Internet of Things (IIoT) provided by this invention. The joint optimization system for model decomposition and resource scheduling in edge-to-edge collaboration of the IIoT described below can be referred to and correspond to the joint optimization method for model decomposition and resource scheduling in edge-to-edge collaboration of the IIoT described above. Figure 9 As shown, the industrial IoT edge-device collaborative model decomposition and resource scheduling joint optimization system provided by this invention includes: The task splitting and resource scheduling module 910 is used to obtain training sample batches, divide the training sample batches into m sub-batches, divide the encoder of the model to be trained into S segments, each segment including multiple encoder blocks, and allocate the segments to k computing devices in the device cluster. The forward inference module 920 is used to sequentially send the embedded features of the training samples in the sub-batch to the first computing device among the k computing devices, which are arranged according to the connection order of the encoder in the model to be trained. The transmission module 930 is used to receive the kth intermediate data sent by the kth computing device and send the kth intermediate data to the edge server. The (n+1)th intermediate data is calculated by the (n+1)th computing device based on the nth intermediate data and the segment of the (n+1)th computing device. The receiving module 940 is used to receive the decoder gradient sent by the edge server and send the decoder gradient to the k-th computing device; The parameter update module 950 is used to receive the updated encoder parameters sent by k computing devices respectively, and send the updated encoder parameters to the edge server.

[0189] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communications bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other through the communications bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a joint optimization method for model splitting and resource scheduling in industrial IoT edge-device collaboration. This method includes: acquiring a batch of training samples; dividing the batch of training samples into m sub-batches; dividing the encoder of the model to be trained into S segments, each segment including multiple encoder blocks; allocating the segments to k computing devices in the device cluster; sequentially sending the embedding features of the training samples in the sub-batches to the first computing device among the k computing devices, with the k computing devices arranged according to the connection order of the encoder in the model to be trained; receiving the kth intermediate data sent by the kth computing device and sending the kth intermediate data to the edge server, wherein the (n+1)th intermediate data is calculated by the (n+1)th computing device based on the nth intermediate data and the segments of the (n+1)th computing device; receiving the decoder gradient sent by the edge server and sending the decoder gradient to the kth computing device; and receiving the updated encoder parameters sent by the k computing devices respectively and sending the updated encoder parameters to the edge server.

[0190] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, an edge server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0191] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the joint optimization method for model splitting and resource scheduling for edge-to-edge collaboration in the industrial Internet of Things provided by the above methods. The joint optimization method for model splitting and resource scheduling for edge-to-edge collaboration in the industrial Internet of Things includes: obtaining a batch of training samples, dividing the batch of training samples into m sub-batches, dividing the encoder of the model to be trained into S segments, each segment including multiple encoder blocks, and allocating the segments to k computing devices in the device cluster. The embedded features of the training samples in the sub-batch are sequentially sent to the first of the k computing devices, which are arranged according to the connection order of the encoder in the model to be trained. The kth intermediate data sent by the kth computing device is received and sent to the edge server. The (n+1)th intermediate data is calculated by the (n+1)th computing device based on the nth intermediate data and the segment of the (n+1)th computing device. The decoder gradient sent by the edge server is received and sent to the kth computing device. The updated encoder parameters sent by the k computing devices are received and sent to the edge server.

[0192] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the joint optimization method for model decomposition and resource scheduling for edge-to-edge collaboration in the Industrial Internet of Things (IIoT) provided by the above methods. This joint optimization method includes: acquiring a batch of training samples; dividing the batch of training samples into m sub-batches; dividing the encoder of the model to be trained into S segments, each segment including multiple encoder blocks; allocating the segments to k computing devices in a device cluster; and sequentially embedding the training samples in the sub-batches into... The input features are sent to the first of the k computing devices, which are arranged according to the connection order of the encoder in the model to be trained; the kth intermediate data sent by the kth computing device is received and sent to the edge server, where the (n+1)th intermediate data is calculated by the (n+1)th computing device based on the nth intermediate data and the segment of the (n+1)th computing device; the decoder gradient sent by the edge server is received and sent to the kth computing device; the updated encoder parameters sent by the k computing devices are received and sent to the edge server.

[0193] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0194] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, an edge server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of the invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A joint optimization method for model decomposition and resource scheduling in edge-device collaboration of the Industrial Internet of Things, characterized in that, include: Obtain training sample batches, divide the training sample batches into m sub-batches, divide the encoder of the model to be trained into S segments, each segment including multiple encoder blocks, and allocate the segments to k computing devices in the device cluster. The embedded features of the training samples in the sub-batch are sequentially sent to the first of the k computing devices, which are arranged according to the connection order of the encoder in the model to be trained. Receive the kth intermediate data sent by the kth computing device, and send the kth intermediate data to the edge server, wherein the (n+1)th intermediate data is calculated by the (n+1)th computing device based on the nth intermediate data and the segment of the (n+1)th computing device; Receive the decoder gradient sent by the edge server and send the decoder gradient to the k-th computing device; The updated encoder parameters are received from the k computing devices respectively, and then sent to the edge server.

2. The method for joint optimization of model decomposition and resource scheduling for edge-to-edge collaboration in the Industrial Internet of Things according to claim 1, characterized in that, Before dividing the training sample batch into m sub-batches, the following steps are included: Obtain the values ​​of the parameters to be optimized, which include the number of sub-batches m, the number of segments S, and the encoders in each of the k computing devices; The value of the parameter to be optimized is obtained by optimizing the parameter to minimize the maximum training latency of each of the device clusters. The parameter to be optimized includes the number of sub-batches m, the number of segments S, and the number of encoder blocks in each of the k computing devices. .

3. The joint optimization method for model decomposition and resource scheduling in industrial IoT edge-end collaboration according to claim 2, characterized in that, The training latency includes a first training latency and a second training latency. The first training latency includes the computation latency of devices within the device cluster and the transmission latency between devices. The second training latency includes the transmission latency between the device cluster and the edge server.

4. The joint optimization method for model decomposition and resource scheduling in industrial IoT edge-end collaboration according to claim 3, characterized in that, The parameters to be optimized also include the transmission bandwidth between the edge server and the edge server, as well as the transmission channel configuration between the edge server and the edge server.

5. The joint optimization method for model decomposition and resource scheduling in industrial IoT edge-end collaboration according to claim 2, characterized in that, The optimization of the parameters to be optimized, with the goal of minimizing the maximum value among the training latency of the multiple device clusters, includes: The maximum value of the total training latency for each of the device clusters across multiple training rounds is obtained. The optimization of the parameters to be optimized is then performed with the goal of minimizing the maximum value of the total training latency.

6. The joint optimization method for model decomposition and resource scheduling in industrial IoT edge-end collaboration according to claim 2, characterized in that, The optimization of the parameters to be optimized, with the objective of minimizing the maximum training latency among the multiple device clusters, includes: Using Lyapunov optimization theory, the optimization problem with the objective of minimizing the maximum training latency of each device cluster is transformed into a virtual stable queue for solution.

7. A joint optimization system for model decomposition and resource scheduling in industrial IoT edge-device collaboration, characterized in that, The system includes: The task splitting and resource scheduling module is used to obtain training sample batches, divide the training sample batches into m sub-batches, divide the encoder of the model to be trained into S segments, each segment including multiple encoder blocks, and allocate the segments to k computing devices in the device cluster. The number of encoder blocks allocated to each of the k computing devices is [number missing]. ; The forward inference module is used to sequentially send the embedded features of the training samples in the sub-batch to the first computing device among the k computing devices, and the k computing devices are arranged according to the connection order of the encoder in the model to be trained; The transmission module is used to receive the kth intermediate data sent by the kth computing device and send the kth intermediate data to the edge server, wherein the (n+1)th intermediate data is calculated by the (n+1)th computing device based on the nth intermediate data and the segment of the (n+1)th computing device; A receiving module is used to receive the decoder gradient sent by the edge server and send the decoder gradient to the k-th computing device; The parameter update module is used to receive the updated encoder parameters sent by the k computing devices respectively, and send the updated encoder parameters to the edge server.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the joint optimization method for model splitting and resource scheduling for edge collaboration in the industrial Internet of Things as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the joint optimization method for model decomposition and resource scheduling for edge collaboration in the industrial Internet of Things as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the joint optimization method for model decomposition and resource scheduling for edge collaboration in the industrial Internet of Things as described in any one of claims 1 to 6.

Images