Heterogeneous split learning co-optimization method and system based on feature alignment adapter

Abstract

The application discloses a heterogeneous split learning collaborative optimization method and system based on a feature alignment adapter, and belongs to the technical field of edge computing and distributed machine learning. Based on an edge split learning framework, each client constructs a personalized local bottom model according to local computing resources and data distribution characteristics, and collaboratively trains the local bottom model with a top model shared by a server. The method introduces a feature adapter module in the model split layer to align heterogeneous intermediate feature representations from different clients, thereby eliminating training obstacles caused by inconsistent model architectures and output dimensions. The application further includes a construction initialization mechanism to accelerate the convergence of split training, including initialization of the adapter and a heterogeneous client model initialization strategy based on feature distillation, and further based on a global mini-batch stochastic gradient descent algorithm, jointly optimizes the server model, the adapter and the heterogeneous client model, effectively improving the accuracy and training efficiency of the split model in a device resource and data heterogeneous environment.

Description

Technical Field

[0001] This invention belongs to the field of big data analysis, including applications in industrial big data analysis, specifically involving the fields of edge computing and distributed machine learning technology, and particularly relating to a heterogeneous split learning collaborative optimization method based on feature alignment adapter under the split learning framework. Background Technology

[0002] Intelligent edge devices are increasingly relying on deep neural networks to perform complex inference tasks. However, the hardware limitations of most mobile edge devices, such as CPU and memory, cannot meet the requirements for deploying large-scale, state-of-the-art models for training and inference. Therefore, they can only train smaller, lighter models, and methods based on lightweight models (model compression / quantization) significantly degrade model performance. To address this, split learning offers a more flexible solution. As an emerging distributed learning paradigm, split learning differs from traditional centralized cloud training and federated learning. It splits the deep neural network between the client and the cloud server. The client only performs forward computation up to the split layer and transmits the intermediate feature representations to the server for subsequent computation and inference. This training model significantly reduces the computational load on edge devices, making it suitable for more resource-constrained devices; it also avoids directly transmitting raw data and achieves privacy protection through feature compression. Therefore, split learning has significant value in edge intelligence tasks that are sensitive to computational efficiency and privacy, such as real-time perception for autonomous driving, remote sensing analysis of drones, and wearable medical monitoring. However, traditional split learning methods typically assume that all clients use a uniform model architecture and parameters, which has serious limitations in real-world edge scenarios. Due to the highly heterogeneous hardware capabilities (such as CPU / GPU computing power and memory capacity) and data distribution of edge devices, decomposition learning faces the following specific challenges: 1) The computing capabilities of edge computing devices vary significantly. For example, some devices are low-power IoT sensors that can only support small neural networks, while other high-performance devices are capable of running larger deep learning models. A unified client model design often needs to be adapted to the devices with the weakest computing capabilities, resulting in idle computing power on high-performance devices and limited global training efficiency.

[0003] 2) Data collected by edge devices typically exhibits non-independent and identically distributed (non-IID) characteristics, with significant differences in data categories and feature distributions across different devices. Since existing split-learning methods employ a globally uniform client model, this approach struggles to generalize effectively in heterogeneous data environments, potentially leading to decreased training performance on some clients and ultimately impacting the convergence and performance of the global model.

[0004] To address the aforementioned issues, each client needs to use a personalized heterogeneous model. The model size and architecture can be adjusted based on the client's hardware resources to better adapt to system heterogeneity and optimize for each client's local data distribution. However, the training process for heterogeneous split learning presents several challenges, primarily in the following aspects: a. Traditional split-learning methods typically rely on client-side model aggregation to ensure stable convergence of the global model. However, in heterogeneous split-learning scenarios, the different architectures, parameter shapes, and feature spaces of each client-side model render standard model aggregation strategies unsuitable for direct application. In the absence of a unified model aggregation mechanism, ensuring consistent optimization directions for heterogeneous models during joint training and guaranteeing stable convergence of the server-side model becomes a pressing issue.

[0005] b. In a heterogeneous split learning framework, each client selects a model of different scales based on its own computing resources. This leads to inconsistent output dimensions of the client models, and the intermediate feature representations from different clients may not be directly and uniformly processed by the server-side model, thus hindering training. Therefore, how to achieve efficient adaptation between the intermediate feature representations output by heterogeneous client models and the server-side model without reducing feature representation capabilities becomes a key challenge.

[0006] c. Classical split training methods typically maintain an independent copy of the server model for each client and update it separately in each training round. However, in highly heterogeneous data environments, the gradient update directions of different clients may deviate significantly. As the number of training rounds increases, this offset accumulates, causing the aggregated global model to deviate from the optimal solution, resulting in a significant decrease in convergence efficiency and severely impacting model performance. Summary of the Invention

[0007] Purpose of the invention: To address the problems of poor adaptability to heterogeneous devices, difficulty in feature representation alignment, and unstable convergence in existing split learning techniques, this invention provides a heterogeneous split learning collaborative optimization method based on a feature alignment adapter.

[0008] Technical solution: A heterogeneous split learning collaborative optimization method based on feature alignment adapter. In the method, each client first constructs a personalized local underlying model based on local computing resources and data distribution characteristics, and then performs collaborative training with the top-level model shared by the server. Then, by introducing a feature adapter module in the model splitting layer to align the heterogeneous intermediate feature representations from different clients, the training obstacles caused by the inconsistency of model architecture and output dimension are eliminated. The proposed method is based on a split learning framework and is designed for collaborative training scenarios involving multiple heterogeneous edge devices. By introducing a feature alignment adapter module and a global mini-batch stochastic gradient descent algorithm, it achieves efficient alignment of features from heterogeneous clients and alleviates gradient drift problems under heterogeneous data distribution, thereby improving the global training convergence efficiency and stability.

[0009] The method includes the following steps: (1) The server first sends the complete deep neural network model from the first... The layer is split into two parts, with the input layer up to the first layer. Layers constitute the client model The encoder extracts basic features; the remaining layers constitute the server model. It acts as a decoder to perform deep forward propagation and compute the final task prediction results; in For client model parameters, These are server model parameters.

[0010] (2) The server divides the client into groups based on the client's resource constraints. Cluster And model pruning techniques are used to modify the client model. Uniform reduction is performed in both the width (i.e., the number of convolutional kernels per layer) and depth (the number of layers in the model) directions, thereby reducing the number of kernels per cluster. Generate customized client heterogeneous sub-models , m This indicates the number of clusters, which also corresponds to the number of clients.

[0011] Specifically, for resource-constrained low-power device clusters, highly lightweight models are generated to suit their limited computing power; for relatively resource-rich device clusters, relatively light pruning is performed to generate more complex models to fully utilize their computing capabilities. During the pruning process, it is ensured that the output dimension of all models does not exceed their input dimension, while allowing differences in output dimensions between heterogeneous sub-models. Finally, all generated sub-models are stored in a heterogeneous client model pool. middle; (3) For heterogeneous client model pools All possible output feature dimensions set The server is for each dimension Design and maintain an adapter Its function is to divide the dimension into Feature mapping to the input dimensions required by the server-side model During the adapter initialization process, for each feature dimension... The design introduces a encoder and decoder A lightweight autoencoder module is composed of features from dimensions. Compress to The decoder then converts features from dimensions Restore to dimension .

[0012] (4) Pre-train the original deep neural network model on the public dataset on the server side. After pre-training is completed, the split layer is inserted in step (1). A self-encoder Each autoencoder corresponds to a heterogeneous client model pool. One of the feature dimensions During adapter initialization training, freeze... The parameters are updated only for the learnable parameters of the intermediate autoencoder module. The specific training process is as follows: (41) For input data in the public dataset First, it goes through the client-side model. Generate the corresponding feature representation Then The input is fed into various autoencoders for compression and restoration: Ultimately through the server model Calculate the prediction results: .

[0013] (42) During backpropagation, only the autoencoder parameters are optimized. and ,and parameters and The process remains unchanged, and the above process is iterated repeatedly until each autoencoder model converges.

[0014] (43) Save the decoder portion of these trained autoencoders as an initialized feature alignment adapter for subsequent heterogeneous split learning joint training. That is... .

[0015] (5) Heterogeneous client model pool The model in the model also needs to be initialized before being deployed to a specific client. This step reuses the encoder part of the autoencoder trained during the adapter initialization phase. , The following initialization scheme based on feature distillation was designed: (51) The pre-trained client model from step (4) With encoder Combining the teacher model and the heterogeneous client model As a student model. Since multiple client clusters may share the same output dimension. Therefore, they are assigned to the same teacher model. Client collection defined as ,in For client sub-model The output dimension. Let the public dataset be... ,in Let N be the i-th sample, and N be the dataset size. For each feature dimension... Teacher Model and student model The definition is as follows:

[0016] (52) During the distillation process, each client model learns from the shared teacher model. Learning knowledge in the middle, freezing the parameters of the teacher model Update the parameters of the student model Knowledge distillation is achieved by minimizing the mean squared error (MSE) between feature representations. The distillation loss is expressed as follows:

[0017] This initialization process applies to all Execute in parallel.

[0018] (53) After initialization, each model Distribute to cluster The client in the example is taken as its initial client model, where the model assigned to client k is denoted as... .

[0019] (6) Based on the pre-initialized client model and adapter, perform heterogeneous split learning training. The following describes the steps performed in the t-th training iteration.

[0020] (61) Each client From local dataset Independently sample a small batch of data and uses its pre-trained model to generate intermediate feature representations. ,in The client represents the features it computes. Uploaded to the server.

[0021] (62) For those sharing the same feature dimensions client collection Server-side adapter allocation It represents all features Mapping aligned to a uniform dimension To adapt to the server model The input requirements are as follows: the aligned feature representation is... The projected feature representations are concatenated along the batch dimension to obtain... ,in .

[0022] (63) Aggregate the outputs of all adapters into a global data batch. And input the server model By reasoning, the prediction result is obtained: .

[0023] (64) During the backpropagation phase, the server first calculates the loss for each client k:

[0024] in Indicates client k The model parameters involved in performing full inference, where y is the true label of the training sample x. The loss function is defined as follows (e.g., cross-entropy loss or mean squared error loss). The global loss is a weighted sum of the losses for all clients: , in This represents the global batch size processed by the server. Based on the calculated global loss, the server updates the parameters using gradient descent, with a learning rate of... The process is represented as follows:

[0025] in, This represents the proportion of client k in the current global batch. For the adapter also deployed on the server side, the update process is as follows:

[0026] in This indicates that the number of samples from client k in the current global batch represents a percentage of the total number of clients in the client set. The proportion of all clients in the current global batch of samples.

[0027] (65) On the client side, each client first receives the gradient returned by the server:

[0028] Then calculate the gradient of the local model:

[0029] Subsequently, each client updates its local model parameters using the following formula:

[0030] After completing steps (61)-(65) above, a global mini-batch iteration is performed. t This process continues until the global model converges.

[0031] This invention also provides a heterogeneous splitting learning collaborative optimization system based on a feature alignment adapter. The system includes a storage component and a processing component. The storage component stores one or more computer instructions. The one or more computer instructions are invoked and executed by the processing component to implement the heterogeneous splitting learning collaborative optimization method based on the feature alignment adapter. Based on this method, local models and global models deployed on clients and servers are trained to enable collaborative processing of computational tasks, including image classification and text analysis, by intelligent edge devices.

[0032] Furthermore, the system constructs a personalized local underlying model based on local computing resources and data distribution characteristics, and performs collaborative training with the top-level model shared on the server side, specifically including: The client executes the client model. As the encoder extracts basic features, it also includes a split learning framework to achieve collaborative training of multiple clients. By introducing a feature alignment adapter module and a global mini-batch stochastic gradient descent algorithm, it achieves the alignment of features from heterogeneous clients. Server, execution server model It acts as a decoder to perform deep forward propagation and compute the final task prediction results.

[0033] The present invention also provides a computer storage medium storing a computer program, which, when executed by a computer, implements the above-described heterogeneous splitting learning collaborative optimization method based on feature alignment adapter.

[0034] Beneficial effects: Compared with the prior art, the substantial progress and significant effects of the present invention include: 1) This invention breaks through the limitation of traditional split learning on the uniformity of client model architecture, allowing each client to build a personalized model according to its own computing resources and data characteristics, thereby improving device utilization and enhancing the system's adaptability and flexibility; 2) The method described in this invention introduces a feature alignment adapter module in the split layer, which enables heterogeneous intermediate features generated by different clients to be aligned and adapted to the unified decoder on the server side, eliminating training obstacles caused by inconsistent model architecture and output dimensions, and realizing efficient collaboration between heterogeneous devices.

[0035] 3) In order to overcome the optimization instability caused by heterogeneous data distribution, this invention proposes to abandon the traditional parallel scheme of maintaining an independent copy of the server model for each client. Instead, it uses a unified server model to aggregate data batches from all clients for training. This enables the server-side model to optimize based on the data distribution of heterogeneous clients, reducing the deviation of update directions of different clients, thereby improving the convergence stability and optimization effect of the global model.

[0036] 4) The method described in this invention addresses the problem of difficulty in convergence during collaborative training of heterogeneous models by designing a pre-training initialization mechanism for the adapter and client models. This enables heterogeneous models to obtain better feature representation capabilities in the early stages of training, thereby accelerating convergence and improving the generalization performance of split learning in scenarios with heterogeneous system resources and data with extremely low additional training overhead. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the workflow of the split training framework constructed in the method described in this invention; Figure 2 The figure shown is a personalized accuracy distribution diagram of 10 clients under different resource device ratio settings in the embodiment; Figure 3 This is a comparison of the convergence performance of each algorithm module in the heterogeneous split training on a non-IID dataset, as shown in the example. Detailed Implementation

[0038] To illustrate the technical solutions disclosed in this invention in detail, further explanation will be provided below with reference to the accompanying drawings and specific embodiments.

[0039] To address the problems of traditional distributed training relying on a single global model, which struggles to adapt to differences in device computing power and non-independent, identically distributed data, leading to wasted computing power on high-performance devices, limited training resources on low-performance devices, and decreased generalization ability of the global model, this invention provides a heterogeneous split-learning collaborative optimization method based on a feature alignment adapter. This method introduces a feature adapter module into the model splitting layer to align heterogeneous intermediate feature representations from different clients, thereby eliminating training obstacles caused by inconsistencies in model architecture and output dimensions. Furthermore, this invention proposes an efficient initialization mechanism, including a pre-training strategy for the feature adapter and a heterogeneous client model initialization scheme based on feature distillation, to accelerate the convergence process of split training. In addition, to address the optimization challenges posed by the heterogeneity of data and computing resources, this invention designs a global mini-batch stochastic gradient descent optimization strategy, enabling the server to jointly optimize the adapter module, the global model, and the heterogeneous client models, improving training stability and the convergence efficiency of the global model.

[0040] In the method described in this invention, a distributed collaborative training approach for multiple heterogeneous edge devices is implemented based on a split learning framework. Unlike existing split learning techniques and other distributed collaborative training methods, this method proposes a training mechanism that flexibly adapts to different model architectures, addressing the differences in computing power and data distribution among heterogeneous devices. This ensures that each client can efficiently collaborate with the server while meeting its own resource constraints. To this end, this invention introduces an adapter module for feature space transformation at the model splitting layer, enabling intermediate features from different clients to be mapped to a unified feature space while maintaining their representational capabilities, thus eliminating the impact of heterogeneous model structures on the training process. Furthermore, this invention optimizes the server-side gradient aggregation strategy to reduce training bias caused by non-independent identically distributed (non-IID) data, thereby improving the convergence stability and final performance of the global model.

[0041] The method described in this invention was initially based on splitting the federated learning framework to parallelize client updates and improve time efficiency. It was then extended to simultaneously consider system resource heterogeneity and data heterogeneity to make corresponding adjustments to the model training structure and strategy.

[0042] Based on the aforementioned technical solution and existing technologies well-known to those skilled in the art, further, combined with Figure 1 The implementation process of the method described in this invention is explained in detail below.

[0043] Step 1: Construct a problem model for heterogeneous decomposition learning and build a heterogeneous decomposition learning system.

[0044] This step involves mathematically modeling and defining the problem for split training scenarios involving collaboration between multiple heterogeneous edge clients and a single server, and then constructing a system model.

[0045] Assume the system has a total of Each client and a central server collaboratively train a deep learning model. Each client... Holding a private local dataset Its data size is The client weight coefficient is defined as follows: This reflects the proportion of its data volume contributing to the overall training.

[0046] In traditional split federated learning, all clients are required to use the same client model. With server-side model Co-parallel training aims to minimize the globally weighted loss function.

[0047] in,

[0048] However, this isomorphic model assumption leads to two core problems: i) Clients with weaker computing power are slower in processing forward and backward propagation, resulting in excessively high synchronization latency during collaborative training, which becomes a bottleneck in training, while clients with abundant resources cannot fully utilize their computing potential. ii) Client data distribution In highly heterogeneous environments, a single model struggles to adapt to all client features, leading to a decline in model generalization performance.

[0049] To solve the above problem, let each client Use a customized model Its parameters Model architecture, such as the number of layers and the number of convolutional kernels per layer, can be designed on demand. Existing work on model heterogeneity typically forces all client-side models to have the same output dimension in order to match server-side models. However, this requirement for a uniform output dimension presents significant practical challenges: an excessively low uniform dimension inevitably discards valuable information, thus reducing feature quality, while an excessively high uniform dimension increases the computational burden and communication overhead on resource-constrained clients. Therefore, this modeling approach abandons the constraint of a uniform output dimension and proposes a dynamic dimension adaptation mechanism: client model... Output feature dimension The server-side model can be flexibly configured based on its local data characteristics and computing resources, avoiding redundant dimension filling or truncation. Then through a set of adapters Align the different output dimensions of the client to the input dimensions of the server model. Define the global training objective for heterogeneous split learning as:

[0050] Among them, the local loss function Defined as:

[0051] Here, the global objective function optimizes the client model parameters jointly. Adapter parameters and server model parameters This design enables efficient collaborative training of heterogeneous clients. It resolves the conflict between forced dimension alignment and resource heterogeneity in traditional methods, providing a scalable system foundation for collaborative training of multiple types of edge devices.

[0052] Step 2: Build a heterogeneous client model pool to realize the partitioning and resource adaptation of client models.

[0053] This step aims to generate sub-models adapted to different resource conditions from a larger unified client model using model pruning techniques, based on the client's resource constraints and computing power. This achieves client model personalization and efficient resource utilization. Specific implementation details are as follows: Suppose the server holds a complete deep neural network model with L layers, and it is transferred from the Lth layer... The layer is initially split, and the input layer is pushed to the next layer. The layer portion serves as the client model. Its function is to extract basic features as an encoder; the remainder is derived from the first... The portion from layer L to layer L is used as the server-side model. Its function is to act as a decoder to perform advanced semantic information extraction and computational prediction tasks. This partitioning ensures that the client only needs to perform lightweight forward computation, offloading most of the computational burden of the complete model to the server.

[0054] Then, based on the client's resource constraints (such as computing power, memory size, etc.), the server divides the clients into M clusters, denoted as Mc. Each cluster contains clients with similar resource conditions. For each client cluster... The server uses model pruning techniques to modify the original client model. Perform balanced pruning to generate sub-models adapted to the cluster's resource conditions. Pruning techniques involve adjusting the model's depth (e.g., reducing the number of layers) and width (e.g., reducing the number of channels or convolutional kernels per layer) to ensure efficient model operation on resource-constrained devices.

[0055] Through the above pruning and optimization process, the server generates a set of heterogeneous client models. And store it in a heterogeneous client model pool. In the middle. Each sub-model Output dimension The configuration can be flexibly adjusted based on available resources and task requirements, and it is not mandatory for all sub-models to have the same output dimension. Furthermore, the server needs to ensure that the output dimension of each sub-model does not exceed its input dimension to reduce the communication overhead of split learning.

[0056] Step 3: Implement adapter initialization.

[0057] This step proposes an adapter initialization scheme based on a lightweight autoencoder, aiming to improve the convergence efficiency of heterogeneous split learning through pre-trained adapters. The adapter initialization process consists of two stages: autoencoder insertion and training, and adapter extraction and deployment.

[0058] In heterogeneous split learning frameworks, the model architectures of different clients may lead to inconsistent intermediate feature dimensions, thus requiring the construction of adapters to align these feature representations. To this end, a heterogeneous client model pool is defined. All possible output feature dimensions set ,in This represents the r-th (out of R) possible client model output feature dimension. During the initialization phase, the server first splits the feature dimension at the split layer. Insert a lightweight self-encoder module, which consists of an encoder. and decoder Composition. Encoder Client model Output features from Dimensional compression to Dimension, and decoder Take it from Dimensional expansion back Dimension, that is:

[0059] Specifically, the server first pre-trains the original model on a public dataset. Alternatively, you can directly load publicly available pre-trained weights. Then, for each feature dimension... ,exist and Insert the corresponding autoencoder between and freeze and The parameters are used to train the autoencoder. For public datasets, where For the input sample, For the corresponding ground truth labels, the server performs the following forward propagation calculations during training:

[0060] The goal of training is to minimize the predicted output. With real labels Losses between:

[0061] loss function Cross-entropy loss or mean squared error loss can be used, depending on the task type. During backpropagation, only the autoencoder parameters are updated. ,and and The parameters remain unchanged. The above training process consists of R autoencoders. The process is executed in parallel, iterating and optimizing multiple times until each autoencoder gradually converges. After training, the decoder portion of each autoencoder is retained and used as an initialized feature alignment adapter. These adapters will be used for feature alignment during subsequent heterogeneous split-learning joint training to ensure that intermediate features from different client models can be effectively mapped to a unified feature space.

[0062] Because the autoencoder structure is lightweight, it introduces only a small number of trainable parameters; and it freezes... and The server can then avoid duplicate calculations. The output features only need to be calculated once and then directly copied. Parallel input of all self-encoders Training is performed automatically, thus the adapter initialization training time and computational overhead proposed in this invention are low. Through this scheme, the system can quickly generate initial adapters adapted to different feature dimensions, laying the foundation for rapid convergence of subsequent heterogeneous splitting learning.

[0063] Step 4: Initialization of Heterogeneous Client Model Based on Feature Distillation

[0064] This step aims to provide initialization for heterogeneous client models through knowledge distillation, ensuring compatibility between the client models and pre-trained adapters. During this process, the encoder portions of the R trained autoencoders obtained in step 3 are reused. Furthermore, an initialization scheme based on feature distillation was designed, wherein the pre-trained client model obtained in step 3... With encoder Together, they serve as teacher models, providing knowledge transfer for heterogeneous client models.

[0065] Since multiple client models may share the same output feature dimensions Therefore, for a specific The allocated client set is defined as follows ,in For client model The output feature dimension. Using a public dataset. Perform distillation training for each feature dimension. Define the teacher model and student model for:

[0066] During the distillation process, each client model learns from the shared teacher model. Learning takes place during this process. Specifically, the teacher model parameters are first frozen. The student model parameters are updated by minimizing the mean squared error (MSE) between the intermediate feature representations output by the teacher model and the student model. :

[0067] This initialization process applies to all And all heterogeneous client models Parallel execution. After distillation training is complete, each trained heterogeneous client model... Distribute to the corresponding client cluster All clients in the model are used as their respective initial models, where the client is assigned a model. The model is denoted as Note that this process is shared by all teacher models. The pre-extracted feature can be reused during the distillation process. Further improve computational efficiency.

[0068] Step 5: Heterogeneous Splitting Learning and Training Based on Global Mini-Batch SGD

[0069] This step proposes a distributed mini-batch stochastic gradient descent (SGD) framework based on a global dataset. By uniformly optimizing the parameters of the client model, adapter, and server model, it solves the parameter drift problem caused by the non-independent and identically distributed (Non-IID) client data in traditional split learning, thereby improving the efficiency and convergence of heterogeneous collaborative training.

[0070] The following describes in detail how the client and server collaborate to complete training in a single iteration within the heterogeneous split learning framework: In the t-th iteration, each client First, each independently retrieves its own local dataset. Randomly sample a small batch of data The intermediate feature representation of the mini-batch data is then calculated using the pre-trained client model distributed in step 4.

[0071] in, Then, the client uses the features it has calculated. Send to the server.

[0072] For each sharing the same intermediate feature dimension client collection The server assigns a unique corresponding adapter to each of them. This adapter will connect the client collection Features of all client uploads All are mapped to a unified dimension. To adapt to the server model The input requirements, the process is represented as , .

[0073] Mapped feature representation Global batch intermediate feature representation is formed by concatenating along the batch dimension. :

[0074] in .

[0075] Server-side model take over Then, the forward propagation continues, yielding the inference result:

[0076] Server based on Calculate global loss For client k in small batches Its loss function is expressed as:

[0077] The global loss is obtained by weighted summation of the losses of all clients:

[0078] in This indicates the global batch size processed by the server. The server then performs the following gradient descent update:

[0079] in, , This represents the proportion of client k in the current global batch.

[0080] For adapters deployed on the server side, the update process is as follows:

[0081] in, , This indicates that the number of samples from client k in the current global batch represents a percentage of the total number of clients in the client set. The proportion of all clients in the current global batch of samples.

[0082] Then, on the client side, each client first receives the gradient returned by the server:

[0083] Then each client is based on Next, perform backpropagation to calculate the gradient of the local model:

[0084] The process by which each client ultimately updates its local model parameters is represented as follows:

[0085] Once the above steps are completed, one iteration of global mini-batch SGD is finished. This process is repeated in consecutive training rounds until the global model converges to the optimal solution or meets the preset stopping condition.

[0086] Compared to traditional split federated learning, the global mini-batch SGD strategy described in this invention trains all clients based on a unified server model, ensuring that the server-side update direction takes into account the heterogeneous data of all clients, thereby converging to a better solution and improving convergence stability and optimization efficiency in heterogeneous environments.

[0087] like Figure 2 As shown, we consider training 10 clients with different proportions of strong, medium, and weak devices through heterogeneous splitting. It can be seen that compared with training a compressed lightweight model locally on each client, the heterogeneous splitting collaborative training scheme proposed in this invention achieves higher overall personalized model accuracy. At the same time, the model accuracy difference (variance) between each client is also significantly reduced. This shows that the heterogeneous splitting training framework described in this invention has good robustness to heterogeneous resource and data scenarios.

[0088] Combination Figure 3 As shown, the convergence performance of different heterogeneous split-learning variants on non-IID datasets is demonstrated to verify the contribution of each module to the overall performance of heterogeneous split-learning. It can be seen that the HeteroSL approach (i.e., complete split-learning) outperforms the HeteroSL w / o GMS approach (split-learning without global mini-batch SGD) and the HeteroSL w / o init approach (split-learning without initializing the adapter module and the client heterogeneous model). Removing global mini-batch SGD and encoder initialization significantly reduces the model's convergence accuracy, indicating that these two modules play a crucial and positive role.

Claims

1. A heterogeneous splitting learning collaborative optimization method based on feature alignment adapter, characterized in that, In the method, each client first constructs a personalized local underlying model based on local computing resources and data distribution characteristics, and then performs collaborative training with the top-level model shared by the server. Then, by introducing a feature adapter module in the model splitting layer to align the heterogeneous intermediate feature representations from different clients, the training obstacles caused by the inconsistency of model architecture and output dimension are eliminated. The implementation steps of the method include: (1) The server first sends the complete deep neural network model from the first... The layer is split into two parts, with the input layer up to the first layer. Layers constitute the client model The encoder extracts basic features; the remaining layers constitute the server model. It acts as a decoder to perform deep forward propagation and compute the final task prediction results; (2) The server divides the client into groups based on the client's resource constraints. Cluster And model pruning techniques are used to modify the client model. Perform uniform reduction in both width and depth directions to achieve this for each cluster. Generate customized client heterogeneous sub-models ; During the pruning process, it is ensured that the output dimension of all models does not exceed their input dimension, while allowing differences in output dimensions between heterogeneous sub-models. Finally, all generated sub-models are stored in a heterogeneous client model pool. middle; (3) For the set of all possible output feature dimensions in the heterogeneous client model pool The server is any dimension within it. Design and maintain an adapter Its function is to divide the dimension into Feature mapping to the input dimensions required by the server-side model ; During the adapter initialization process, for each dimension By encoder and decoder The lightweight autoencoder that is composed of features from dimensionality Compress to The decoder then converts features from dimensions Restore to dimension ; (4) Pre-train the original deep neural network model on the public dataset on the server side. After pre-training is completed, the split layer is inserted in step (1). A self-encoder Each autoencoder corresponds to a heterogeneous client model pool. One of the feature dimensions During the adapter initialization training process, the deep neural network model is frozen. The parameters are updated only for the learnable parameters of the intermediate autoencoder module. ; (5) Construct a heterogeneous client model initialization strategy based on feature distillation for the heterogeneous client model pool. The model is initialized using the adapter. The encoder part of the autoencoder trained during the initialization phase ,in Then, based on the pre-trained client model With encoder Together, they serve as teacher models, providing knowledge transfer for heterogeneous client models; (6) Conduct heterogeneous splitting learning training based on the already initialized client model and adapter.

2. The heterogeneous splitting learning collaborative optimization method based on feature alignment adapter according to claim 1, characterized in that, The width mentioned in step (2) refers to the number of convolutional kernels in each layer, and the depth refers to the number of layers in the model. For resource-constrained low-power device clusters, a highly lightweight model is generated to adapt to their limited computing power; for device clusters with relatively abundant resources, a relatively light pruning is performed to generate a more complex model in order to make full use of their computing power.

3. The heterogeneous splitting learning collaborative optimization method based on feature alignment adapter according to claim 1, characterized in that, Step (4) The specific training process includes: (41) For input data in the public dataset First, it goes through the client-side model. Generate the corresponding feature representation Then The input is fed into various autoencoders for compression and restoration: Ultimately through the server model Calculate the prediction results: ; (42) During backpropagation, only the autoencoder parameters are optimized. and ,and parameters and The above process is repeated iteratively until each autoencoder model converges, while remaining unchanged. (43) Save the decoder portion of these trained autoencoders as an initialized feature alignment adapter for subsequent heterogeneous split learning joint training, i.e. .

4. The heterogeneous splitting learning collaborative optimization method based on feature alignment adapter according to claim 1, characterized in that, The specific initialization strategy for the heterogeneous client model based on feature distillation described in step (5) is as follows: (51) The pre-trained client model from step (4) With encoder Combining the teacher model and the heterogeneous client model As a student model, multiple client clusters may share the same output dimension. Therefore, they are assigned to the same teacher model. The client set is defined as ,in For client sub-model The output dimension; Let the public dataset be ,in Let N be the i-th sample and N be the dataset size. For each feature dimension... Teacher Model and student model The definition is as follows: (52) During the distillation process, each client model learns from the shared teacher model. Learning knowledge in the middle, freezing the parameters of the teacher model Update the parameters of the student model Knowledge distillation is achieved by minimizing the mean squared error (MSE) between feature representations. The distillation loss is expressed as follows: This initialization process applies to all Execute in parallel; After initialization, each model Distribute to cluster The client in the example is taken as its initial client model, where the model assigned to client k is denoted as... .

5. The heterogeneous splitting learning collaborative optimization method based on feature alignment adapter according to claim 1, characterized in that, In the learning and training performed in step (6), the steps executed during the t-th training iteration are as follows: (61) Each client From local dataset Independently sample a portion of the data and uses its pre-trained model to generate intermediate feature representations. And the client represents the features it calculates. Uploaded to the server; (62) For those sharing the same feature dimensions client collection Server-side adapter allocation It represents all features Mapping aligned to a uniform dimension To adapt to the server model The input requirements are as follows: the aligned feature representation is... Then, the projected feature representations are concatenated along the batch dimension to obtain... ,in ; (63) Aggregate the outputs of all adapters into a global data batch. And input the server model Reasoning is performed to obtain the prediction result. ; (64) In the backpropagation phase, the server first calculates the number of clients. k Losses: in Indicates client k The model parameters involved in performing full inference, where y is the true label of the training sample x. Let the loss function be the global loss, which is the weighted sum of the losses of all clients. in This represents the global batch size processed by the server. Based on the calculated global loss, the server updates the parameters using gradient descent, with a learning rate of [missing information]. The process is represented as follows: in, Indicates client k The sample proportion in the current global batch, for adapters also deployed on the server, is updated as follows: in This indicates that the number of samples from client k in the current global batch represents a percentage of the total number of clients in the client set. The proportion of all clients in the current global batch to the total number of samples; (65) On the client side, each client first receives the gradient returned by the server: Then calculate the gradient of the local model: Subsequently, each client updates its local model parameters using the following formula: After completing steps (61)-(65) above, a global mini-batch iteration is performed. This process continues until the global model converges.

6. A heterogeneous splitting learning collaborative optimization system based on feature alignment adapter, characterized in that, The system includes a storage component and a processing component. The storage component stores one or more computer instructions. The one or more computer instructions are invoked and executed by the processing component to implement the heterogeneous splitting learning collaborative optimization method based on feature alignment adapter as described in any one of claims 1 to 5, and to train local and global models deployed on clients and servers according to the method, so as to realize the collaborative processing of computational tasks including image classification and text analysis by intelligent edge devices.

7. The heterogeneous splitting learning collaborative optimization system based on feature alignment adapter according to claim 6, characterized in that, This system constructs a personalized local underlying model based on local computing resources and data distribution characteristics, and performs collaborative training with the top-level model shared on the server side. Specifically, it includes: The client executes the client model. As the encoder extracts basic features, it also includes a split learning framework to achieve collaborative training of multiple clients. By introducing a feature alignment adapter module and a global mini-batch stochastic gradient descent algorithm, it achieves the alignment of features from heterogeneous clients. Server, Execution Server Model It acts as a decoder to perform deep forward propagation and compute the final task prediction results.

8. A computer storage medium, characterized in that, The system contains a computer program that, when executed by a computer, implements the heterogeneous splitting learning collaborative optimization method based on feature alignment adapter as described in any one of claims 1-5.

Images