A cross-bit federated collaborative modeling method under data and model dual heterogeneity

Abstract

The application discloses a cross-bit federal collaborative modeling method under data and model dual heterogeneity. The application enables the edge terminal of the high-bit level model to connect the knowledge transmission between different bit models and reduce quantization error. For the bit heterogeneity scene, an edge local validation set assisted same level model assistance optimization strategy is designed, and the large and small edge terminals communicate with each other at the same level to cooperatively optimize and specifically reduce quantization loss. For the model drift problem under data heterogeneity, a model pseudo-update strategy is additionally designed on the edge terminal of the high-bit model, data and computing power are indirectly shared, deep fusion of similar edge terminals is realized, and the stability of model optimization is improved. In order to reduce the complexity of model screening, a Contact Map indicating potential candidate models is designed and stored in the cloud. The application solves the joint optimization problem of cross-bit models in the edge terminal data heterogeneity under the condition of protecting data privacy, and has superiority in model accuracy and training stability.

Description

Technical Field

[0001] This invention belongs to the field of big data computing and relates to a cloud-edge collaborative federated modeling method for edge-end models with different bit widths and heterogeneous data distribution. Background Technology

[0002] In recent decades, edge device hardware technology has seen tremendous development. Intelligent edge devices, equipped with various sensors, computing, and communication modules, such as embedded sensors, drones, mobile phones, smartwatches, and in-vehicle systems, are experiencing rapid growth in application across industrial production, daily life, and transportation. The widespread adoption of edge devices has facilitated the research and development of federated learning. Federated learning is a learning framework in which multiple edge devices collaboratively train on local private data, while periodically communicating by a server to aggregate and distribute local models. This not only protects data privacy but also fully utilizes the computing resources of edge devices for cloud-edge collaborative modeling.

[0003] A key challenge in federated learning lies in the significant differences in data distribution, tasks, model architectures, and devices among the various endpoints. This often leads to incompatibility between aggregations, making collaborative modeling impossible through simple aggregation and distribution logic. This problem is commonly referred to as the heterogeneous federated learning problem. Many existing federated learning methods address the heterogeneity of data, model architecture, and devices. Among these, there are various personalized optimization methods for data heterogeneity, such as local fine-tuning, aggregation correction, and federated transfer learning. However, in real-world federated learning scenarios, the devices at each endpoint may have different bit widths, i.e., different model bit widths. Some endpoints use models built on lightweight devices based on low-bit hardware operations, such as FPGAs, ASICs, Raspberry Pis, or edge GPUs. In the case of heterogeneous model bit widths, traditional federated learning methods face several challenges. First, after aggregating model parameters with different bit widths, a bottleneck effect occurs due to quantization errors, causing a shift in the true distribution of parameters. After distributing the models, the distribution of parameters from high-bit models tends to converge towards those from low-bit models, and the training loss includes unavoidable quantization errors. Second, models with low-bit parameters have significantly limited representational capabilities. Some existing methods consider directly quantizing high-bit models before participating in aggregation, but this reduces the representational power of the high-bit models themselves. How to achieve federated collaboration in scenarios with limited hardware and heterogeneous data distribution and model bit depth is a problem that urgently needs to be solved. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing federated modeling methods in handling the dual heterogeneity of data and model bits by providing a cross-bit federated optimization modeling method based on multiple model layers. This method constructs a ContactMap in the cloud to guide collaborative optimization between edges during federated modeling. It employs a peer-to-peer model assistance strategy aided by a proposed edge-local validation set, and selects the optimal model for each edge's model layer based on the cross-bit model library of each edge, specifically reducing quantization loss during collaborative optimization. The designed model pseudo-update strategy supports deep fusion of similar edges.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A cross-bit federation collaborative modeling method is proposed for cloud-edge collaborative scenarios with heterogeneous data distribution and model bits. The cloud-edge collaborative scenario includes N edge terminals and one cloud terminal. The N edge terminals are divided into large edge terminals and small edge terminals based on the number of models they can handle at a bit level. Small edge terminals can only handle one low-bit model. Each edge terminal contains a local database D. i A set of cross-bit model libraries M i The local database is used to store local historical data and construct training and validation datasets; the cross-bit model library is used to store models at one or more bit levels constructed for corresponding edges; the method specifically includes:

[0007] Initialize the model for each edge's cross-bit model library:

[0008] For the large edge, the model with the highest bit level is randomly initialized, then input into the training dataset, and optimized using the loss function to obtain the initial model with the highest bit level. Based on the initial model with the highest bit level, the models of the other bit levels are initialized step by step using a quantization-aware strategy. For the small edge, the model with the corresponding bit level is randomly initialized.

[0009] Each edge's cross-bit model library alternately performs local model updates and cloud-edge collaborative model optimization until a set number of times is reached;

[0010] The method for local model updates across bit-based model libraries at each edge is as follows:

[0011] For large edges, samples from the training dataset are used as input to the model with the highest bit level. The model is then optimized using a loss function to obtain a locally updated model with the highest bit level. Based on this model, the models at other bit levels are updated level by level using a quantization-aware strategy and transmitted to the cloud. For small edges, samples from the training dataset are used as input to the model at the corresponding bit level. The model is then optimized using a loss function to obtain a locally updated model and transmitted to the cloud. The cloud is used to receive and distribute the trained and updated models from each edge to other edges.

[0012] The method for each edge to optimize the cloud-edge collaboration model based on the received models from other edges is as follows:

[0013] Each edge receives models from other edges as candidate models, inputs the validation dataset into each candidate model and the local bit-level model corresponding to the edge, validates them, and calculates the accuracy. The model with an accuracy greater than that of the local bit-level model corresponding to the edge is selected as the optimal model for each edge level. If the accuracy of all candidate models is less than that of the local bit-level model corresponding to the edge, then the local bit-level model corresponding to the edge is selected as the optimal model for the corresponding bit level. After filtering, the edge's cross-bit model library is updated.

[0014] Furthermore, for the smaller edge, the optimal model for each bit level corresponding to the edge is obtained by selecting the model whose accuracy is greater than that of the local bit level corresponding to the edge, as follows:

[0015] Candidate models are first screened from the models at the same bit level as the large edge. If there is a model with an accuracy greater than the accuracy of the local model at the same bit level corresponding to the edge, the model with the highest accuracy is selected as the optimal model at the bit level corresponding to the edge after collaborative update. If the model at the same bit level of the large edge does not meet the condition, the screening continues from the models at the same bit level of other small edges. If there is a model with an accuracy greater than the accuracy of the local model at the same bit level corresponding to the edge, the model with the highest accuracy is selected as the optimal model at the bit level corresponding to the edge after collaborative update. If none of the candidate models at the same bit level meet the condition, the local model at the same bit level corresponding to the edge is selected as the optimal model at the bit level corresponding to the edge.

[0016] Furthermore, for large edges, the candidate model also includes:

[0017] The current large edge is sent to other large edges, and the pseudo-updated model is trained and optimized using the training datasets of these other large edges before being returned. To address the model drift problem under heterogeneous data, an additional pseudo-updating strategy is designed for the large edges to achieve deep fusion of similar edges.

[0018] Furthermore, for the large edge, the optimal model for each bit level corresponding to the edge is obtained by selecting the model whose accuracy is greater than that of the local bit level corresponding to the edge, as follows:

[0019] For each bit level, candidate models are first screened from the pseudo-update models of the same bit level. If there is a model with an accuracy greater than the accuracy of the local model of the same bit level corresponding to the edge, then the model with the highest accuracy is selected as the optimal model for the bit level corresponding to the edge after collaborative update. If the pseudo-update models of the same bit level do not meet the condition, the screening continues from the models of other large edges at the same bit level. If there is a model with an accuracy greater than the accuracy of the local model of the same bit level corresponding to the edge, then the model with the highest accuracy is selected as the optimal model for the bit level corresponding to the edge after collaborative update. If the models of other large edges at the same bit level do not meet the condition, the screening continues from the models of other small edges at the same bit level. If all candidate models for the same bit level do not meet the condition, then the local model of the same bit level corresponding to the edge is selected as the optimal model for the bit level corresponding to the edge.

[0020] Furthermore, the cloud also includes a Contact Map, which is an N*N matrix. The element in the i-th row and j-th column records the total number of times edge i selects the model of edge j as the optimal model in the cloud-edge collaborative model optimization. The cloud selects and distributes models to each edge based on the Contact Map, as follows:

[0021] After the CM round of cloud-edge collaborative model optimization, the cloud, based on the communication graph, only distributes the models of other edges with elements in the communication graph greater than the threshold q to the corresponding edges as candidate models.

[0022] The cloud-based design incorporates a communication control strategy, using a Contact Map to indicate candidate models from other edges to each edge. Large and small edges collaborate on candidate model optimization within the same layer, specifically reducing quantization loss.

[0023] Furthermore, after the CM round of cloud-edge collaboration model optimization, the communication graph stops updating.

[0024] Furthermore, the loss function can be cross-entropy loss, binary cross-up loss, mean squared error loss, squared loss, etc.

[0025] Furthermore, for the smaller edge, its loss function is expressed as:

[0026] L i =CE(f(x) i ;(w i b i ));yi )

[0027] f(·;(w i b i )) indicates that the current bit type of this edge is b. i The weight is w i The model to be optimized, x i y i These represent the input data and corresponding label contained in the i-th edge, respectively.

[0028] For the large-edge model, the loss function of the model with the highest bit level is expressed as:

[0029]

[0030] In the formula, CE represents the cross-entropy loss. This indicates that the current bit type of this edge is b1, and the weight is w. i，1 The model to be optimized, x i y i These represent the input data and corresponding label contained in the i-th edge, respectively. b1 is the highest bit type in the bit level of this large edge.

[0031] The loss function for models at all bit levels except the highest bit level is expressed as follows:

[0032]

[0033] In the formula, CE represents the cross-entropy loss. This indicates that the edge weight is Simulation b k The scenario is calculated, but the actual operation involves an optimization model of type b1, x i y i These represent the input data and corresponding label contained in the i-th edge, respectively. The simulation model case during the calculation is represented by b. k The introduced quantization loss. b1 is the highest bit type in this large-edge bit level.

[0034] Traditional federated learning algorithms do not consider the model bit heterogeneity problem caused by hardware limitations at the edges, and cannot effectively handle quantization loss when co-optimizing different models with varying accuracy. Compared with existing technologies, the beneficial effects of this invention include at least the following:

[0035] 1. A cross-bit model-by-bit level collaborative optimization strategy is proposed. This invention designs a federated collaborative modeling strategy from the perspective of unified quantization error level. The large edge connects the knowledge transfer tasks between models of different bit levels. The large and small edges communicate with each other at the bit level to reduce quantization loss in a targeted manner, thereby improving the performance of the low-bit model without reducing the representation ability of the high-bit model.

[0036] 2. A pseudo-update strategy for the model is designed. When performing co-optimization at the same level, this invention additionally designs a pseudo-update strategy for the model at the large edge, indirectly sharing data and computing power, realizing deep fusion of similar edges, and improving the stability of model optimization.

[0037] 3. A cloud-based communication control strategy was designed. This invention records the model selection during collaborative optimization of each edge in history through a cloud-based Contact Map, indicates the source of the current collaborative model for each edge, and temporarily stores the alternative models and verification results during model optimization of each edge, thereby reducing the storage and computing resources of the edge and effectively reducing the amount of communication. Attached Figure Description

[0038] Figure 1 This is a flowchart of the overall modeling steps of the present invention;

[0039] Figure 2 This is a schematic diagram of N cross-bit edges in one embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the step-by-step initialization strategy for the large edge model;

[0041] Figure 4 A schematic diagram illustrating the optimization strategy for models with small edges;

[0042] Figure 5 This is a schematic diagram illustrating the model selection order during large-edge collaborative updates.

[0043] Figure 6 A schematic diagram illustrating the method for establishing a pseudo-update model for large edges;

[0044] Figure 7 A flowchart for creating a cloud-based Contact Map;

[0045] Figure 8 The results of the Contact Map under different ablation methods are shown in (a) for the method of this invention, (b) for the result when the pseudo-model is missing, and (c) for the result when the cross-bit model layer is missing. Detailed Implementation

[0046] This invention discloses a cross-bit federation collaborative modeling method for data distribution and model bit heterogeneity in a cloud-edge collaborative scenario. The cloud-edge collaborative scenario includes N edge terminals and one cloud terminal. The N edge terminals are divided into large edge terminals and small edge terminals according to the number of bit levels of models they can handle. Small edge terminals can only handle one low-bit model, while large edge terminals can handle at least two bit levels of models. Figure 2 The diagram shown contains three types of edges, with N edges. N=K a +K b +K c K a K b K c The number of edges included in the three categories are as follows: Category A edges can use model bit types up to FP32 and can simultaneously support models with four bit levels: FP32, INT16, INT8, and INT4; these are considered large edges. Category B edges can use model bit types up to INT8 and can only support models with one bit level (INT8); these are considered small edges. Category C edges can use model bit types up to INT4 and can only support models with one bit level (INT4); these are also considered small edges. It should be noted that although this embodiment does not include edges with a model bit type up to INT16 due to experimental limitations, this can be extended to this extent according to the specification.

[0047] Each edge contains a local database D i A set of cross-bit model libraries M i , where i is the current edge number; the local database is used to store local historical data and construct training and validation datasets; taking image classification as an example, each sample {x, y} in the training and validation datasets contains image data x and the corresponding classification label y; the cross-bit model library is used to store one or more bit-level models constructed for the corresponding edge; specifically, for small edges, which only support one model stored at low bit levels, only the low-bit level model is retained, M i ={(w i b i )},(w i b i ) i Represents the i-th edge b i Bit-level model parameters, b i This indicates the current model bit type. If the lower bit is INT4, and w represents the model weight, then M... i ={(w i ,INT4)}. For the large-edge model, it contains multiple bit-level models. middle Represents the i-th edge b i Bit-level model parameters, b i This indicates the current model bit type. If the largest edge can store the model at a maximum of FP32, then different model layers are set from high bits to low bits. In this case, M... i ={(w i，FP32 ,FP32), (w i，INT16 ,INT16), (w i，INT8 ,INT8), (w i，INT4 ,INT4)}. Models with the same bit type at different ends form a model layer at that bit level.

[0048] The method steps of this invention are as follows: Figure 1 As shown, the details are as follows:

[0049] Step 1: Initialize the cross-bit model library for each edge. For large edges, design multiple model layers and perform step-by-step initialization using a quantization-aware strategy. Input the edge image training data and output the initialized model for each layer. For small edges, design a single model layer and perform random initialization.

[0050] Step 2: Cloud-Edge Collaborative Model Optimization. Using the proposed edge-local validation set-assisted same-level model optimization strategy, the edge image validation dataset is input into each candidate model, and the optimal model for each edge level is selected. Specifically, each edge uses the models received from other edges as candidate models, and inputs the validation dataset into each candidate model and the edge's corresponding local bit-level model for validation and accuracy calculation. The optimal model for each edge level is selected based on its accuracy greater than that of the edge's corresponding local bit-level model. If the accuracy of all candidate models is less than that of the edge's corresponding local bit-level model, then the edge's corresponding local bit-level model is selected as the optimal model for the edge's corresponding bit level. After selection, the edge's cross-bit model library is updated. This invention provides collaborative optimization of candidate models within the same level by both large and small edges, specifically reducing quantization loss.

[0051] Step 3: Local Model Update: For large edges, samples from the training dataset are used as input to the model with the highest bit level. The loss function is then used to optimize and obtain the locally updated model with the highest bit level. Based on the model with the highest bit level, the models of the remaining bit levels are updated step by step using a quantization-aware strategy and then transmitted to the cloud. For small edges, samples from the training dataset are used as input to the model with the corresponding bit level. The loss function is then used to optimize and obtain the locally updated model, which is then transmitted to the cloud. The cloud is used to receive and distribute the trained and updated models from each edge to other edges.

[0052] Step 3 is repeated alternately with Step 2 until the set number of times is reached, thus completing the modeling of each edge.

[0053] After collaborative modeling is completed, each edge uses the local model with the highest precision bit level as the model used in the inference stage. For example, the image classification model can be used to verify the classification effect on image test data.

[0054] Due to hardware limitations, the modeling precision varies across different edge devices. Some edge devices have sufficient storage and computing resources to support full-precision modeling, while others only support low-precision model training and inference. To achieve cross-bit model collaborative training, this invention constructs a cross-model database across different bit segments. In a specific implementation, during the model initialization phase, for low-bit models in the cross-bit model library of large edge devices, a quantization-aware training strategy is used for step-by-step initialization; for individual models of small edge devices, a random initialization method is used to reduce computational load. Specifically, during the model initialization phase, if it is a small edge device, the bit type is b... i Then M i ={(w i b i )},w i Perform random initialization; if the model has a large margin, use the local image training dataset as input to each model in the model layer, and train the model to minimize the classification loss. middle Represents the i-th edge b i Bit-level model parameters, b i This indicates the current model bit type, specifically M in this embodiment. i ={(w i，FP32 ,FP32), (w i，INT16 ,INT16), (w i，INT8 ,INT8), (w i，INT4 ,INT4)}。 Specifically, the training adopts a step-by-step initialization method. During step-by-step training, since the operation process in the large edge is still performed in FP32 type, the corresponding quantization loss can only be added through a quantization-aware strategy to simulate the training of models of other bit types. Therefore, for models at lower bit levels, during initialization, the model of the previous level is quantized to the current bit type using the quantization operation Q(·), and then dequantized Q. -1 (·) Continue training on FP32. Specific steps are as follows: Figure 3 As shown, it represents the following:

[0055] (1)w i，FP32 Randomly initialize, then update R1 rounds using the local training set, with the following loss function:

[0056] L i，FP32=CE(f(x) i ;(w i，FP32 ,FP32));y i ).

[0057] In the formula, L i，FP32 Let f(·;(w) represent the loss function for the i-th edge with bit level FP32, CE represent the cross-entropy loss, and f(·;(w)) represent the loss function for the ith edge with bit level FP32. i，FP32 FP32) indicates that the current bit type of this edge is FP32 and the weight is w. i The model to be optimized, x i y i These represent the input data and corresponding label contained in the i-th edge, respectively.

[0058] (2) For each low-bit level model, the model from the previous level is used as the initial model for this level after quantization and dequantization operations. Then, the local training set is used as input to minimize the loss function for model training. Taking the INT16 model layer as an example, quantization and dequantization operations are first performed on the model from the previous level: (w i，INT16 ,FP32)=Q -1 (Q INT16 (w i，FP32 ,FP32), and then update the R rounds using the local image training dataset. During training, a quantization-aware strategy is used, and the loss function is as follows:

[0059] L i，INT16 =CE(f(x) i ;(w i，INT16 ,FP32));y i )+E INT16 ,

[0060] Among them, L i，FP16 Let f(·;(w) represent the loss function for the i-th edge with bit level FP16. i，INT16 FP32) indicates that the edge weight is w i，INT16 The model to be optimized simulates the INT16 case but is actually of type FP32 in actual operation. INT16 This represents the quantization loss introduced by INT16 in the simulated model during computation. This quantization loss is introduced by performing a quantization operation Q(·) on the FP32 model parameters in each training round, quantizing to INT16 precision, and then performing a dequantization operation to return to FP32 precision, i.e., (w i，INT16 ,FP32)=Q- 1 (Q INT16 (w i，INT16 After training, perform a quantization operation Q(·), (w i，INT16 ,INT16)=Q INT16(w i，INT16 , FP32), to obtain the model after training the model layer.

[0061] After model initialization, the same-level collaborative optimization of large and small edges alternates with local model updates. Among them, the same-level collaborative optimization ensures that the quantization loss level is consistent. In this collaborative optimization, the large edge acts as a bridge to connect models at different levels and bridge the multi-level quantization errors between cross-bit models.

[0062] In a specific embodiment, the same-level model-assisted optimization strategy proposed in this invention is implemented as follows:

[0063] In the cloud-edge collaborative model optimization phase, the proposed same-level model assistance optimization strategy is used for model selection, which is divided into two cases: small edge and large edge.

[0064] For the small edge, the edge ID is i, and the bit type is b. i At this time, M i ={(w i b i Each edge will receive models of the same level from other edges, and these models will be used as candidate models. Calculate the accuracy (acc) of different candidate models on the small-side validation set. i，j With the local model w of the current edge i accuracy acc i Screening strategies such as Figure 4 As shown. Candidate models are first selected from those at the same level as the larger edge, and if existing models are found... Make acc i，j >acc i If the model with the highest accuracy is selected as the optimal model after collaborative updating, it becomes the updated local model. If the model at the same level as the larger edge does not meet the criteria, the selection continues from the models at the same level as the smaller edges, again selecting the model with the highest accuracy as the local model after collaborative updating. If none of the candidate models at the same level meet the criteria, the current optimal model is the local model. The sequential selection process of the smaller edges is denoted as operation Bs(·), which records the source edge ch of the selected model. i =Bs(Candidate) i w i If ch i =j, the selected model is used as the new local model. If ch i =i,w i It remains unchanged.

[0065] For the large edge, the edge ID is i, and M is... i ={(wi，FP32 ,FP32), (w i，INT16 ,INT16), (w i，INT8 ,INT8), (w i，INT4 For each bit level, the candidate models also include: pseudo-update models that are sent from the current large edge to other large edges and trained and optimized using the training datasets of those other large edges, and then returned. Each bit level prioritizes selecting the optimal model from the pseudo-update models, using the following selection strategy: Figure 5 As shown. In bit type b i Taking the hierarchical level as an example, models from other edge levels of the same level are used as candidate models. Calculate the accuracy of different candidate models on the large edge validation set. Compared with the current local model w i accuracy Candidate models are first screened from the year-on-year pseudo-update model with large margins. If a model exists... Make The model with the highest accuracy is selected as the optimal model after collaborative updating. If the pseudo-update model at the bit level of the large edge does not meet the condition, the selection continues from the non-pseudo-update models at the bit level of other large edges, and then from the bit level models at the bit level of other small edges. If all candidate models at the bit level do not meet the condition, the current optimal model is the local model at that bit level. The sequential selection process of the large edge is denoted as operation Bb(·), which records the source edge ch of the selected model. i，bi =Bb(Candidate) i w i，bi If ch i，bi =j, the selected model is used as the new local model. If ch i，bi =i,w i，bi It remains unchanged.

[0066] The pseudo-update operation of the large edge is as follows: Figure 6 As shown, a pseudo-update model is constructed in each model layer of the large edge to make fuller use of the large edge's computing and storage resources. Specifically, the models of each bit level of the large edge are sent to other large edges via the cloud. These models are then trained and optimized on the datasets of other large edges, mimicking the models at different bit levels in their local model libraries, and then returned to obtain pseudo-update models. These pseudo-update models participate in subsequent model selection and have the highest selection priority. The loss function of the pseudo-update model at each bit level is as follows:

[0067] L i，j，FP32 =CE(f(x) j ;(w i，FP32 ,FP32));y j ), j≠i

[0068] L i，j，INT16 =CE(f(x) j ;(w i，INT16 ,FP32));y j )+E INT16 j≠i

[0069] L i，j，INT8 =CE(f(x) j ;(w i，INT8 ,FP32); y j )+E INT8 j≠i

[0070] L i，j，INT4 =CE(f(x) j ;(w i，INT4 ,FP32));y j )+E INT4 j≠i

[0071] Where i is the index of the current large edge, j is the index of another large edge, and x j y j Let L represent the input data and corresponding label contained in the j-th edge, respectively. i，j，* E represents the loss function E that is used to train the model with the i-th edge and bit level * using the training dataset of the j-th edge. * This represents the quantization loss introduced during the simulation of the model during computation. A quantization-aware strategy is used during low-level training; for example, at the INT16 level, f(·;(w i，INT16 FP32) indicates that the edge weight is w i，INT16 The model to be optimized simulates the INT16 case but is actually of type FP32 in actual operation. INT16 This represents the quantization loss introduced by INT16 in the simulated model during computation. This quantization loss is introduced by performing a quantization operation Q(·) on the FP32 model parameters at each model update round, quantizing to INT16 precision, and then performing a dequantization operation to return to FP32 precision, i.e., (w i，INT16 ,FP32)=Q -1 (Q INT16 (w i，INT16 After training, perform another quantization operation (w) i，INT16 ,INT16)=Q INT16 (w i，INT16 , FP32), to obtain the model after training the model layer.

[0072] The cloud-edge collaborative model optimization and local update are performed alternately. In the local update phase, the input is the edge training dataset, and the output is the locally updated model. At this time, the model training is divided into two different cases: small edge and large edge.

[0073] If it is a small edge, the bit type is b. i Then M i ={(w i b i The model training loss function is:

[0074] L i =CE(f(x) i ;(w i b i ));y i )

[0075] f(·;(w i，1 b i )) indicates that the current bit type of this edge is b. i The weight is w i The model to be optimized.

[0076] If it is a large edge, then model M needs to be updated. i ={(w i，1 ,FP32), (w i，2 ,INT16), (w i，3 ,INT8), (w i，4 The model in each bit level is updated separately, and the corresponding loss function is as follows:

[0077] L i，FP32 =CE(f(x) i ;(w i，FP32 ,FP32));y i ),

[0078] L i，INT16 =CE(f(x) i ;(w i，INT16 ,FP32));y i )+E INT16 ,

[0079] L i，INT8 =CE(f(x) i ;(w i，INT8 ,FP32));y i )+E INT8 ,

[0080] L i，INT4 =CE(f(x) i ;(w i，INT4 ,FP32));y i)+E INT4 ,

[0081] Where i is the index of the current large edge. The low-bit level model training employs a quantization-aware strategy; for example, in INT16, f(·;(w i，INT16 FP32) indicates that the edge weight is w i，INT16 The model to be optimized simulates the INT16 case but is actually of type FP32 in actual operation. INT16 This represents the quantization loss introduced by INT16 in the simulated model during computation. Additionally, a quantization operation needs to be performed after training. (w) i，INT16 ,INT16)=Q INT16 (w i，INT16 , FP32), to obtain the trained model of this layer.

[0082] In a specific implementation case, the cloud is designed with a communication control strategy. Based on the selection results of each edge auxiliary model, a communication map is constructed and updated to guide the selection of subsequent candidate models for each edge.

[0083] After local model initialization, the cloud-edge collaborative model optimization phase begins. In this invention, the cloud is designed with a communication control strategy, such as... Figure 7 As shown, the details are as follows:

[0084] The cloud communication graph, Contact Map, is an N*N matrix that records the selection results of candidate models for each edge. Map = {chosen} i，j}, i≠j, i, j∈{1,2,...,N}. Its element chosen in the i-th row and j-th column. i，j Record the total number of times edge i selects the model of edge j as the optimal model in the cloud-edge collaborative model optimization; all elements in the Map are initialized to 0. During model selection, if edge i selects the candidate model of edge j to update its local model, that is, for the smaller edge, ch i =j, for the larger edge, ch i，bi =j, then choose i，j +1. After the CM rounds of collaborative updates, the maintenance of the Contact Map ends. At this point, for edge i, only choices are obtained from the cloud. i，j The model of edge j with >q is used as a candidate model, and it no longer indirectly connects to other edges, thereby reducing the storage, computing resources and communication costs required by the edges.

[0085] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0086] This embodiment uses the CIFAR10 color image dataset as an example, which contains 10 classes: airplane, car, bird, cat, deer, dog, frog, horse, boat, and truck. Each class has 6000 images, and each image is 32×32 pixels. 40,000 samples in this dataset are used as the training set, 10,000 samples as the validation set, and the remaining 10,000 samples as the test set. The distribution of sample classes differs across different edge data points; in this embodiment, it is assumed that they conform to different hypergeometric distributions. At the edge, edges that can support multiple high-bit models and low-bit models (including FP32, INT16, INT8, INT4, etc.) are denoted as large edges, while edges that can only support one low-bit model (such as INT8, INT4) are denoted as small edges. This embodiment sets up a total of 20 edges, including 8 large edges and 12 small edges, with a total of 6 data distributions. The data distributions of each edge are in the following order by ID: 1, 1, 2, 2, 3, 3, 4, 4, 1, 1, 2, 2, 3, 3, 4, 4, 0, 0, 5, 5. The precision of the edge model parameters is in the following order by ID: FP32, FP32, FP32, FP32, FP32, FP32, FP32, FP32, INT8, INT4, INT8, INT4, INT8, INT4, INT8, INT4, INT8, INT4, INT8, INT4.

[0087] A cross-bit federation collaborative modeling method for image classification models in a cloud-edge collaborative scenario with dual heterogeneity in data distribution and model bits, based on the method of this invention, includes the following steps:

[0088] (1) Model initialization: For large-edge models, which can support multiple high-bit and low-bit CNN models, multiple model layers are designed and initialized step by step using a quantization-aware strategy. For small-edge models, which can only support one low-bit model, a single model layer is designed and initialized randomly. In the cloud, a Contact Map is constructed and initialized, with each element in the Contact Map set to 0.

[0089] (2) Cloud-Edge Collaborative Model Optimization: Model collaborative optimization and local model updates are performed alternately. The proposed edge-local validation set-assisted optimization strategy uses the same-level model assistance set. The edge image validation dataset is input into each candidate model, and the optimal model at each edge level is selected. A communication control strategy is designed in the cloud, using a Contact Map to indicate candidate models from other edges to each edge. Both large and small edges provide collaborative optimization of candidate models within the same level, specifically reducing quantization loss. To address the model drift problem under heterogeneous data, an additional pseudo-update strategy is designed for large edges to achieve deep fusion of similar edges. The CM is set to 25 and q is set to 10 during Contact Map updates.

[0090] (3) Local Model Update: The large and small edges are optimized based on the local edge data and loss function, and the local model is updated. This step is performed alternately with step (2) for 200 iterations.

[0091] After the model is co-trained, the local model of the highest precision layer is used as the model for the inference stage in different edges. The image classification model constructed by this invention can be used online for image classification tasks, and the classification effect can be verified on the test set.

[0092] Federated modeling training was conducted using edge data distributions with distribution ID 1 to study the case where edge data distributions are similar. When edge data distributions are similar, the model classification accuracy under different types of edges is shown in Table 1. The table also shows the average accuracy of large and small edges in the early stage of training, namely the tenth round of global collaboration and the last five rounds of collaborative updates. It can be found that regardless of whether the edge is large or small, the method of this invention is better than the comparison method, and its performance is quite good in the early stage of optimization.

[0093] Table 1. Comparison of classification accuracy of this invention with other federated modeling methods when data distribution is independent and identically distributed.

[0094]

[0095] When edge data is heterogeneous, the classification accuracy of the model under different edge types is shown in Table 2. The table also shows the average accuracy of large and small edges in the early stage of training, namely the tenth round of global collaboration and the last 5 rounds of collaborative updates. It can be seen that regardless of the size of the edge, the method of the present invention is better than the comparison method, and its performance is excellent in the early stage of optimization. The table also shows the classification effect for locally seen and unseen categories. It can be seen that the method of the present invention has a significant improvement in the classification effect of unseen categories compared with the comparison method. Figure 8 The visualization results of Contact Map under different ablation methods are shown. The three sub-figures (a), (b), and (c) respectively show the Contact Map when using the proposed method, the proposed method without pseudo-update model, and the proposed method without multiple model layers. It can be seen that the model pseudo-update mechanism and multiple model layers are necessary. If these mechanisms are missing, the smaller edge is more likely to select a model with a different data distribution from its edge for optimization.

[0096] Table 2. Comparison of classification accuracy of this invention with other federated modeling methods in the case of heterogeneous data.

[0097]

[0098] This invention is not limited to the field of image classification as described above. Anyone skilled in the art can make equivalent modifications or substitutions without departing from this invention, and all such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A cross-bit federation collaborative modeling method for data distribution and model bit heterogeneity in a cloud-edge collaborative scenario, characterized in that, The cloud-edge collaboration scenario includes N edge devices and one cloud platform. The N edge devices are divided into large edge devices and small edge devices based on the number of bit-level models they can handle. Small edge devices can only handle one low-bit model. Each edge device contains a local database D. i A set of cross-bit model libraries M i The local database is used to store local historical data and construct training and validation datasets; the cross-bit model library is used to store models at one or more bit levels constructed for corresponding edges; the method specifically includes: Initialize the model for each edge's cross-bit model library: For the large edge, the model with the highest bit level is randomly initialized, then input into the training dataset, and optimized using the loss function to obtain the initial model with the highest bit level. Based on the initial model with the highest bit level, the models of the other bit levels are initialized step by step using a quantization-aware strategy. For the small edge, the model with the corresponding bit level is randomly initialized. Each edge's cross-bit model library alternately performs local model updates and cloud-edge collaborative model optimization until a set number of times is reached; The method for local model updates across the cross-bit model library at each edge is as follows: For large edges, samples from the training dataset are used as input to the model with the highest bit level, and optimization is performed using a loss function to obtain the locally updated model with the highest bit level. Based on the model with the highest bit level, the models of other bit levels are updated level by level using a quantization-aware strategy and then transmitted to the cloud. For small edges, samples from the training dataset are used as input to the model with the corresponding bit level, and optimization is performed using a loss function to obtain the locally updated model and then transmitted to the cloud. The cloud is used to receive and distribute the trained and updated models of each edge to other edges. The method for each edge to optimize the cloud-edge collaboration model based on the received models from other edges is as follows: Each edge receives models from other edges as candidate models, inputs the validation dataset into each candidate model and the local bit-level model corresponding to the edge, validates them, and calculates the accuracy. The model with an accuracy greater than that of the local bit-level model corresponding to the edge is selected as the optimal model for each edge level. If the accuracy of all candidate models is less than that of the local bit-level model corresponding to the edge, then the local bit-level model corresponding to the edge is selected as the optimal model for the corresponding bit level. After filtering, the edge's cross-bit model library is updated.

2. The method according to claim 1, characterized in that, For the small edge, the optimal model for each bit level corresponding to the edge is obtained by selecting the model whose accuracy is greater than that of the local bit level corresponding to the edge, as follows: Candidate models are first screened from the models at the same bit level as the large edge. If there is a model with an accuracy greater than the accuracy of the local model at the same bit level corresponding to the edge, the model with the highest accuracy is selected as the optimal model at the bit level corresponding to the edge after collaborative update. If the model at the same bit level of the large edge does not meet the condition, the screening continues from the models at the same bit level of other small edges. If there is a model with an accuracy greater than the accuracy of the local model at the same bit level corresponding to the edge, the model with the highest accuracy is selected as the optimal model at the bit level corresponding to the edge after collaborative update. If none of the candidate models at the same bit level meet the condition, the local model at the same bit level corresponding to the edge is selected as the optimal model at the bit level corresponding to the edge.

3. The method according to claim 1, characterized in that, For the large edge, the candidate model further includes: The current large edge is sent to other large edges, and the pseudo-update model is returned after training and optimization using the training datasets of other large edges.

4. The method according to claim 3, characterized in that, For large edges, the optimal model for each bit level corresponding to the edge is obtained by selecting models whose accuracy is greater than that of the local bit level corresponding to the edge, as follows: For each bit level, candidate models are first screened from the pseudo-update models of the same bit level. If there is a model with an accuracy greater than the accuracy of the local model of the same bit level corresponding to the edge, then the model with the highest accuracy is selected as the optimal model for the bit level corresponding to the edge after collaborative update. If the pseudo-update models of the same bit level do not meet the condition, the screening continues from the models of other large edges at the same bit level. If there is a model with an accuracy greater than the accuracy of the local model of the same bit level corresponding to the edge, then the model with the highest accuracy is selected as the optimal model for the bit level corresponding to the edge after collaborative update. If the models of other large edges at the same bit level do not meet the condition, the screening continues from the models of other small edges at the same bit level. If all candidate models for the same bit level do not meet the condition, then the local model of the same bit level corresponding to the edge is selected as the optimal model for the bit level corresponding to the edge.

5. The method according to claim 1, characterized in that, The cloud also includes a Contact Map, which is an N*N matrix. The element in the i-th row and j-th column records the total number of times edge i selects the model of edge j as the optimal model in the cloud-edge collaborative model optimization. The cloud selects and distributes models to each edge based on the Contact Map, as follows: After the CM round of cloud-edge collaborative model optimization, the cloud, based on the communication graph, only distributes the models of other edges with elements in the communication graph greater than the threshold q to the corresponding edges as candidate models.

6. The method according to claim 5, characterized in that, After the CM round of cloud-edge collaboration model optimization, the communication graph stopped updating.

7. The method according to claim 1, characterized in that, The loss function may be cross-entropy loss, binary cross-up loss, mean squared error loss, or squared loss.

Images