Parallel segmentation learning method and device based on knowledge distillation, equipment and medium

Abstract

The application provides a parallel segmentation learning method and device based on knowledge distillation, equipment and medium, relating to the technical field of terminal learning. The method provides a parallel segmentation learning framework based on knowledge distillation, including a cross-layer projection distillation module, an EMA prototype guidance module and a minimum frequency threshold constraint module. The three modules work together to form a closed-loop training mechanism. The cross-layer projection distillation module uses the global deep features of the server side as the teacher signal to guide the alignment of the shallow features of the client side to the global features, and suppresses local overfitting and feature drift. The EMA prototype guidance module maintains a global class prototype pool based on exponential moving average update on the server side, provides anti-noise and stable distillation targets for the client side, and smooths the class center drift. The minimum frequency threshold constraint module statistically compares the number of times of participating in training of the client side with the threshold value, forcibly guarantees the minimum participation opportunity of the long-tail client, and prevents data distribution forgetting.

Description

Technical Field

[0001] This invention relates to the field of terminal learning technology, specifically to a parallel segmentation learning method, apparatus, device, and medium based on knowledge distillation. Background Technology

[0002] With the rapid growth in the number of IoT devices and the continuous advancement of information and communication technologies, massive numbers of IoT terminals are accessing the network in real time, constantly generating large-scale data. Traditional machine learning typically employs a centralized learning model, requiring the raw data from each terminal to be uploaded to a central server for processing and model training. However, in real-world applications, due to strict privacy regulations and the lack of trusted mechanisms between terminal devices, edge devices often cannot directly share their private data, severely restricting the large-scale deployment of centralized AI systems. Furthermore, continuously uploading raw data to a remote cloud for a large number of edge devices incurs significant bandwidth consumption and transmission latency, making this solution difficult to implement effectively in resource-constrained IoT environments.

[0003] To address these issues, Federated Learning (FL) emerged. Federated Learning achieves collaborative modeling while protecting privacy by training models locally and uploading model parameters or gradients without sharing the original data. However, Federated Learning requires all training to be completed on the terminal device, placing high demands on the computing power and energy consumption of edge devices. For example, the typical image recognition model ResNet-50 has approximately 25 million parameters, a model size exceeding 100MB, and a single forward propagation computation of approximately 4 GFLOPs, still posing a significant burden for computing- and energy-constrained mobile terminals or IoT devices. To alleviate the computational burden on terminals, Split Learning (SL) was proposed. This framework divides deep neural networks into client and server parts, allowing lightweight computations to be performed on the client side, while computationally intensive tasks are handled by the server. This achieves a reasonable distribution of computational load between the edge and the cloud, effectively alleviating the computational pressure on terminal devices.

[0004] However, traditional split learning employs a sequential training mechanism, requiring frequent forward and backward propagation communication between the client and server, which can easily introduce significant communication latency. To address this, researchers have proposed parallel split learning (PSL) and split federated learning (SFL) frameworks to improve system concurrency and reduce communication overhead. However, within these frameworks, the data distribution across different clients typically exhibits non-independent and identically distributed characteristics, meaning significant statistical heterogeneity exists. This can disrupt the model aggregation process, leading to a decline in global model performance and impacting training stability and generalization ability.

[0005] To address the aforementioned issues, existing technologies have proposed several improvements. For example, one method divides all devices into multiple clusters and performs split federated learning independently within each cluster; however, this method heavily relies on real-time monitoring of device status. Another method uses a server to perform global random sampling and dynamically allocate client batch sizes, but this requires prior knowledge of the data volume of each client. Yet another method achieves semantic alignment of high-level features by performing cross-client weighted aggregation of the activation gradients of the last layer, thereby suppressing gradient direction shifts and update oscillations under data heterogeneity. Furthermore, some solutions combine knowledge distillation and incentive mechanisms, achieving cross-client knowledge sharing through knowledge distillation of server-side outputs and using a game-theoretic mechanism to guide clients to select data with larger volume and more even distribution for training, thus adapting to non-independent and identically distributed data; however, this method relies on the actual data distribution information reported by the clients.

[0006] While existing technologies have mitigated training instability caused by data heterogeneity in parallel split learning to some extent, they still have significant limitations. On one hand, these methods typically rely on strong prior assumptions, such as the need to accurately obtain client system states, data size, or distribution information. This is difficult to reliably obtain in dynamic and privacy-constrained real-world scenarios, limiting the applicability of these methods. On the other hand, some methods increase communication and computational overhead by introducing additional aggregation, distillation, or scheduling mechanisms, raising system complexity and reducing scalability and robustness in large-scale deployments. Furthermore, most schemes assume honest client participation and accurate information reporting, lacking constraints on untrusted or unstable clients, thus posing potential risks of privacy breaches and model performance degradation.

[0007] In view of the above, this application is hereby submitted. Summary of the Invention

[0008] This invention provides a parallel segmentation learning method, apparatus, device, and medium based on knowledge distillation, which can at least partially improve the above-mentioned problems.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A parallel segmentation learning method based on knowledge distillation, comprising:

[0011] The number of training parameters for each client is compared with the preset minimum frequency threshold to generate the client set for this round of training.

[0012] The deep features containing global semantic information on the server side are used as teacher features. Relevant data corresponding to the current task is obtained from the server side. Based on the teacher features and relevant data, the parameters of the clients in the client set are updated to obtain the updated client parameters. The updated client parameters are then sent to the server side.

[0013] Based on the updated client parameters, determine the average feature value uploaded by each client for each category, and perform smooth updates on the feature prototypes of the same category based on the average feature value. The updated feature prototypes will then be used as the feature prototypes to be sent to the client in the next round of training.

[0014] Repeat the above steps until all clients meet the preset training conditions.

[0015] The present invention also provides a parallel segmentation learning device based on knowledge distillation, comprising:

[0016] The client confirmation unit is used to compare the number of training parameters for each client with the preset minimum frequency threshold, and generate the client set for this round of training.

[0017] The parameter update unit is used to take the deep features containing global semantic information on the server as teacher features, obtain the relevant data corresponding to the current task issued by the server, and perform parameter update processing on the clients in the client set based on the teacher features and relevant data to obtain the updated client parameters, and then send the updated client parameters to the server.

[0018] The prototype update unit is used to determine the average feature value uploaded by each category client based on the updated client parameters, and to perform smooth update processing on the feature prototypes of the same category based on the feature average value. The updated feature prototypes are then sent to the client as the feature prototypes for the next round of training.

[0019] The iterative unit is used to repeat the above steps until all clients have met the preset training conditions.

[0020] The present invention also provides a parallel segmentation learning device based on knowledge distillation, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the parallel segmentation learning method based on knowledge distillation as described above.

[0021] The present invention also provides a readable storage medium storing a computer program that can be executed by a processor of the device in which the storage medium is located, to implement the parallel segmentation learning method based on knowledge distillation as described in any of the above claims.

[0022] In summary, in distributed deep learning, data between clients typically exhibits non-independent and identically distributed characteristics, meaning that the data sources and distributions of different clients differ significantly (e.g., some clients only contain data from certain categories). This disrupts the consistency of the global optimization objective, easily leading to gradient shifts and model update oscillations, ultimately resulting in decreased global model performance and limited generalization ability. In this situation, the optimization directions of each client's local model often conflict, and the update results obtained from server-side aggregation fail to accurately reflect the overall data distribution characteristics. The model training process exhibits slow convergence, instability, and even getting stuck in suboptimal solutions, especially when there are many clients or strong data heterogeneity. Existing technologies often lack privacy protection and stability. To address this problem, this invention proposes a parallel split learning method based on knowledge distillation, aiming to solve the significant performance degradation caused by the lack of client model aggregation in current parallel split learning methods when dealing with data heterogeneity. This method significantly improves model performance without requiring additional information, thus achieving efficient and stable distributed deep learning.

[0023] Specifically, this invention provides a parallel segmentation learning framework based on knowledge distillation to address the performance degradation and training instability issues caused by the non-independent and identically distributed nature of client data in distributed environments. This framework employs three synergistic modules to form a closed-loop training mechanism: a cross-layer projection distillation module uses deep features containing global semantics on the server side as teacher signals to guide shallow features on the client side to align with global features, effectively suppressing local overfitting and feature drift; an EMA prototype guidance module maintains and smoothly updates the global class prototype pool on the server side using an exponential moving average strategy, providing a noise-resistant and stable distillation target for heterogeneous clients, significantly mitigating class center shift and model oscillation caused by data heterogeneity; and a minimum frequency threshold constraint module audits the frequency of client participation in training based on long-term statistics and enforces a minimum participation threshold, ensuring that long-tail clients with sparse data distribution receive sufficient training opportunities, thereby preventing the permanent forgetting of specific categories of data.

[0024] This invention achieves efficient, stable, and fair collaborative training of non-independent and identically distributed data without relying on prior information about the client's data distribution or introducing significant communication and computational overhead. It significantly improves the convergence speed, generalization performance, and system robustness of the global model, and is particularly suitable for large-scale distributed learning scenarios with limited resources, such as the Internet of Things. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the parallel segmentation learning method based on knowledge distillation provided in the first embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram of the framework of the parallel segmentation learning method based on knowledge distillation provided in the first embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of the module of the parallel segmentation learning device based on knowledge distillation provided in the second embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0029] refer to Figure 1 , Figure 2 As shown, the first embodiment of the present invention discloses a parallel segmentation learning method based on knowledge distillation, which can be executed by a parallel segmentation learning device based on knowledge distillation (hereinafter referred to as the learning device), specifically, by one or more processors within the learning device, to implement the following method:

[0030] S1, compare the number of training parameters for each client with the preset minimum frequency threshold to generate the client set for this round of training;

[0031] Specifically, step S1 further includes: obtaining the number of training participations recorded for each client from a preset sliding time window, and sequentially determining whether the number of training participations for each client is less than a preset minimum frequency threshold. Generate the judgment result;

[0032] Clients whose number of training participations is less than a preset minimum frequency threshold are marked as forced priority, while clients whose number of training participations is not less than the preset minimum frequency threshold are marked as loss preference. Forced priority has a higher priority than loss preference.

[0033] Based on the judgment results, the selected clients for this round of training are identified, resulting in the client set for this round of training. The formula is as follows: , This is the set of clients selected in this round (i.e., the final result). For all clients, This means selecting the C clients with the largest values ​​(in parentheses) from all clients K. Let k be the classification loss for client k. For the number of training participations of client k, Represents the highest priority (infinity), ensuring that clients with insufficient counts are prioritized before any loss value.

[0034] In this embodiment, the present invention proposes a PSL-DEF framework (Parallel Split Learning with Distillation, EMA, and Fairness), which consists of three core components: Cross-Layer Projection Distillation (CLPD), EMA Prototype Guidance (EPG), and Minimum Frequency Threshold Constraint (MFTC). These three parts form a collaborative closed loop in each training round: CLPD is used to construct directly connected paths for edge-cloud feature mapping, overcoming gradient lag in deep networks; EPG, based on this, utilizes historical momentum to smooth class center drift, providing a robust distillation objective for heterogeneous data; and MFTC audits and enforces compensation for the participation opportunities of long-tail nodes at the global scheduling level, thereby achieving efficient and robust distributed training.

[0035] Specifically, due to limited communication resources, not all clients can be trained simultaneously. Therefore, before each training session, the server calculates the number of times each client has participated in training and compares it to a preset minimum frequency threshold. For clients whose cumulative participation count is below this threshold, the system triggers a forced training mechanism, directly assigning them the highest priority and allowing them to skip the regular competitive selection process and be directly selected for the current training set. For clients that have already met the frequency requirement, selection continues based on their classification loss. This long-term statistical-based guarantee logic ensures that all clients meet the most basic participation requirements, effectively preventing the forgetting of specific data distributions due to prolonged non-selection.

[0036] In this embodiment, before each round of training, a fair scheduling strategy based on Minimum Frequency Threshold (MFTC) is executed; a batch of clients are selected from the client set to participate in local updates. Client selection follows a hierarchical priority strategy, where mandatory priority is higher than loss priority. The cumulative training participation count of each client in completed rounds is obtained from a historical record of a preset sliding time window of length c (e.g., the most recent N rounds of training). Subsequently, the server compares the cumulative participation count of each client with a preset minimum frequency threshold. Based on the comparison result, the server divides all clients into two categories: clients with a cumulative participation count less than the minimum frequency threshold are marked as mandatory priority, i.e., given the highest priority, and prioritized in the client selection process to ensure long-term participation fairness; clients with a cumulative participation count that has reached or exceeded the threshold are marked as loss-preferred. This step ensures that long-tail clients with insufficient long-term participation can obtain mandatory training opportunities, thereby preventing specific data distributions from being forgotten due to scheduling bias, significantly improving the fairness of model training and the ability to cover the features of the entire data domain. Specifically, when the training count of all clients in the current window is less than... (For example, in the early stages of training, when each client participates 0 times), all clients are considered to have the same highest priority. At this time, a batch of clients are randomly selected to participate in this round of training. After the mandatory priority constraint is satisfied, the remaining clients are then sorted and selected according to performance indicators such as training loss.

[0037] Next, the server generates the client set for this round of training according to the following rules and formulas. First, all clients with mandatory priority are directly included in this round's set. Second, if the total number of clients planned to be selected in this round is fixed at C, and the number of mandatory priority clients is less than C, the server will sort the loss-optimized client group according to the classification loss calculated by their current local model on their own data, and prioritize the addition of several clients with larger classification losses to focus on the sample distribution areas where the current model performs poorly. The mathematical expression of this selection process can be summarized as follows: the final selected client set S consists of the top C clients with the largest comprehensive selection function value among all client sets K. This function integrates participation penalty and classification utility, and its specific form is as defined above. Through this dynamic scheduling mechanism that integrates long-term fairness and short-term utility, the system can fundamentally ensure balanced learning of class information for all clients in a heterogeneous data environment while ensuring training efficiency, effectively enhancing the robustness and generalization performance of the global model.

[0038] S2, take the deep features containing global semantic information on the server as teacher features, obtain the relevant data corresponding to the current task issued by the server, and perform parameter update processing on the clients in the client set based on the teacher features and relevant data to obtain the updated client parameters, and send the updated client parameters to the server.

[0039] Specifically, step S2 further includes: taking the deep features containing global semantic information on the server as teacher features, and introducing a feature projection layer in the client to make the dimensions of the last convolutional feature in the client consistent with those of the teacher features;

[0040] Based on the relevant data corresponding to the current task issued by the server, including the feature prototype retrieved from the EMA prototype library by the server based on the label y of the current data. and the main gradient from the task ;

[0041] Obtaining the client's basic characteristics and the basic features Input the feature projection layer to obtain the feature prototype Features with consistent dimensions , For the characteristic projection function, These are the learnable parameters for the feature projection layer;

[0042] The characteristic is calculated using the mean square error (MSE). With feature prototype Losses between And based on the loss Obtain its corresponding distillation gradient ;

[0043] According to the principal gradient and distillation gradient The parameters on the client are updated using the following formula: , For client parameters, As weight, The learning rate;

[0044] Repeat the above steps until the parameters of each client in the client set have been updated.

[0045] In this embodiment, during parallel segmentation learning, the model performance degrades significantly under non-iid data distributions due to the lack of client-side aggregation. Since the server-side model possesses global knowledge in PSL, the CLPD module uses the server-side deep features containing global semantic information as teacher features. On the client side, a feature projection layer is introduced to ensure that the final convolutional feature layer's dimension matches the teacher's feature dimension, forcing the client's shallow features to align with the server's global features. This design leverages the server's global perspective to provide strong auxiliary supervision signals to each heterogeneous client, enabling the client to perceive the feature boundaries of the global data distribution during local training. This effectively suppresses local overfitting and feature drift caused by the lack of model aggregation.

[0046] Specifically, the server, as the holder of global knowledge, possesses deep network output features containing global semantic information learned from all client data. In this step, the server uses these deep features as teacher features. Specifically, for each client k participating in the current training round, the server retrieves the corresponding category feature prototype from its maintained EMA prototype library based on the label y of the client's current batch of training data. This feature prototype is obtained through smooth updates over multiple training rounds and possesses characteristics of being robust against noise and transient data distribution shifts, thus serving as a robust and unified global supervision signal. Simultaneously, after its own forward propagation, the server calculates the task loss and generates the task master gradient that needs to be sent back to the client. The server then sends the retrieved feature prototype and the calculated task master gradient to the corresponding client k. This process transforms the server's global perspective and computational capabilities into distributable supervision information, providing crucial external guidance to heterogeneous clients and preventing their local training from becoming isolated.

[0047] Following this, after receiving the feature prototype from the server, the client begins the local feature alignment process. The client first processes the input data using its local shallow network to obtain basic features. To achieve effective comparison and knowledge transfer with the teacher features sent from the server, the client introduces a local learnable feature projection layer. This projection layer maps the dimensions of the client's locally extracted basic features to a dimensional space consistent with the feature prototype sent from the server, resulting in projected features. This design cleverly resolves potential differences in semantic level and dimensionality between the client's shallow features and the server's deep features, constructing a direct path from the client's local features to the server's global semantics.

[0048] Subsequently, the client performs knowledge distillation based on feature space alignment. It uses mean squared error (MSE) as a metric to calculate the difference between the local projected features and the teacher feature prototypes provided by the server, forming the distillation loss. This loss function forces the client's local feature representation to converge towards the robust prototypes provided by the server, representing the global class center. The client then calculates the corresponding gradient based on this distillation loss, i.e., the distillation gradient. By minimizing this loss, the client indirectly learns the semantic feature boundaries of the global data distribution during local optimization. This effectively suppresses local overfitting and feature drift phenomena that are prone to occur due to the non-independent and identically distributed nature of the client's data, enhancing the client model's ability to extract more generalizable features from its own skewed data.

[0049] The client's final parameter updates are driven by two supervisory signals: one is the main gradient from the server task feedback, which ensures the client model optimizes in the direction of reducing task-specific errors; the other is the distilled gradient from local feature alignment, which guides the client features to align to the global semantic space. The client weights and fuses these two gradients and updates its local parameters, including shallow network parameters and projection layer parameters, using the learning rate. Specifically, the update formula includes... This is a pre-defined weighting coefficient used to balance the importance of task learning and feature distillation. This update mechanism creates a highly efficient synergy: the task gradient ensures the model's discriminative ability, while the distillation gradient acts as a regularization term, guaranteeing semantic consistency across different clients at the feature level. This allows, within the parallel segmentation learning framework, even without direct client-side model aggregation, for the optimization directions of each client to remain coordinated under the guidance of global features, thereby significantly improving the convergence speed of the global model and the stability of its final performance.

[0050] Finally, the above steps are repeated for each client in the client set until all selected clients have completed this round of knowledge distillation-based parameter updates. After each client completes its update, it uploads the necessary information it has calculated (such as gradients or intermediate features) back to the server, allowing the server to update its deep model parameters and EMA prototype library. Through the above specific implementation, the framework described in this invention deeply embeds knowledge distillation into the training loop of parallel segmentation learning. This method not only eliminates the need for clients to upload raw data or expose their local data distribution, thus protecting user privacy, but also effectively alleviates the model divergence problem caused by data heterogeneity with low computational and communication overhead by introducing lightweight local projection and distillation loss calculation, achieving efficient and robust model training in distributed, non-independent, and identically distributed data scenarios.

[0051] S3. Based on the updated client parameters, determine the average feature value uploaded by each client for each category, and perform smooth update processing on the feature prototypes of the same category based on the average feature value. The updated feature prototypes are then used as the feature prototypes to be sent to the client in the next round of training.

[0052] Specifically, step S3 further includes: determining the average value of features uploaded by each category of clients based on the updated client parameters uploaded by all clients in the client set;

[0053] Based on the feature average, the exponential moving average (EMA) strategy is used to smoothly update the feature prototypes of consistent categories. The updated feature prototypes are then used as the feature prototypes sent to the client in the next round of training. The update formula is as follows: , The feature prototype (i.e., class prototype, represented as the global standard feature center of the y-th class (e.g., "cat") maintained by the server) is the feature prototype of the y-th class maintained by the server. The momentum coefficient is typically set to a value close to 1 (e.g., 0.9 or 0.99), indicating how "stubbornly" the model retains historical memory. The average value of features uploaded by clients belonging to category y (i.e., the average value of features in the current batch, the average value of features uploaded by all clients belonging to category y in this round of training).

[0054] S4. Repeat the above steps until all clients meet the preset training conditions.

[0055] In this embodiment, the EMA prototype guidance module is deployed on the server side to build a robust global feature reference system. This module maintains a feature prototype for each class (the output of the last convolutional layer on the server side) (only one feature prototype per class) and uses an exponential moving average (EMA) strategy to smoothly update the prototype. By utilizing historical prototypes to resist statistical noise and distribution drift caused by heterogeneous data (Non-IID) in a single iteration, EPG can generate high-confidence teacher signals and send them to the client. This ensures that even with extremely skewed local data distributions, the client can still perform feature alignment based on a stable and unified global standard, effectively avoiding model oscillations and catastrophic forgetting in distributed training.

[0056] Specifically, after all clients in the client set have completed their local parameter updates based on knowledge distillation, each client k uploads its deep feature information related to a specific category, calculated during this training round (specifically, features before local projection or features recalculated based on the updated model on local data), to the server. The server then classifies and aggregates this feature information according to the true label y attached to the client-uploaded data. For each category y in the data, the server collects the feature vectors uploaded by all clients labeled y and calculates the arithmetic mean of these feature vectors to obtain the feature average for that category. This aggregation process initially fuses similar data features distributed across different clients, potentially with local distribution skew, generating an instantaneous observation center reflecting the overall state of that type of data during this training round. This step is fundamental to the transition from local experience to global knowledge.

[0057] After obtaining the feature averages for each category, the server initiates the core update logic of the EMA Prototype Guidance (EPG) module. The server maintains a global feature prototype library, storing a corresponding feature prototype for each category y. This prototype is considered a robust central representation of that category's data in the global feature space. To update this prototype library, the server employs an exponential moving average strategy. The essence of the prototype update formula is to use historical prototypes as inertia to smooth the feature averages observed in the current round. A larger momentum coefficient value means greater reliance on historical information, resulting in a more conservative and smoother update; a smaller momentum coefficient value is more sensitive to new observations. This smoothing update mechanism effectively suppresses statistical noise and instantaneous class center drift caused by the randomness of client data sampling or extreme skewness in the distribution of individual client data (a typical manifestation of Non-IID) during a single training round. Even if data for a certain category in a particular round is provided only by a few clients with unrepresentative data distributions, and the calculated feature averages may deviate from the true global class center, the EMA strategy can largely offset this instantaneous bias thanks to the stability of historical prototypes, thus maintaining the long-term robustness of the feature prototypes. The updated feature prototypes will be distributed by the server to clients requiring class y supervision signals in the next training round. This ensures the consistency and reliability of the guidance signals, providing a solid foundation for continuous and effective cross-client knowledge distillation. The beneficial effects of this mechanism include significantly improved training stability, avoidance of model oscillations and divergences caused by data heterogeneity, and enhanced global model's ability to maintain balanced memory of features across classes, mitigating catastrophic forgetting.

[0058] Specifically, in this embodiment, the steps S1 (fair scheduling of clients), S2 (client knowledge distillation and update), and S3 (server-side prototype maintenance and update) constitute a complete training round. The system will repeat this cyclical process. At the beginning of each round, the server selects a new set of clients based on the updated Client Participation Record (MFTC) and model state; in each round, the clients use the more robust feature prototypes issued after the previous round's update to perform local feature alignment training; at the end of each round, the server further optimizes its feature prototype library using the EMA strategy based on the new feature information uploaded by the clients. This process continues until a preset training termination condition is met. The training condition may be reaching a preset maximum number of training rounds, the global model's performance on the validation set stabilizing (converging), or the participation of all clients being basically guaranteed through the minimum frequency threshold constraint mechanism, etc.

[0059] Through the above specific implementation methods, this invention constructs a self-reinforcing collaborative learning closed loop. Scheduling based on a minimum frequency threshold constraint ensures fairness in data participation; client updates based on knowledge distillation achieve efficient alignment of local features with global semantics; and prototype updates based on exponential moving averages ensure the noise resistance and stability of the global guidance signal. These three elements work together to enable the entire system to effectively address the severe challenges of non-independent and identically distributed data in distributed environments, without requiring centralized raw data, relying on strong prior assumptions, and with controllable communication and computational overhead. Ultimately, this results in training a high-performance global model with stronger generalization ability and robustness. This method overcomes the distribution shift challenges brought by Non-IID data while significantly improving the generalization ability and fairness of distributed training. This method is particularly suitable for large-scale distributed learning scenarios with high requirements for privacy, efficiency, and stability, such as the Internet of Things and mobile edge computing.

[0060] In summary, this invention, based on a parallel splitting learning framework, proposes a training method combining prototype distillation and long-term frequency-based fair scheduling for Non-Independent Identically Distributed (Non-IID) data scenarios. Its core principle lies in the collaborative operation of two modules: EMA Prototype Guidance (EPG) and Minimum Frequency Threshold Constraint (MFTC), to overcome model divergence and distribution bias caused by data heterogeneity. The EPG module constructs a noise-resistant global class prototype pool on the server side, smoothly updating prototype features using an exponential moving average (EMA) strategy. It leverages the inertia of historical prototypes to correct class center drift caused by skewed data distribution in single nodes, providing a unified and robust distillation objective for clients holding different classes of data. Subsequently, the MFTC module implements participation auditing based on long-term statistics, forcibly setting a minimum cumulative training frequency lower limit for nodes to ensure that long-tail nodes with unique data distributions receive sufficient gradient contribution opportunities. This method effectively mitigates local variance in Non-IID environments, avoids permanent forgetting of specific class information, and achieves efficient collaborative training of heterogeneous data.

[0061] Compared with existing technologies, this method has the following advantages: 1. More efficient feature distillation: By implementing cross-layer projection distillation on the client side and utilizing the global teacher signal issued by the server for local strong supervision, it effectively solves the gradient lag and local overfitting problems caused by the lack of model aggregation in segmentation learning, enabling shallow client networks to accurately capture global semantic features without uploading raw data. 2. More significant class center drift suppression: The prototype guidance strategy based on EMA uses historical features to smooth the statistical noise caused by local data heterogeneity in a single iteration, constructing a robust and unified feature reference system for all clients, effectively preventing model oscillation and divergence in extreme Non-IID environments, and ensuring the stability of the training objective. 3. Fair client selection: By introducing a minimum frequency threshold constraint mechanism, a participation frequency and forced selection strategy based on long-term statistics are established, breaking the traditional scheduling bias based solely on utility, ensuring that long-tail clients with sparse data can obtain sufficient gradient contribution opportunities, fundamentally avoiding catastrophic forgetting of specific categories of data. 4. Enhanced Global Convergence and Robustness: The three modules form a collaborative closed loop of "feature alignment - prototype calibration - fair scheduling". This can improve the quality of single-point features at the micro level through local distillation, and ensure unbiased coverage of the entire data at the macro level through forced scheduling. This significantly enhances the convergence speed and final generalization performance of the model in complex heterogeneous environments.

[0062] Please see Figure 3A second embodiment of the present invention provides a parallel segmentation learning device based on knowledge distillation, comprising:

[0063] The client confirmation unit 101 is used to compare the number of training parameters of each client with the preset minimum frequency threshold to generate the client set for this round of training.

[0064] The parameter update unit 102 is used to take the deep features containing global semantic information on the server as teacher features, obtain the relevant data corresponding to the current task issued by the server, and perform parameter update processing on the clients in the client set based on the teacher features and relevant data to obtain the updated client parameters, and then send the updated client parameters to the server.

[0065] The prototype update unit 103 is used to determine the average feature value uploaded by each category client based on the updated client parameters, and to perform smooth update processing on the feature prototypes of the same category based on the average feature value, and to send the updated feature prototypes to the client in the next round of training.

[0066] Iteration unit 104 is used to repeat the above steps until all clients have met the preset training conditions.

[0067] Preferably, the client confirmation unit 101 is specifically used to: obtain the number of times each client has participated in the training completed from a preset sliding time window, and sequentially determine whether the number of training participations of each client is less than a preset minimum frequency threshold. Generate the judgment result;

[0068] Clients whose number of training participations is less than a preset minimum frequency threshold are marked as forced priority, while clients whose number of training participations is not less than the preset minimum frequency threshold are marked as loss preference. Forced priority has a higher priority than loss preference.

[0069] Based on the judgment results, the selected clients for this round of training are identified, resulting in the client set for this round of training. The formula is as follows: , The set of clients selected in this round. For all clients, This means selecting the C clients with the largest values ​​(in parentheses) from all clients K. Let k be the classification loss for client k. This represents the number of training participations for client k.

[0070] Preferably, the parameter update unit 102 is specifically used to: take the deep features containing global semantic information on the server as teacher features, introduce a feature projection layer in the client, so that the dimensions of the last convolutional feature in the client are consistent with those of the teacher features;

[0071] Based on the relevant data corresponding to the current task issued by the server, including the feature prototype retrieved from the EMA prototype library by the server based on the label y of the current data. and the main gradient from the task ;

[0072] Obtaining the client's basic characteristics and the basic features Input the feature projection layer to obtain the feature prototype Features with consistent dimensions , For the characteristic projection function, These are the learnable parameters for the feature projection layer;

[0073] The characteristic is calculated using the mean square error (MSE). With feature prototype Losses between And based on the loss Obtain its corresponding distillation gradient ;

[0074] According to the principal gradient and distillation gradient The parameters on the client are updated using the following formula: , For client parameters, As weight, The learning rate;

[0075] Repeat the above steps until the parameters of each client in the client set have been updated.

[0076] Preferably, the prototype update unit 103 is specifically used to: determine the average value of features uploaded by each category of clients based on the updated client parameters uploaded by all clients in the client set;

[0077] Based on the feature average, the exponential moving average (EMA) strategy is used to smoothly update the feature prototypes of consistent categories. The updated feature prototypes are then used as the feature prototypes sent to the client in the next round of training. The update formula is as follows: , The feature prototype of class y maintained on the server side. The momentum coefficient, The average value of features uploaded by the client for category y.

[0078] A third embodiment of the present invention provides a parallel segmentation learning device based on knowledge distillation, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the parallel segmentation learning method based on knowledge distillation as described above.

[0079] A fourth embodiment of the present invention provides a readable storage medium storing a computer program that can be executed by a processor of the device in which the storage medium is located, to implement the parallel segmentation learning method based on knowledge distillation as described in any of the above embodiments.

[0080] For example, the various devices and process steps described above can be implemented by a computer program, which can be divided into one or more units, which are stored in the memory and executed by the processor to complete the present invention.

[0081] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0082] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the present invention by running or executing the computer programs and / or modules stored in the memory and by calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0083] If the unit integrated into the electronic device or printer is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0084] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0085] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A parallel segmentation learning method based on knowledge distillation, characterized in that, include: The number of training parameters for each client is compared with the preset minimum frequency threshold to generate the client set for this round of training. The deep features containing global semantic information on the server side are used as teacher features. Relevant data corresponding to the current task is obtained from the server. Based on the teacher features and relevant data, the parameters of the clients in the client set are updated to obtain the updated client parameters. These updated client parameters are then sent to the server. Specifically: The deep features containing global semantic information on the server side are used as teacher features, and a feature projection layer is introduced in the client side to make the dimensions of the last convolutional feature in the client side consistent with those of the teacher features. Based on the relevant data corresponding to the current task issued by the server, including the feature prototype retrieved from the EMA prototype library by the server according to the label y of the current data. and the main gradient from the task ; Obtaining the client's basic characteristics and the basic features Input the feature projection layer to obtain the feature prototype Features with consistent dimensions , For the characteristic projection function, These are the learnable parameters for the feature projection layer; The characteristic is calculated using the mean square error (MSE). With feature prototype Losses between And based on the loss Obtain its corresponding distillation gradient ; According to the principal gradient and distillation gradient The parameters on the client are updated using the following formula: , For client parameters, As weight, The learning rate; Repeat the above steps until the parameters of each client in the client set have been updated; Based on the updated client parameters, determine the average feature value uploaded by each client for each category, and perform smooth updates on the feature prototypes of the same category based on the average feature value. The updated feature prototypes will then be used as the feature prototypes to be sent to the client in the next round of training. Repeat the above steps until all clients meet the preset training conditions.

2. The parallel segmentation learning method based on knowledge distillation according to claim 1, characterized in that, The number of training parameters for each client is compared with a preset minimum frequency threshold to generate the client set for this round of training, specifically: The system retrieves the number of training sessions completed by each client within a preset sliding time window, and then sequentially determines whether the number of training sessions completed by each client is less than a preset minimum frequency threshold. Generate the judgment result; Clients whose number of training participations is less than a preset minimum frequency threshold are marked as forced priority, while clients whose number of training participations is not less than the preset minimum frequency threshold are marked as loss preference. Forced priority has a higher priority than loss preference. Based on the judgment results, the selected clients for this round of training are identified, resulting in the client set for this round of training. The formula is as follows: , The set of clients selected in this round. For all clients, This means selecting the C clients with the largest values ​​(in parentheses) from all clients K. Let k be the classification loss for client k. This represents the number of training participations for client k.

3. The parallel segmentation learning method based on knowledge distillation according to claim 1, characterized in that, Based on the updated client parameters, determine the average feature value uploaded by the client for each category, and perform a smooth update on the feature prototypes of the same category based on the feature average value. The updated feature prototypes will then be used as the feature prototypes to be sent to the client in the next round of training. Specifically: Based on the updated client parameters uploaded by all clients in the client set, determine the average feature value uploaded by each category of clients; Based on the feature average, the exponential moving average (EMA) strategy is used to smoothly update the feature prototypes of consistent categories. The updated feature prototypes are then used as the feature prototypes sent to the client in the next round of training. The update formula is as follows: , The feature prototype of class y maintained on the server side. The momentum coefficient, The average value of features uploaded by the client for category y.

4. A parallel segmentation learning device based on knowledge distillation, characterized in that, include: The client confirmation unit is used to compare the number of training parameters for each client with the preset minimum frequency threshold, and generate the client set for this round of training. The parameter update unit is used to take the deep features containing global semantic information on the server as teacher features, obtain relevant data corresponding to the current task issued by the server, and perform parameter update processing on the clients in the client set based on the teacher features and relevant data to obtain the updated client parameters, and then send the updated client parameters to the server; the parameter update unit is specifically used for: The deep features containing global semantic information on the server side are used as teacher features, and a feature projection layer is introduced in the client side to make the dimensions of the last convolutional feature in the client side consistent with those of the teacher features. Based on the relevant data corresponding to the current task issued by the server, including the feature prototype retrieved from the EMA prototype library by the server according to the label y of the current data. and the main gradient from the task ; Obtaining the client's basic characteristics and the basic features Input the feature projection layer to obtain the feature prototype Features with consistent dimensions , For the characteristic projection function, These are the learnable parameters for the feature projection layer; The characteristic is calculated using the mean square error (MSE). With feature prototype Losses between And based on the loss Obtain its corresponding distillation gradient ; According to the principal gradient and distillation gradient The parameters on the client are updated using the following formula: , For client parameters, As weight, The learning rate; Repeat the above steps until the parameters of each client in the client set have been updated; The prototype update unit is used to determine the average feature value uploaded by each category client based on the updated client parameters, and to perform smooth update processing on the feature prototypes of the same category based on the feature average value. The updated feature prototypes are then sent to the client as the feature prototypes for the next round of training. The iterative unit is used to repeat the above steps until all clients have met the preset training conditions.

5. The parallel segmentation learning device based on knowledge distillation according to claim 4, characterized in that, The client confirmation unit is specifically used for: The system retrieves the number of training sessions completed by each client within a preset sliding time window, and then sequentially determines whether the number of training sessions completed by each client is less than a preset minimum frequency threshold. Generate the judgment result; Clients whose number of training participations is less than a preset minimum frequency threshold are marked as forced priority, while clients whose number of training participations is not less than the preset minimum frequency threshold are marked as loss preference. Forced priority has a higher priority than loss preference. Based on the judgment results, the selected clients for this round of training are identified, resulting in the client set for this round of training. The formula is as follows: , The set of clients selected in this round. For all clients, This means selecting the C clients with the largest values ​​(in parentheses) from all clients K. Let k be the classification loss for client k. This represents the number of training participations for client k.

6. The parallel segmentation learning device based on knowledge distillation according to claim 4, characterized in that, The prototype update unit is specifically used for: Based on the updated client parameters uploaded by all clients in the client set, determine the average feature value uploaded by each category of clients; Based on the feature average, the exponential moving average (EMA) strategy is used to smoothly update the feature prototypes of consistent categories. The updated feature prototypes are then used as the feature prototypes sent to the client in the next round of training. The update formula is as follows: , The feature prototype of class y maintained on the server side. The momentum coefficient, The average value of features uploaded by the client for category y.

7. A parallel segmentation learning device based on knowledge distillation, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the parallel segmentation learning method based on knowledge distillation as described in any one of claims 1 to 3.

8. A readable storage medium, characterized in that, The storage medium contains a computer program that can be executed by a processor of the device in which the storage medium resides, to implement the parallel segmentation learning method based on knowledge distillation as described in any one of claims 1 to 3.

Images