A knowledge transfer method based on conditional cascade distillation

Abstract

The present invention discloses a knowledge transfer method based on conditional cascade distillation, wherein a cloud server constructs a model and sends it to several terminal devices; each terminal device collects training data and inputs it into the sent model for iterative training to obtain a trained terminal device model, and the terminal device updates and uploads the trained terminal device model to the cloud server; cascade distillation allows each terminal device model to absorb the knowledge of the cloud model; the cloud server receives several terminal device model updates and aggregates and optimizes communication through a weighted aggregation algorithm, and sends the optimized global model to the terminal device; the terminal device collects data to be detected and inputs it into the optimized global model to obtain a detection result. The present invention not only improves the generalization ability of the model, but also significantly reduces the communication cost in the process of model updating, and realizes the generation of a global model with better performance through collaborative multi-party training under the premise of protecting the privacy of terminal device data.

Description

Technical Field

[0001] The present invention relates to the field of end-cloud collaborative technology, and in particular to a knowledge migration method based on conditional cascade distillation. Background Art

[0002] In an end-to-end collaborative system, the end device is usually responsible for processing local data and uploading model updates to the cloud for aggregation to improve the performance of the overall model. Traditional end-to-end collaborative learning methods rely on the aggregation of model updates generated by the end device and the cloud model, but these methods have significant defects when processing partially annotated data. For example, the end device may encounter incompletely annotated data during training. In this case, directly uploading model updates to the cloud may cause the performance of the new model to degrade because the model may not be able to learn effective knowledge from the incomplete data.

[0003] In addition, traditional end-cloud collaborative learning methods are not adaptable enough in new scenarios. Since the data distribution of the end device may be different from the data distribution trained by the cloud model, directly applying the knowledge distillation technology of the cloud model may cause confusion in the feature representation of the end device model, thereby affecting the generalization ability of the model. At the same time, frequent model update uploads and aggregation operations cause high communication overhead, which is particularly prominent when the computing resources and communication bandwidth of the end device are limited.

[0004] Previous studies have attempted to solve the problems in end-cloud collaborative learning through knowledge distillation technology. Some studies have proposed using a global model as a teacher model to transfer knowledge to the local model through knowledge distillation. However, these methods often ignore the particularity of end device data and fail to effectively process some labeled data, resulting in information loss during the knowledge transfer process. Furthermore, existing end-cloud collaborative learning methods often use simple averaging or weighted aggregation strategies when processing data from multiple clients, which may lead to uneven model performance. Because the data distribution and quality of different clients may vary significantly, simple aggregation strategies cannot fully consider these differences, thus affecting the performance of the final model.

[0005] In summary, the existing technologies have the following major problems and defects when processing partial labeled data in the end-cloud collaborative architecture: First, the traditional end-cloud collaborative learning method fails to effectively utilize partial labeled data, resulting in insufficient model generalization ability; second, the communication cost between the end device and the cloud is high, especially when the end device resources are limited; finally, the existing aggregation strategy fails to fully consider the diversity and differences of client data, affecting the balance of model performance. Therefore, there is an urgent need for an effective knowledge transfer method applied to the end-cloud collaborative architecture to solve the problem of how to effectively utilize partial labeled data in the end-cloud collaborative architecture to improve the model generalization ability and reduce the communication cost. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a knowledge transfer method based on conditional cascade distillation. By introducing conditional probability representation and cascade distillation loss function, it can effectively process partially labeled data and realize efficient knowledge transfer between terminal devices and the cloud. It also takes into account the diversity and differences of client data, and dynamically adjusts the knowledge distillation strategy to achieve collaborative multi-party training to generate a global model with better performance while protecting the privacy of terminal device data. This model is suitable for intelligent system application scenarios such as autonomous driving, intelligent manufacturing, smart security and medical image analysis.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows:

[0008] A knowledge transfer method based on conditional cascade distillation is applied to a terminal-cloud collaborative system, wherein the terminal-cloud collaborative system includes a cloud server and a plurality of terminal devices connected to the cloud server, and includes the following steps:

[0009] The cloud server constructs the model and sends it to the plurality of terminal devices;

[0010] Each of the terminal devices collects training data and inputs it into the issued model for iterative training to obtain a trained terminal device model, and the terminal device updates and uploads the trained terminal device model to the cloud server;

[0011] The cloud server uses cascade distillation to allow each end device model to absorb the knowledge of the cloud model. In the cascade distillation process, the cloud model serves as the teacher model and the end device model serves as the student model. The cloud server receives updates from several end device models and performs aggregation and communication optimization through a weighted aggregation algorithm, and sends the optimized global model to the end device.

[0012] The terminal device collects the data to be tested and inputs it into the optimized global model to obtain the test results.

[0013] Preferably, during the iterative training of the model sent by the terminal device, the feature extraction process is optimized by conditional probability representation and cascade distillation loss function, which specifically includes the following steps:

[0014] A dataset is constructed based on the training data, and the probability of known categories in the dataset is regarded as conditions, and the probability of unknown categories is regarded as unlabeled; for each category C i , calculate the conditional probability P(C i |x), that is, given input x, category C i The probability of occurrence can be expressed as:

[0015]

[0016] in, represents a set of known categories, represents the set of unknown categories, p F ore and p B ack represents the probability distribution of foreground and background respectively;

[0017] Defining marginal sets It contains all unknown categories and combines the probabilities of background categories into a single probability value p B , the original probability value of the foreground class Keeping it unchanged, the marginal loss function has the following form:

[0018]

[0019] Among them, y i is a binary indicator, when category C i The value is 1 if it is a correct label, otherwise it is 0;

[0020] In each training iteration, the model's predicted probability for each category is calculated, and the model's parameters are updated according to the marginal loss function, expressed as:

[0021]

[0022] Among them, θ represents the parameters of the model, η is the learning rate, Represents the gradient of the marginal loss function with respect to the model parameters.

[0023] Preferably, a feature calibration network is preset in the terminal device, and the feature calibration network is based on self-supervised learning to capture the feature mapping relationship between the terminal device model and the cloud model, and calibrates the feature expression according to the feature mapping relationship between the terminal device model and the cloud model, specifically including the following steps:

[0024] The feature mapping process is expressed by the following formula:

[0025] f enc (x;θ enc )=φ(x) (4)

[0026] f dec (z;θ dec )=ψ(z) (5)

[0027] Among them, x represents the input data, z is the feature expression after mapping, and θ enc and θ dec Represent the parameters of the encoder and decoder respectively, φ and ψ are the corresponding mapping functions; when training the feature calibration network, the feature calibration loss function is introduced to quantify the encoder f enc Output and decoder f encThe difference between the outputs, the feature calibration loss function can be expressed as:

[0028]

[0029] The end-to-end training strategy is implemented through joint optimization, the formula is as follows:

[0030]

[0031] Among them, θ model Represents the parameters of the end device model or cloud model, is the loss function of the main task, and λ is the weight of the feature calibration loss, which is used to balance the main task loss and the feature calibration loss.

[0032] Preferably, the cloud server enables each end device model to absorb the knowledge of the cloud model through cascade distillation, which specifically includes the following steps:

[0033] Designing Cascade Distillation Loss Function Focus on the prediction output of the end device model and the prediction output of the cloud model And the knowledge transfer is achieved by minimizing the difference between the two, which is expressed as follows:

[0034]

[0035] Among them, D k represents the data set of end device k, is the distance metric function;

[0036] The parameter update of the end device model adopts the following formula:

[0037]

[0038]

[0039] Among them, v k,t is the momentum term for parameter update, μ is the momentum coefficient, which is used to accumulate past gradient directions to form a more stable update direction. The end device k is based on its local data set D k Perform model training, θ k,t is the model parameter of the tth iteration, η k is the learning rate; θ l,t+1 is the aggregated local model parameter, w k It is a weight that is dynamically adjusted according to the local performance of the end device k.

[0040] Preferably, the cloud server receives several end device model updates and performs aggregation and communication optimization through a weighted aggregation algorithm, and sends the optimized global model to the end device, specifically including the following steps:

[0041] Regularization terms are introduced to prevent overfitting and ensure the smoothness of model updates. The aggregation strategy can be expressed as:

[0042]

[0043] Among them, θ g,t+1 is the global model parameter after aggregation, L(D k ,θ g,t ) is the loss function of the end device k, w′ k is the weight of the end device k, λ k is a regularization parameter used to balance the local loss of the end device and the smoothness of the model update;

[0044] When the model update in the end device exceeds the threshold, the end device model update is sent to the cloud server, and the data compression technology using gradient quantization is used, which can be expressed as:

[0045] θ k,comp =f qua (θ k -θ g ) (12)

[0046] Among them, f qua is a quantization function used to map continuous gradient values ​​to a finite discrete set, θ k,comp Updated for the compressed model;

[0047] The dynamic adjustment process of model update frequency and communication strategy can be expressed as:

[0048] α k,t+1 =α k,t ·f fre (Δ k,g ,r rec ) (13)

[0049] Among them, α k,t refers to the model update frequency of end device k at time step t, Δ k,g Represents the difference between the end device model parameters and the global model parameters, r rec Indicates the current communication resource status, function f fre (·,·) is used to adjust the update frequency according to the model difference and communication resource conditions. When the model difference exceeds the preset threshold, it tends to increase the update frequency, and when the communication resources are limited, it tends to adopt a more aggressive data compression rate. The model difference Δ k,gThe evaluation is achieved by the output difference between the representative data in the computing device and the cloud model, which is expressed as:

[0050]

[0051] in, and They represent the prediction functions of the end device and the global model respectively, KL represents the Kullback-Leibler divergence, is a representative dataset used for evaluation, and the data compression ratio C k is an adjustable parameter that can be adjusted based on the model update amount Δ of the end device k k,t Dynamic adjustment, adjustment C k This is equivalent to changing the quantization level or compression rate to control the communication cost C. k,t :

[0052] C k,t =g(C k ,Δ k,t ) (15)

[0053] Among them, C k,t represents the communication cost of end device k at time step t, g(·,·) is a cost function used to calculate the communication cost based on the data compression rate and the model update amount, Δ k,t is the model update amount of end device k at time step t.

[0054] Based on the above technical solution, the beneficial effects of the present invention are:

[0055] 1) Improve the generalization ability of the model: The conditional probability representation and marginal loss function of the present invention are optimized for partially annotated data, effectively avoiding knowledge conflicts caused by incomplete labels. Compared with the traditional end-cloud collaborative training method, experiments show that the method of the present invention can improve the average accuracy of the model on similar data sets. Experimental data show that on the medical imaging data set Medical Segmentation Decathlon, the method of the present invention can improve the average Dice coefficient by 2.1% on the same data set compared with the existing method iCaRL, proving the improvement of the generalization ability of the model;

[0056] 2) Optimize communication efficiency: This invention significantly reduces the frequency of updating model parameters between the end device and the cloud server through feature calibration network and cascade distillation loss function. In a simulated end-to-cloud environment, the data transmission volume of multiple rounds of model update process is accurately measured. Compared with the existing method BiC, this invention achieves a 10.6% reduction in data transmission volume, which greatly reduces the communication cost;

[0057] 3) Optimize computing resource allocation: The cloud model aggregation and communication optimization strategy of the present invention can dynamically adjust the update frequency according to the communication resource status of the end device and the urgency of the model update. Experimental data show that under bandwidth-constrained conditions, the method of the present invention can save 5.2% of bandwidth resources compared to the end-cloud collaboration method iCaRL, while maintaining a 2.1% improvement in model performance;

[0058] 4) Protecting data privacy: The present invention performs more calculations and knowledge extraction on the end device, reducing the transmission of data between the end device and the cloud, thereby reducing the risk of data leakage. In addition, encryption technology and anonymization processing are used to further ensure that sensitive information of the end device is not leaked during model update and aggregation;

[0059] 5) Improve system adaptability: The technical solution of the present invention enables the end-cloud collaborative learning system to better adapt to the dynamic changes in data distribution and fluctuations in the communication environment; through the introduction of a dynamic adjustment mechanism, the system can quickly respond to environmental changes and maintain or improve model performance. When faced with rapid changes in data distribution, the method of the present invention can quickly adapt to the new data distribution and maintain or even improve model performance, while traditional learning methods may require more time and resources to adapt to these changes;

[0060] 6) Reducing data annotation costs: The method of the present invention can learn more information from partially annotated data, which means that the amount of annotated data required can be greatly reduced when obtaining the same level of model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The figure is a flow chart of a knowledge transfer method based on conditional cascade distillation in an embodiment. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0063] like Figure 1 As shown, this embodiment provides a knowledge transfer method based on conditional cascade distillation, which is applied to a terminal cloud collaboration system. The terminal cloud collaboration system includes a cloud server and a plurality of terminal devices connected to the cloud server, including the following steps:

[0064] Step 1: The cloud server builds a model and sends it to several of the terminal devices.

[0065] In this embodiment, the knowledge transfer method of conditional cascade distillation is applied to the end-cloud collaborative system, and the joint segmentation task of multiple organs and tumors in medical image analysis is completed based on the end-cloud collaborative system.

[0066] The cloud server initializes a global model, which is usually composed of a pre-trained deep learning network, such as a convolutional neural network (CNN). After receiving this global model, the end device starts iterative training of the model.

[0067] In step 2, each of the terminal devices collects training data and inputs it into the sent model for iterative training to obtain a trained terminal device model. The terminal device updates and uploads the trained terminal device model to the cloud server.

[0068] In this embodiment, the terminal device collects training data to build a local data set, and uses the local data set to iteratively train the received model to adapt to specific data distribution and characteristics. During the iterative training process, the feature extraction process is optimized through conditional probability representation and marginal loss function, which specifically includes the following steps:

[0069] Conditional probability representation: For a given dataset, the probability of known categories is considered as conditional, while the probability of unknown categories is considered as unlabeled. By distinguishing these two probability distributions, the model can learn more effectively from limited labeled data. For each category C i , calculate the conditional probability P(C i |x), that is, given input x, category C i In the partially labeled scenario, the probabilities of all known categories are merged into a foreground probability distribution, and the probabilities of all unknown categories are merged into a background probability distribution, which can be expressed as:

[0070]

[0071] in, represents a set of known categories, represents the set of unknown categories, and p Fore and p Back Represent the probability distribution of foreground and background respectively.

[0072] Marginal loss function: used to solve the knowledge conflict problem caused by incomplete labels. Traditional loss functions, such as cross entropy loss, assume that the prediction probability of each category is independent. However, when dealing with partially labeled data, this assumption may cause the model's prediction of unknown categories to become unstable, thus affecting its generalization ability. To alleviate this problem, this function reduces the model's dependence on unlabeled categories by incorporating the probability of background categories. Specifically, first define a marginal set It contains all unknown categories. Then, the probabilities of the background categories are combined into a single probability value p B , and the original probability value of the foreground class Remain unchanged. The marginal loss function is as follows:

[0073]

[0074] Among them, y i is a binary indicator, when category C i The value is 1 if it is a correct label, otherwise it is 0.

[0075] Training process: Use limited labeled data on the end device to initialize a model; use conditional probability representation to adjust the model's predictions to make it pay more attention to known categories and reduce the uncertainty of predictions for unknown categories. In this way, the model can learn more effectively from limited labeled data and improve its generalization ability for unknown categories. Subsequently, the marginal loss function is applied to further optimize the model. In each training iteration, the model's predicted probability for each category is calculated, and the model's parameters are updated according to the marginal loss function. This process can be formalized as:

[0076]

[0077] Among them, θ represents the parameters of the model, η is the learning rate, and Represents the gradient of the marginal loss function with respect to the model parameters.

[0078] In this embodiment, a feature calibration network is preset in the terminal device. The feature calibration network is based on self-supervised learning to capture the feature mapping relationship between the terminal device model and the cloud model. The feature expression is calibrated according to the feature mapping relationship between the terminal device model and the cloud model. Specifically, the following steps are included:

[0079] Feature calibration network F fc Design: F fc Based on self-supervised learning, the feature mapping relationship between the end device model and the cloud model is accurately captured to achieve effective calibration of feature expression. Its structure includes encoder f enc and decoder f dec Two parts, encoder f enc Responsible for mapping the feature expression of the end device model to the new feature space, the decoder f dec Responsible for restoring the new feature expression to the original space. This bidirectional mapping mechanism makes F fc Learn a mapping function that minimizes the difference in feature expression. The feature mapping process is expressed by the following formula:

[0080] f enc (x;θ enc )=φ(x) (4)

[0081] f dec (z;θ dec )=ψ(z) (5)

[0082] Among them, x represents the input data, z is the feature expression after mapping, and θ enc and θ dec

[0083] Represent the parameters of the encoder and decoder respectively, φ and ψ are the corresponding mapping functions. fc

[0084] When , a feature calibration loss function is introduced to quantize the encoder f enc Output and decoder f enc The difference between the outputs, the feature calibration loss function can be expressed as:

[0085]

[0086] By minimizing this loss function, F fc It is possible to learn an effective feature mapping so that the feature expression of the end device model is closer to the feature expression of the cloud model.

[0087] Efficient end-to-end training strategy: In practical applications, F fc The training of is usually performed simultaneously with the training of the end-device model and the cloud model. This end-to-end training strategy is achieved through joint optimization:

[0088]

[0089] Among them, θ model Represents the parameters of the end device model or cloud model, is the loss function of the main task (such as the loss of classification or segmentation tasks), and λ is the weight of the feature calibration loss, which is used to balance the main task loss and the feature calibration loss.

[0090] The role of feature calibration: F fc By optimizing the mapping relationship of the feature space, the difference in feature expression between the end device model and the cloud model is significantly reduced, which will further improve the efficiency of knowledge distillation. This feature calibration process can be regarded as a preprocessing step, which provides more accurate and consistent feature expression for knowledge distillation, allowing subsequent knowledge distillation to proceed more smoothly.

[0091] Step 3: The cloud server uses cascade distillation to allow each end device model to absorb the knowledge of the cloud model. During the cascade distillation process, the cloud model serves as the teacher model and the end device model serves as the student model. The cloud server receives updates from several end device models and performs aggregation and communication optimization through a weighted aggregation algorithm, and sends the optimized global model to the end device.

[0092] In this embodiment, cascade distillation not only promotes the interaction between the end device model and the cloud model to improve the performance of the end device model, but also provides an effective update mechanism for the cloud model, so that it can better adapt to the data characteristics of the end device. By optimizing the prediction consistency of the end device model and the cloud model on unlabeled data, the system can make more full use of the knowledge generated by the end device and play an active role in reducing the communication cost during the model update process, thereby achieving effective knowledge transfer and co-evolution of the model. Specifically, it includes the following:

[0093] Design of cascade distillation: The design of the cascade distillation loss function is inspired by the principle of knowledge distillation. Its core is to use a complex teacher model (cloud model) to guide the learning process of a simplified student model (end device model). In the end-cloud collaborative learning environment, by minimizing the difference between the outputs of the two, the student model can gradually absorb the knowledge of the teacher model, especially when processing unlabeled data.

[0094] In order to quantify the consistency between the output of the end device model and the cloud model, a cascade distillation loss function is designed This function focuses on the prediction output of the end device model and the prediction output of the cloud model And the knowledge transfer is achieved by minimizing the difference between the two, which is expressed as follows:

[0095]

[0096] Among them, D k represents the data set of end device k, and is a distance metric function, such as KL divergence or cross entropy loss.

[0097] Adjustment of the end device model: After applying the cascade distillation loss function on the end device, the prediction output of the model needs to be optimized to better align with the prediction output of the cloud model. This optimization process is achieved through the momentum gradient descent algorithm, which can accelerate the learning process and reduce oscillations, so as to converge to the optimal solution faster. The parameter update of the end device model uses the following formula:

[0098]

[0099] Among them, v k,t is the momentum term for parameter update, and μ is the momentum coefficient, which is used to accumulate past gradient directions to form a more stable update direction. k Perform model training, θ k,t is the model parameter of the tth iteration, η k is the learning rate, is the cascade distillation loss function.

[0100] Aggregation of end-device models: On the end-device, a special aggregation strategy is adopted that not only considers the performance of the model, but also the computing power and communication resources of the end-device. This aggregation strategy integrates the updates of the end-device model through a weighted average method, where the weight w k It reflects the efficiency and accuracy of the end device when processing local data, which can be expressed by the following formula:

[0101]

[0102] Among them, θ l,t+1 is the aggregated local model parameter, w k It is a weight that is dynamically adjusted according to the local performance of the end device k. This weight allocation mechanism ensures that when resources are limited, the end devices with better performance can make a greater contribution to the aggregation model.

[0103] In this embodiment, cloud model aggregation and communication optimization are adopted, which are the key links to ensure performance and reduce communication overhead. It is responsible for integrating model updates from various end devices, and updating the global model on this basis, while optimizing the communication strategy to reduce the amount of data transmission. The specific instructions are as follows:

[0104] Cloud model aggregation strategy: The cloud server is responsible for aggregating model updates from all end devices and fusing these updates through an aggregation algorithm to form a new global model. Considering the differences in feature calibration and knowledge distillation between different end devices, it is necessary to introduce a weighted aggregation strategy to ensure that end devices with outstanding performance play a greater role in global model updates. The cloud server not only aggregates the updates of the end device models, but also introduces regularization terms to prevent overfitting and ensure the smoothness of model updates. This aggregation strategy can be expressed as:

[0105]

[0106] Among them, θ g,t+1 is the global model parameter after aggregation, L(D k ,θ g,t ) is the loss function of the end device k, w′ k is the weight of the end device k, λ k is a regularization parameter used to balance the local loss of the end device and the smoothness of the model update. Note that in the aggregation of the end device model, w k represents the weight of the efficiency and accuracy of the end device when processing local data. In the cloud model aggregation strategy, w′ kIt is represented as the cloud model weight, which is used to balance the contribution of different end devices to the global model update. Both are part of the aggregation strategy, but are used for end devices and cloud servers respectively. The former is used to weighted average the model updates on the end devices to form a new local model parameter. The cloud model aggregation strategy is to perform weighted aggregation on the cloud server, and consider regularization to form the global model parameters.

[0107] Communication cost optimization: To reduce the communication overhead between the end device and the cloud server, communication compression and sparse update strategies are further adopted. Specifically, the end device only sends the update to the cloud when the model update exceeds a certain threshold; or only sends a sparse representation of the model update. In addition, the data compression technology of gradient quantization is used to effectively reduce the amount of data transmitted. The mathematical expression of communication compression is:

[0108] θ k,comp =f qua (θ k -θ g ) (12)

[0109] Among them, f qua is a quantization function used to map continuous gradient values ​​to a finite discrete set, θ k,comp Updated for compressed models.

[0110] Dynamic adjustment of model updates and communication strategies: The end-to-end collaborative process is that the end device is responsible for performing local training and model updates, while the cloud server is responsible for aggregating these updates and optimizing the global model. This collaborative working method can fully utilize the computing resources of the end device and the aggregation capabilities of the cloud, while reducing communication overhead. The dynamic adjustment process of the model update frequency and communication strategy can be expressed as:

[0111] α k,t+1 =α k,t ·f fre (Δ k,g ,r rec ) (13)

[0112] Among them, α k,t Refers to the model update frequency of end device k at time step t. Δ k,g represents the difference between the end device model parameters and the global model parameters, and r rec Indicates the current communication resource status. Function f fre (·,·) is used to adjust the update frequency according to the model difference and communication resource conditions. It tends to increase the update frequency when the model difference exceeds the preset threshold, and tends to adopt a more aggressive data compression rate when communication resources are limited. Model difference Δ k,gThe evaluation is achieved by computing the output difference between the representative data in the end device and the global model, using the Kullback-Leibler divergence as the metric, expressed as:

[0113]

[0114] in, and They represent the prediction functions of the end device and the global model respectively, KL represents the Kullback-Leibler divergence, is a representative dataset used for evaluation. Data compression ratio C k is an adjustable parameter that can be adjusted based on the model update amount Δ of the end device k k,t Dynamic adjustment. Adjust C k This is equivalent to changing the quantization level or compression rate to control the communication cost C. k,t :

[0115] C k,t =g(C k ,Δ k,t ) (15)

[0116] Among them, C k,t represents the communication cost of end device k at time step t, g(·,·) is a cost function used to calculate the communication cost based on the data compression rate and the model update amount, and Δ k,t is the model update amount of the end device k at time step t. In this way, the end-cloud collaborative learning system can evaluate the model differences and communication resource conditions in real time, dynamically adjust the model update frequency and data compression rate of the end device, and thus optimize the overall learning efficiency and communication efficiency.

[0117] Step 4: The terminal device collects the data to be tested and inputs it into the optimized global model to obtain the test results.

[0118] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0119] The above are only preferred embodiments of the embodiments of the present application and are not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A knowledge transfer method based on conditional cascade distillation, applied to a terminal-cloud collaborative system, wherein the terminal-cloud collaborative system includes a cloud server and a plurality of terminal devices connected to the cloud server, characterized in that: The steps include: The cloud server constructs the model and sends it to the plurality of terminal devices; Each of the terminal devices collects training data and inputs it into the model sent down for iterative training to obtain a trained terminal device model. The terminal device updates and uploads the trained terminal device model to the cloud server. In the process of iterative training of the sent model by the terminal device, the feature extraction process is optimized by conditional probability representation and cascade distillation loss function, which specifically includes the following steps: constructing a data set based on the training data, treating the probability of known categories in the data set as conditions, and treating the probability of unknown categories as unlabeled; for each category C i , calculate the conditional probability P(C i |x), that is, given input x, category C i The probability of occurrence can be expressed as: in, represents a set of known categories, represents the set of unknown categories, p Fore and p Back Represent the probability distribution of foreground and background respectively; Defining marginal sets It contains all unknown categories and combines the probabilities of background categories into a single probability value p B , the original probability value of the foreground class Keeping it unchanged, the marginal loss function has the following form: Among them, y i is a binary indicator, when category C i The value is 1 if it is a correct label, otherwise it is 0; In each training iteration, the model's predicted probability for each category is calculated, and the model's parameters are updated according to the marginal loss function, expressed as: Among them, θ represents the parameters of the model, η is the learning rate, Represents the gradient of the marginal loss function with respect to the model parameters; The cloud server uses cascade distillation to allow each end device model to absorb the knowledge of the cloud model. In the cascade distillation process, the cloud model serves as the teacher model and the end device model serves as the student model. The cloud server receives updates from several end device models and performs aggregation and communication optimization through a weighted aggregation algorithm, and sends the optimized global model to the end device. The terminal device collects the data to be tested and inputs it into the optimized global model to obtain the test results.

2. The knowledge transfer method based on conditional cascade distillation according to claim 1, characterized in that: The feature calibration network is preset in the terminal device, and the feature calibration network is based on self-supervised learning to capture the feature mapping relationship between the terminal device model and the cloud model, and calibrates the feature expression according to the feature mapping relationship between the terminal device model and the cloud model, specifically including the following steps: The feature mapping process is expressed by the following formula: f enc (x;θ enc )=φ(x) (4) f dec (z;θ dec )=ψ(z) (5) Among them, x represents the input data, z is the feature expression after mapping, and θ enc and θ dec Represent the parameters of the encoder and decoder respectively, φ and ψ are the corresponding mapping functions; when training the feature calibration network, the feature calibration loss function is introduced to quantify the encoder f enc Output and decoder f enc The difference between the outputs, the feature calibration loss function can be expressed as: The end-to-end training strategy is implemented through joint optimization, the formula is as follows: Among them, θ model Represents the parameters of the end device model or cloud model, is the loss function of the main task, and λ is the weight of the feature calibration loss, which is used to balance the main task loss and the feature calibration loss.

3. The knowledge transfer method based on conditional cascade distillation according to claim 2, characterized in that: The cloud server uses cascade distillation to allow each end device model to absorb the knowledge of the cloud model, which includes the following steps: Designing Cascade Distillation Loss Function Focus on the prediction output of the end device model And the prediction output of the cloud model And the knowledge transfer is achieved by minimizing the difference between the two, which is expressed as follows: Among them, D k represents the data set of end device k, is the distance metric function; The parameter update of the end device model uses the following formula: Among them, v k，t is the momentum term for parameter update, μ is the momentum coefficient, which is used to accumulate past gradient directions to form a more stable update direction. The end device k is based on its local data set D k Perform model training, θ k，t is the model parameter of the tth iteration, η k is the learning rate; θ l，t+1 is the aggregated local model parameter, w k It is a weight that is dynamically adjusted according to the local performance of the end device k.

4. The knowledge transfer method based on conditional cascade distillation according to claim 3, characterized in that: The cloud server receives several end device model updates and performs aggregation and communication optimization through a weighted aggregation algorithm, and sends the optimized global model to the end device. The specific steps include: Regularization terms are introduced to prevent overfitting and ensure the smoothness of model updates. The aggregation strategy can be expressed as: Among them, θ g，t+1 is the global model parameter after aggregation, L(D k ,θ g,t ) is the loss function of the end device k, w′ k is the weight of the end device k, λ k is a regularization parameter used to balance the local loss of the end device and the smoothness of the model update; When the model update in the end device exceeds the threshold, the end device model update is sent to the cloud server, and the data compression technology using gradient quantization is used, which can be expressed as: i k,comp =f qua (i k -θ g ) (12) Among them, f qua is a quantization function used to map continuous gradient values ​​to a finite discrete set. θ k,comp Updated for the compressed model; The dynamic adjustment process of model update frequency and communication strategy can be expressed as: a k，t+1 =a k,t ·f fre (D k，g ,r rec ) (13) Among them, α k，t refers to the model update frequency of end device k at time step t, Δ k，g Represents the difference between the end device model parameters and the global model parameters, r rec Indicates the current communication resource status, function f fre (·,·) is used to adjust the update frequency according to the model difference and communication resource conditions. When the model difference exceeds the preset threshold, the update frequency tends to be increased, and when the communication resources are limited, a more aggressive data compression rate tends to be adopted. The model difference Δ k，g The evaluation is achieved by the output difference between the representative data in the computing device and the cloud model, which is expressed as: in, and They represent the prediction functions of the end device and the global model respectively, KL represents the Kullback-Leibler divergence, is a representative dataset used for evaluation, and the data compression ratio C k is an adjustable parameter that can be adjusted based on the model update amount Δ of the end device k k，t Dynamic adjustment, adjustment C k This is equivalent to changing the quantization level or compression rate to control the communication cost C. k，t : C k，t =g(C k ,D k，t ) (15) Among them, C k，t represents the communication cost of end device k at time step t, g(·,·) is a cost function used to calculate the communication cost based on the data compression rate and the model update amount, Δ k，t is the model update amount of end device k at time step t.

Images