Federal extraterritorial generalization method and application thereof in cross-scene classification task
By introducing a diffusion model semantic manifold into federated learning, pseudo-semantic samples are generated and feature fusion is optimized, which solves the generalization problem of traditional federated learning under multi-domain data and achieves stable adaptation and high-precision prediction of cross-domain data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-23
- Publication Date
- 2026-04-03
AI Technical Summary
Traditional federated learning methods fail to generalize effectively when dealing with non-independent and identically distributed data, especially when the data distributions of multiple clients are inconsistent or the data come from different domains. This results in a significant drop in model performance on out-of-domain data, and existing domain adaptation and domain generalization methods increase computational and communication costs.
We adopt a federated out-of-domain generalization method based on the semantic manifold of the diffusion model. By building a local dataset on the client, we perform feature fusion and diffusion model training to generate pseudo-semantic samples. We also introduce energy gradient and confidence weighting mechanisms to optimize the cross-domain feature fusion process.
It improves the generalization ability of cross-domain data under the federated learning framework, enhances the adaptability and prediction accuracy of the model in multi-domain environments, reduces the classification bias and misjudgment rate caused by scene switching, and improves the robustness and usability of cross-scene recognition.
Smart Images
Figure CN121786570A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of federated semantic communication and generative artificial intelligence, and in particular to a federated out-of-domain generalization method based on a diffusion model semantic manifold and its application in cross-scene classification tasks. Background Technology
[0002] With the rapid development of artificial intelligence and machine learning technologies, federated learning, as a distributed machine learning method, has received widespread attention. It allows multiple distributed devices (such as smartphones, sensors, and cars) to train models locally and upload updated model parameters to a central server for global aggregation, thus avoiding centralized storage and sharing of raw data and effectively protecting user privacy. However, federated learning faces many challenges when dealing with non-independent and identically distributed data. Especially when data distributions across multiple clients are inconsistent or data comes from different domains, traditional federated learning methods often fail to generalize effectively, leading to a significant decrease in model performance on out-of-domain data.
[0003] In many practical applications, such as autonomous driving, healthcare, and smart manufacturing, data typically originates from diverse devices or environments, exhibiting significant domain-specific differences. These differences manifest not only in variations in data distribution but also in variations in data type and acquisition methods. Traditional domain adaptation and generalization methods often rely on large amounts of labeled data or domain-specific fine-tuning, which contradicts the decentralized and privacy-preserving characteristics of federated learning. Furthermore, these methods often perform poorly when dealing with new domains or unseen conditions, necessitating continuous model retraining and adjustments, thus increasing computational and communication costs. Summary of the Invention
[0004] The purpose of this invention is to provide a federated out-of-domain generalization method based on the semantic manifold of the diffusion model and its application in cross-scenario classification tasks, aiming to improve the generalization ability of cross-domain data under the federated learning framework.
[0005] To achieve the above objectives, this invention provides a federated out-of-domain generalization method based on a diffusion model semantic manifold, comprising the following steps:
[0006] S1. Build local datasets with different distributions on each client participating in the task;
[0007] S2. Perform feature fusion on the semantic features output by the local encoder of each client's local data, and train the local encoder and global classifier of each client by combining the local dataset of each client and the corresponding feature fusion results, and proceed to step S3.
[0008] S3. Determine whether federated learning has converged. If yes, proceed to step S9; otherwise, proceed to step S4.
[0009] S4. Each client trains its local decoder using the stable feature representation obtained from the pre-trained encoder trained in step S2.
[0010] S5. Each client trains a local diffusion model using the stable feature representation obtained from the pre-trained encoder trained in step S2.
[0011] S6. The stable features obtained by the pre-trained encoder trained in step S2 are semantically generalized by the pre-trained diffusion model trained in step S5 to obtain the corresponding pseudo-semantics.
[0012] S7. The pseudo-semantics obtained in step S6 are processed by the pre-trained decoder obtained in step S3 to obtain the corresponding pseudo-samples.
[0013] S8. Return to step S2. Each client will add the corresponding pseudo-samples to the local dataset and retrain the local encoder and global classifier of each client.
[0014] S9. The pre-trained encoders of all current clients, together with the global classifier, constitute the global model for inference and prediction.
[0015] Preferably, the local dataset in step S1 includes data samples, labels, and semantic difference data between different domains, and the data samples include at least one of image, video, and audio data.
[0016] Preferably, the feature fusion using the spherical linear interpolation method in step S2 includes:
[0017]
[0018] in, These are the semantic features output by the client u after its local data has been processed by its local encoder. These are the semantic features output by the client v's local data after it has been processed by its local encoder. yes and Feature fusion results It follows the parameter as The interpolation ratio of the Beta distribution, yes and The angle between them.
[0019] Preferably, step S6 specifically includes:
[0020]
[0021] in, It is a client The The local sample at the ... The pseudo-semantics of time step; It is a client The The local sample at the ... Noisy semantics of time steps; It is the first Noise attenuation coefficient at time step; It is in the The predicted noise value added at each time step;
[0022] The initial value is the client The Stable features obtained from local samples by the pre-trained encoder trained in step S2 When the preset time step is reached, the client obtains... The Pseudo-semantics corresponding to each local sample .
[0023] Preferably, the domain generalization sampling pair based on energy gradient Make corrections:
[0024]
[0025] in, yes The correction results; It is the prediction noise ratio factor; It is the first The time step controls the coefficient of the gradient correction magnitude; It is the first Energy gradient at time step , Indicates about gradient operator, It is a client The The local sample at the ... Noisy semantics of time steps It is a client The The local sample at the ... Time step The estimate, It is a global classifier pair energy, y is the category label, Y is the total number of categories. It is a global classifier pair The predicted score.
[0026] Preferably, step S9 further includes:
[0027] Randomly select a pretest subset from the dataset to be predicted in the target domain For each from The pre-test samples are generated by the pre-trained encoder of each client for each sample from... After representing the pre-test samples using a defined dimension embedding, the corresponding predicted score vector is obtained through a global classifier. Based on the predicted score vector, the confidence score of the pre-trained encoder for each client is calculated.
[0028]
[0029] in, yes The m-th pretest sample, It is aimed at The confidence score of the pre-trained encoder of the current client u. Y is the predicted value of the class label y in the output of the Softmax activation function of the predicted score vector corresponding to the current client u, where Y is the total number of classes.
[0030] Then obtain the pre-trained encoder of each client. Confidence score on:
[0031]
[0032] Where M is Number of pre-tested samples;
[0033] Take in The pre-trained encoders of the current clients with the highest confidence scores are combined with the global classifier to form a global model for inference and prediction; where k is a preset value.
[0034] Preferably, the inference prediction includes:
[0035] Calculation in The weights of the k pre-trained encoders with the highest confidence scores for the current client:
[0036]
[0037] in, It is aimed at The weights of the pre-trained encoder of the current client u. Is The set of clients corresponding to the k current clients' pre-trained encoders with the highest confidence scores;
[0038] For The k prediction score vectors of the current client with the highest confidence scores are weighted and aggregated, and the category label corresponding to the largest prediction value in the aggregation result is taken as the final inference prediction result.
[0039] This invention also provides applications of the above-described generalization method in cross-scene classification tasks, including:
[0040] For each client participating in the classification task, a local dataset with a different distribution is constructed. The local dataset includes at least one of image, video and audio data collected by the client, as well as corresponding category labels.
[0041] Based on steps S2 to S9 in the above generalization method, a global model for classification reasoning prediction is constructed.
[0042] Based on the global model, cross-scene classification results are obtained.
[0043] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the execution of the program on the processor implements the steps in the federated out-of-domain generalization method based on the diffusion model semantic manifold as described above.
[0044] The present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the steps in the federated out-of-domain generalization method based on the diffusion model semantic manifold as described above.
[0045] Compared with the prior art, the present invention has the following beneficial effects:
[0046] (1) This invention employs a federated out-of-domain generalization method based on a diffusion model semantic manifold, aiming to improve the generalization ability of cross-domain data under the federated learning framework. This method introduces a diffusion model to generate inter-domain bridging feature representations. While protecting privacy, the local client can fully utilize the feature representations of the data to generate robust pseudo-feature samples. This method effectively reduces the differences between cross-domain data, improves the model's adaptability to out-of-domain data and prediction accuracy, and significantly enhances the application effect of federated learning in dynamic, multi-domain environments.
[0047] (2) When the above generalization method is applied to cross-scene classification tasks, the model can still maintain stable class discrimination ability even when there are obvious differences between the training scene and the test scene. By supplementing the transition samples between scenes in the feature space and aligning the intra-class semantic structure of each client, the classification bias and misjudgment rate caused by scene switching can be significantly reduced, and the classification accuracy and robustness on the cross-scene test set can be improved, thereby enhancing the usability of this method in multi-scene recognition and actual deployment.
[0048] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0049] Figure 1 This is a flowchart of an embodiment of the present invention. Detailed Implementation
[0050] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0051] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0052] like Figure 1 As shown, the technical solution designed in this invention includes the following steps:
[0053] S1. Each local user participating in the task constructs a local dataset with a different distribution. The local dataset mentioned here generally includes data samples, labels, etc. from different domains. The data samples include at least one of the image, video, and audio data used in conventional model prediction.
[0054] The operation steps of step S1 above are as follows:
[0055] Local users can be terminal devices or institutional nodes deployed in different scenarios, such as visual inspection terminals installed on different factory production lines, or camera terminals distributed in stores in different cities.
[0056] In one specific implementation, taking a cross-scene classification task as an example, the local dataset includes several image samples and their category labels. The image samples can be production line defect detection images, equipment operation status monitoring images, or scene monitoring images, etc. Images collected by different clients have differences in lighting conditions, shooting angles, background environments, and noise levels, forming data distributions in different fields.
[0057] S2. Perform feature fusion on the semantic features output by each client's local data after passing through its local encoder, and train the local encoder and global classifier of each client together with the local dataset of each client and the corresponding feature fusion results.
[0058] Specifically, this invention performs feature fusion using a spherical linear interpolation method:
[0059]
[0060] in, These are the semantic features output by the client u after its local data has been processed by its local encoder. These are the semantic features output by the client v's local data after it has been processed by its local encoder. yes and Feature fusion results It follows the parameter as The interpolation ratio of the Beta distribution, yes and The angle between them.
[0061] This feature fusion effectively integrates feature representations between clients, thereby reducing inter-domain differences and enhancing the diversity and robustness of the global model.
[0062] S3. Determine whether federated learning has converged. If yes, proceed to step S7; otherwise, proceed to step S6.
[0063] S4. Each client trains its local decoder using the stable feature representation obtained from the pre-trained encoder trained in step S2, until convergence.
[0064] Each client uses its trained encoder Generate stable feature representations ,in Indicates the first... One local sample, Label it. Then, use that representation to train the decoder.
[0065] S5. Each client trains a local diffusion model using the stable linguistic feature representation of a pre-trained encoder. The diffusion model obtains pseudo-semantics through domain generalization sampling, and the pseudo-semantics are then processed by a pre-trained decoder to obtain corresponding regionalized pseudo-samples. Each client utilizes the feature representation... The local diffusion model is trained by progressively moving towards the semantic representation. Adding noise and then denoising it generates cross-domain robust pseudo-semantics. During the diffusion process, semantic representation Noise is gradually added until a certain set time step is reached. This renders the sample as pure noise. Each step in the diffusion process follows this procedure:
[0066]
[0067] in, It is a client The The local sample at the ... The pseudo-semantics of time step; It is a client The The local sample at the ... Noisy semantics of time steps; It is the first The noise attenuation coefficient of the time step is used to control the degree of noise addition; It is a noise prediction network learned by the diffusion model, used to predict the noise at the th... Noise added step by step.
[0068] This invention also introduces energy gradient-based domain generalization sampling to optimize the denoising process:
[0069] First, the energy function of the global classifier is defined as:
[0070]
[0071] in, It is a global classifier pair The energy represents its uncertainty regarding classification; It is a client The The local sample at the ... Noisy semantics of time steps It is a client The The local sample at the ... Time step The estimate, It is a global classifier pair The predicted score is given by y, where y is the category label and Y is the total number of categories.
[0072] Secondly, calculate the energy gradient:
[0073]
[0074] in, Indicates about The gradient operator.
[0075] Finally, the corrected prediction noise is:
[0076]
[0077] in, It is the prediction noise proportionality factor, used to control the proportion of prediction noise used; It is a dynamic coefficient for the time step, used to control the magnitude of gradient correction.
[0078] S6. Return to step S2. Each client adds the pseudo-samples to its local dataset and retrains the local encoder and global classifier. The pseudo-samples added to the local dataset also need to undergo global feature fusion.
[0079] Cross-client feature fusion using spherical linear interpolation:
[0080]
[0081] in, The interpolation ratio follows the parameter. Beta distribution, pseudo-semantics and The angle between them.
[0082] S7. After the last round of communication, the encoders of all current clients and the global classifier together form a global model for inference and prediction.
[0083] Before formal inference, a small pretest subset is randomly selected from the dataset to be predicted in the target domain. For each from Input samples of the target domain with a distribution different from that of the training domain Each encoder generates a fixed-dimensional embedding representation that characterizes a single input sample. , After classifier Get predicted logits .
[0084] The server calculates the confidence score of each client's local encoder using the following formula:
[0085] in, yes The m-th pretest sample, It is aimed at The confidence score of the pre-trained encoder of the current client u. It is the predicted value of the class label y in the output of the Softmax activation function of the predicted score vector corresponding to the current client u, where Y is the total number of classes. Represents Shannon entropy. This is the output of the Softmax activation function applied to the predicted score vector corresponding to the current client u. Then, the mean-based entropy score is used to summarize the performance of each client-side local encoder on the pre-test subset. Confidence score on:
[0086]
[0087] Where M is the number of samples in the pretest subset.
[0088] Take in The k pre-trained encoders with the highest confidence scores from the current client, together with the global classifier, constitute the global model for subsequent formal inference and prediction. Here, k is a preset value given by those skilled in the art based on actual needs.
[0089] To avoid relying on a single encoder for decision-making, the present invention will also... The confidence scores of the k encoders that achieve the highest confidence scores are converted into weights for their respective clients. The final prediction outputs of the test samples are then weighted and aggregated according to the weights of each client. The weight calculation formula is as follows:
[0090]
[0091] Final prediction label The category is determined by the category corresponding to the largest predicted value in the weighted aggregation output:
[0092]
[0093] in, It is the predicted value at label y, obtained by weighting and aggregating the predicted score vectors of the k encoders.
[0094] This weighting mechanism can effectively address the distribution differences caused by data from outside the domain and improve the inference accuracy of the global model on unseen data.
[0095] S8. Based on the global model constructed in step S7, obtain the classification results for the cross-scene classification task.
[0096] Therefore, this invention proposes a federated out-of-domain generalization method based on a diffusion model semantic manifold. By introducing energy gradient-guided diffusion model sampling optimization, each client can generate robust and cross-domain adaptive pseudo-features, thereby improving the generalization ability of the global model on out-of-domain data. Specifically, the diffusion model gradually adds noise to the original features and generates pseudo-features during the denoising process, ensuring that the generated pseudo-features retain the key structural features of the original data while exhibiting strong robustness to data from different domains. Furthermore, this invention combines spherical linear interpolation technology and a confidence-weighted logit aggregation mechanism to further optimize the cross-domain feature fusion process. Through the spherical linear interpolation method, pseudo-feature samples generated by different clients are effectively fused, reducing the differences between cross-domain data.
[0097] Furthermore, when the above generalization method is applied to cross-scene classification tasks, it can significantly reduce the classification bias and misjudgment risk caused by scene switching, even under changes in scene factors such as lighting, background, style, and imaging equipment, thereby improving the recognition stability and classification accuracy in out-of-domain scenes. During the inference phase, several samples are randomly sampled from the test set, and each client's encoder generates predicted values for pre-testing. The Topk suitable encoder is selected using Shannon entropy. During formal inference, the server performs weighted aggregation based on the confidence level of each client, ensuring that more reliable client models have a greater weight in the final aggregation, thereby further improving the robustness and usability of cross-scene classification tasks in unknown scenarios.
[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A method for generalizing from outside a federal territory, characterized in that, Includes the following steps: S1. Build local datasets with different distributions on each client participating in the task; S2. Perform feature fusion on the semantic features output by the local encoder of each client's local data, and train the local encoder and global classifier of each client by combining the local dataset of each client and the corresponding feature fusion results, and proceed to step S3. S3. Determine whether federated learning has converged. If yes, proceed to step S9; otherwise, proceed to step S4. S4. Each client trains its local decoder using the stable feature representation obtained from the pre-trained encoder trained in step S2. S5. Each client trains a local diffusion model using the stable feature representation obtained from the pre-trained encoder trained in step S2. S6. The stable features obtained by the pre-trained encoder trained in step S2 are semantically generalized by the pre-trained diffusion model trained in step S5 to obtain the corresponding pseudo-semantics. S7. The pseudo-semantics obtained in step S6 are processed by the pre-trained decoder obtained in step S3 to obtain the corresponding pseudo-samples. S8. Return to step S2. Each client will add the corresponding pseudo-samples to the local dataset and retrain the local encoder and global classifier of each client. S9. The pre-trained encoders of all current clients, together with the global classifier, constitute the global model for inference and prediction.
2. The generalization method according to claim 1, characterized in that, The local dataset in step S1 includes data samples and labels from different fields, and the data samples include at least one of image, video and audio data.
3. The generalization method according to claim 1, characterized in that, The feature fusion using the spherical linear interpolation method in step S2 includes: ， in, These are the semantic features output by the client u after its local data has been processed by its local encoder. These are the semantic features output by the client v's local data after it has been processed by its local encoder. yes and Feature fusion results It follows the parameter as The interpolation ratio of the Beta distribution, yes and The angle between them.
4. The generalization method according to claim 1, characterized in that, Step S6 specifically includes: ， in, It is a client The The local sample at the ... The pseudo-semantics of time steps; It is a client The The local sample at the ... Noisy semantics of time steps; It is the first Noise attenuation coefficient for the time step; It is in the The predicted noise value added at each time step; The initial value is the client The Stable features obtained from local samples by the pre-trained encoder trained in step S2 When the preset time step is reached, the client obtains... The Pseudo-semantics corresponding to each local sample .
5. The generalization method according to claim 4, characterized in that, Domain generalization sampling pairs based on energy gradient Make corrections: ， in, yes The correction results; It is the prediction noise scaling factor; It is the first The time step controls the coefficient of the gradient correction magnitude; It is the first Energy gradient at time step , Indicates about gradient operator, It is a client The The local sample at the ... Noisy semantics of time steps It is a client The The local sample at the ... Time step The estimate, It is a global classifier pair energy, y is the category label, Y is the total number of categories. It is a global classifier pair The predicted score.
6. The generalization method according to claim 1, characterized in that, Step S9 further includes: Randomly select a pretest subset from the dataset to be predicted in the target domain For each from The pre-test samples are generated by the pre-trained encoder of each client for each sample from... After representing the pre-test samples using a defined dimension embedding, the corresponding predicted score vector is obtained through a global classifier. Based on the predicted score vector, the confidence score of the pre-trained encoder for each client is calculated. ， in, yes The m-th pretest sample, It is aimed at The confidence score of the pre-trained encoder of the current client u. Y is the predicted value of the class label y in the output of the Softmax activation function of the predicted score vector corresponding to the current client u, where Y is the total number of classes. Then obtain the pre-trained encoder of each client. Confidence score on: ， Where M is Number of pre-tested samples; Take in The pre-trained encoders of the current clients with the highest confidence scores are combined with the global classifier to form a global model for inference and prediction; where k is a preset value.
7. The generalization method according to claim 6, characterized in that, The inference prediction includes: Calculation in The weights of the k pre-trained encoders with the highest confidence scores for the current client: ， in, It is aimed at The weights of the pre-trained encoder of the current client u. Is The set of clients corresponding to the k current clients' pre-trained encoders with the highest confidence scores; For The k prediction score vectors of the current client with the highest confidence scores are weighted and aggregated, and the category label corresponding to the largest prediction value in the aggregation result is taken as the final inference prediction result.
8. The application of the generalization method as described in any one of claims 1 to 7 in cross-scene classification tasks, characterized in that, include: For each client participating in the classification task, a local dataset with a different distribution is constructed. The local dataset includes at least one of image, video and audio data collected by the client, as well as corresponding category labels. According to steps S2 to S9 in the generalization method as described in any one of claims 1 to 7, a global model for classification reasoning prediction is constructed. Based on the global model, cross-scene classification results are obtained.
9. An electronic device, comprising a memory and a processor, characterized in that, The memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the generalization method as described in any one of claims 1-7.
10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps in the generalization method as described in any one of claims 1-7.