Federated learning methods that support heterogeneous model architecture search and zero-shot knowledge fusion

Abstract

This invention relates to a federated learning method that supports heterogeneous model architecture search and zero-shot knowledge fusion. It includes: providing a federated learning system to be federated; configuring the federated learning system to perform federated learning; when any client performs teacher model generation processing, based on local private data within the client, searching and generating a neural network local teacher model that best matches the client, and sending the generated neural network local teacher model to a connected server; after the server performs student model generation processing, generating at least a globally shared student model; wherein, when generating the globally shared student model, a base student model constructed using the pseudo-supervised training dataset is distilled and trained, and the globally shared student model is generated after distilling and training the base student model. This invention effectively supports heterogeneous models on clients, improves personalization capabilities and privacy protection, reduces communication costs, and exhibits excellent model generalization.

Description

Technical Field

[0001] This invention relates to a federated learning method, and more particularly to a federated learning method that supports heterogeneous model architecture search and zero-shot knowledge fusion. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, deep neural networks have made significant progress in tasks such as image recognition and natural language processing. However, the success of these tasks relies on centralized data collection and model training processes, posing serious challenges to user privacy and data security. Against this backdrop, Federated Learning (FL) has been proposed. As a novel distributed collaborative learning paradigm, Federated Learning supports multiple terminal devices in collaboratively training models without sharing local private data, effectively mitigating data privacy leakage issues. Each terminal device acts as a client in the Federated Learning process.

[0003] Despite its robust privacy protection mechanisms, federated learning still faces numerous challenges in practical applications. Specifically: First, the data distribution among clients often exhibits significant differences (i.e., non-independent and identically distributed), making it difficult for a unified model architecture to achieve ideal performance on local data. Second, variations in computing power and memory bandwidth among different clients hinder the efficient operation of a unified model architecture across all clients. Furthermore, to improve model performance, traditional federated learning methods typically rely on multiple rounds of communication to iteratively synchronize model parameters, resulting in high system load and communication overhead, and potentially introducing additional privacy risks.

[0004] To address the aforementioned issues, Personalized Federated Learning (PFL) has become a research hotspot in recent years. Its goal is to provide each client with a customized model that better suits their local data characteristics while ensuring privacy. Currently, in the field of Personalized Federated Learning (PFL), various methods have attempted to improve the adaptability of models in scenarios with heterogeneous data distribution and resource constraints. These methods can be mainly categorized into the following two types:

[0005] The first category comprises personalized federated optimization methods based on a unified model architecture. Representative methods include pre-FedAvg, FedBN, Ditto, FedPer, and LG-FedAvg. These methods typically assume that all clients share a unified model structure and achieve personalized training through local fine-tuning, regularization constraints, or module partitioning. For example, the FedBN method mitigates feature distribution shifts by localizing batch normalization layer parameters, while the FedPer method divides the model into shared and personalized layers for separate processing. However, this type of method essentially still uses a static model structure, making it difficult to dynamically adapt the structure based on significantly different data distributions and computational resources among clients, resulting in limited performance and efficiency.

[0006] The second category is personalized federated learning methods that introduce automatic structural optimization mechanisms. These methods typically employ Neural Architecture Search (NAS) or model pooling mechanisms. Methods like FedNAS optimize models through a distributed NAS collaborative process. In this case, the control and decision-making for the architecture search are primarily handled by the server, with clients only participating in performance evaluation. Ultimately, a unified optimal model is selected for global sharing. Although FedNAS formally considers client differences, its search process still heavily relies on server coordination, making it difficult to truly achieve structural heterogeneity and local adaptation.

[0007] In addition, some methods utilize predefined models to achieve model personalization, such as DENSE and FedMMD. These methods employ existing excellent architectures like ResNet and MobileNet for on-demand deployment to achieve model heterogeneity. However, FedMMD requires distillation using real public data, while DENSE requires external generators to synthesize images, both of which pose risks of privacy leakage and high system complexity. Furthermore, FedMHO utilizes model pools for structurally differentiated deployment, using discriminative models on resource-rich clients and generative models on resource-constrained clients, achieving one-time distillation collaboration across heterogeneous models.

[0008] While personalized federated learning has made some progress in supporting personalization and model heterogeneity, existing methods still have the following limitations, including:

[0009] 1) The homogeneity of the model structure on the client side limits the personalized performance. Specifically, most federated learning methods assume that all clients adopt a uniform model architecture, which lacks the ability to adapt to heterogeneous hardware resources and non-independent and identically distributed data, and cannot meet the personalized modeling needs of different clients.

[0010] 2) The structure relies on a predefined set of models or manual selection of model architecture, lacking automatic search capabilities. Specifically, some methods that support heterogeneous modeling usually rely on a predefined set of models or manual selection of model structures, lacking an automated search mechanism, making it difficult to dynamically optimize the model architecture for the client's local data, and limiting the degree of personalization.

[0011] 3) High communication overhead makes it difficult to deploy in low-bandwidth scenarios. Specifically, existing personalized methods based on neural architecture search generally rely on multi-round communication and intermediate structure transmission, resulting in high communication costs and increased privacy risks, making them difficult to apply in communication-constrained scenarios.

[0012] 4) Existing federated distillation methods rely on public data, making it difficult to meet privacy constraints. Specifically, while distillation can be used for knowledge transfer between heterogeneous models, most federated distillation methods (such as FedDF and FedMD) require the introduction of a portion of pre-defined public data as a distillation medium for uniformly training student models. However, in practical applications, such public data is difficult to obtain or cannot cover the non-independent data distribution of clients, and the introduction of public data itself may bring new privacy risks. Therefore, how to achieve efficient and secure knowledge fusion under the premise that the server cannot access any real data remains a major challenge in current federated distillation research.

[0013] In summary, existing personalized federated learning methods still have significant shortcomings in terms of model heterogeneity adaptability, automated architecture optimization, communication efficiency, and knowledge distillation under conditions of no public data, making it difficult to meet the needs of practical applications. Summary of the Invention

[0014] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a federated learning method that supports heterogeneous model architecture search and zero-shot knowledge fusion. It can effectively support heterogeneous models on the client side, improve personalization capabilities and privacy protection, reduce communication costs, and has excellent model generalization performance.

[0015] According to the technical solution provided by this invention, a federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion is provided, the federated learning method comprising:

[0016] A federated learning system is provided for federated learning, wherein the federated learning system includes a server and a client group adapted to and connected to the server. The client group includes several clients, each client having local private data, and the local private data of all clients has the characteristic of non-independent and identically distributed.

[0017] Configuring a federated learning system for federated learning includes at least configuring each client to perform teacher model generation processing and configuring the server to perform student model generation processing.

[0018] When any client performs teacher model generation processing, it searches for and generates a neural network local teacher model that best matches the client based on the client's local private data, and sends the generated neural network local teacher model to the connected server.

[0019] After the server performs the student model generation process, it generates at least a globally shared student model. The generation of the globally shared student model includes:

[0020] Based on all received neural network local teacher models, a pseudo-supervised training dataset is generated. The pseudo-supervised training dataset includes several pseudo-supervised training samples. Each pseudo-supervised training sample includes a pseudo-training image that can reproduce the knowledge representation of the corresponding client and a fused soft label corresponding to the pseudo-training image. When generating each pseudo-supervised training sample, the server does not access the local private data of the corresponding client in the client group.

[0021] A basic student model is constructed by distillation training using the pseudo-supervised training dataset, and a globally shared student model is generated after distillation training of the basic student model.

[0022] Each client performing teacher model generation processing includes:

[0023] Construct a network particle swarm, wherein the network particle swarm consists of several network particles, each of which contains a network encoding sequence and represents a candidate convolutional neural network based on the corresponding network encoding sequence, and the length of the network particle is the network depth of the candidate convolutional neural network it represents.

[0024] Based on the local private data of the current client, the above-constructed network particle swarm is optimized by PSO, so as to obtain the optimal candidate convolutional neural network corresponding to the optimal network particle.

[0025] The optimal candidate convolutional neural network is trained based on the local private data of the current client, and a local teacher model of the neural network is generated after the model is trained.

[0026] At least one local training set and one local validation set should be constructed based on the local private data within each client.

[0027] When performing PSO optimization on a network, it includes at least several sequential network training optimization processes, where each network training optimization process includes sequential optimization training processes and particle update processes.

[0028] During the optimization training process, the network encoding sequence of each network particle is converted into a corresponding candidate convolutional neural network, and the model training of each converted candidate convolutional neural network is performed on the target round using the optimization local training set, so as to generate the corresponding candidate convolutional neural network after model training.

[0029] When performing particle update processing, each candidate convolutional neural network is validated using the optimization validation set, and the network accuracy calculated by the network validation is used as the fitness value of the current candidate convolutional neural network.

[0030] Based on the fitness value of each candidate convolutional neural network, the individual optimal convolutional neural network corresponding to each candidate convolutional neural network is determined, and the global optimal convolutional neural network is determined based on all the individual optimal convolutional neural networks.

[0031] When the PSO optimization of the network particle swarm fails to reach the target state, each network particle is updated based on the global optimal convolutional neural network and the individual optimal convolutional neural network of each network particle to update the network encoding sequence of each network particle and perform the next network training optimization process.

[0032] When the PSO optimization of the network particle swarm reaches the target state, the network particle corresponding to the globally optimal convolutional neural network is taken as the optimal network particle, and the globally optimal convolutional neural network configuration is taken as the optimal candidate convolutional neural network.

[0033] When generating a pseudo-supervised training dataset, the following are included:

[0034] Within the server, based on the local teacher model of each neural network, a pseudo-training image generation mechanism optimized by random noise input is used to generate several pseudo-training images, wherein each pseudo-training image satisfies the requirement to reproduce the knowledge representation of the corresponding client.

[0035] Each pseudo-training image is loaded into each neural network local teacher model within the server to configure the neural network local teacher model to perform inference prediction on the loaded pseudo-training images and generate personalized model inference output information.

[0036] The server will perform consistency-aware fusion processing on all corresponding personalized model inference output information for each pseudo-training image to generate fused soft labels after consistency-aware fusion processing.

[0037] The generated fused soft labels and corresponding pseudo-training images are used as pseudo-supervised training samples in the pseudo-supervised training dataset.

[0038] For each neural network local teacher model, when generating pseudo-training images using a pseudo-image generation mechanism optimized with random noise input, the following steps are included:

[0039] Extract the task layer weight information of the local teacher model of the neural network, and construct a category similarity matrix based on the extracted task layer weight information;

[0040] Based on the constructed category similarity matrix, a category-aware pseudo-label modeling method is used to construct the corresponding soft label distribution group. The soft label distribution group includes several soft label distribution vectors. When the neural network local teacher model makes a classification prediction on the pseudo-training image, the number of distribution probabilities in the soft label distribution vector is consistent with the number of categories predicted by the neural network local teacher model.

[0041] The server initializes a set of random noise images for the current neural network local teacher model, and optimizes and updates each random noise image using each soft label distribution vector, so as to generate a corresponding pseudo training image with class discrimination after optimization and update.

[0042] When optimizing and updating each image with random noise, the following steps are included:

[0043] Random noise images are loaded into a neural network local teacher model, and the corresponding noise image logits output information is obtained through inference and prediction by the neural network local teacher model.

[0044] Temperature scaling is applied to the logits output information of the noisy image, and the prediction probability distribution of the noisy image is generated by softmax normalization.

[0045] The cross-entropy loss between each soft label distribution vector and the predicted probability distribution of the noisy image is used as the optimization objective. During the optimization process, the random noise image is updated by gradient descent until the cross-entropy loss, which is the optimization objective, is minimized, and then the corresponding pseudo-training image is generated.

[0046] When the server performs consistency-aware fusion processing with all corresponding personalized model inference output information for each pseudo-training image, it includes:

[0047] The server calculates the average output information of the personalized model corresponding to the personalized model inference output information of the current pseudo-training image, and generates a teacher consistency metric based on the calculated average output information of the personalized model.

[0048] When the teacher consistency metric is not greater than the consistency judgment threshold, the average output information of the personalized model is used as the fusion soft label. Otherwise, based on the main teacher fusion mechanism, all personalized model inference output information is output fused to generate the corresponding fusion soft label after output fusion.

[0049] When calculating the teacher consistency metric, we have:

[0050]

[0051] in, Pseudo-training images Teacher consistency metrics This represents the number of local teacher models of the neural network received by the server. For the first A neural network local teacher model uses pseudo-training images Personalized model inference output information generated through inference and prediction. The average output information of the personalized model To calculate the personalized model inference output information based on the average output information of the personalized model The divergence.

[0052] When fusing all personalized model inference output information based on the master teacher fusion mechanism, it includes:

[0053] The server selects one neural network local teacher model as the master teacher model and uses the remaining neural network local teacher models as assistant teacher models. The master teacher model is at least the neural network local teacher model with the most complex network structure.

[0054] When performing output fusion, we have:

[0055]

[0056] in, Pseudo-training images The integration of soft tags, The weighting coefficients for the main teacher model. The main teacher model uses pseudo-training images Personalized model inference output information generated through inference and prediction. This represents the number of local teacher models of the neural network received by the server. For the first A neural network local teacher model uses pseudo-training images Personalized model inference output information generated through inference and prediction.

[0057] When generating a globally shared student model after distillation training of the basic student model, the following steps are included:

[0058] The basic student model is trained using a pseudo-supervised training dataset. During model training, the pseudo-training images in each pseudo-supervised training sample are loaded into the basic student model. After the basic student model performs inference and prediction, the inference output information of the student model is generated.

[0059] After each round of model training, for each student model inference output information, the training divergence between the student model inference output information and the corresponding fused soft label is calculated, and the calculated training divergence is used as the distillation loss. Thereafter, the base student model is updated with the optimization objective of minimizing the distillation loss.

[0060] The updated basic student model was tested using the student training test set, and the corresponding test accuracy was determined after the test.

[0061] After training the basic student model to the target number of training rounds, determine the highest test accuracy and use the basic student model corresponding to the highest test accuracy as the globally shared student model.

[0062] The advantages of this invention are: the client searches for and generates the optimal matching neural network local teacher model through PSO optimization, the server constructs a pseudo-supervised training dataset based on the neural network local teacher model uploaded by the client, and uses the pseudo-supervised training dataset to train the basic student model, thereby obtaining a globally shared student model; the federated learning method of this invention achieves the following technical effects:

[0063] Enhanced personalization capabilities: Since each client autonomously searches for and generates the best-matching local neural network teacher model based on its own local private data distribution, it can more accurately fit the feature distribution of local private data, significantly improve the performance and generalization ability of the local neural network teacher model, avoid the performance bottleneck caused by homogeneous architecture, and effectively support the search and generation of heterogeneous model architecture.

[0064] Significantly reduced communication costs: Federated learning adopts a "one-time communication" mechanism, which only requires the client to send the local teacher model of the neural network to the server for data communication. After that, the server can independently complete the training and generation of the globally shared student model, eliminating the need for multiple rounds of parameter synchronization and iteration required by traditional federated learning, and greatly reducing communication overhead and training cycle.

[0065] Achieving knowledge aggregation without real data: The server does not need to access any client's local private data, nor does it need to rely on public datasets. It only uses the local teacher model of the neural network uploaded by the client to generate pseudo-training images and constructs fused soft labels corresponding to each pseudo-training image, thus avoiding the risk of real data leakage and enhancing the system's privacy protection capabilities.

[0066] In summary, this invention achieves efficient cross-structure knowledge integration and basic student model training under the premise of heterogeneous local teacher model structure in neural networks and no server access to any real local private data in the client. It has technical advantages such as low communication cost, strong personalization capability, good privacy protection, and excellent model generalization. It solves the technical problem in existing federated learning methods that cannot take into account model heterogeneity, knowledge fusion and privacy constraints. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating one embodiment of the federated learning method of the present invention.

[0068] Figure 2 This is a schematic diagram of one embodiment of the federated learning system of the present invention. Detailed Implementation

[0069] The present invention will be further described below with reference to specific accompanying drawings and embodiments.

[0070] To effectively support heterogeneous models on clients, improve personalization capabilities and privacy protection, and reduce communication costs, this invention provides a federated learning method that supports heterogeneous model architecture search and zero-shot knowledge fusion. Specifically, the federated learning method includes:

[0071] A federated learning system is provided for federated learning, wherein the federated learning system includes a server and a client group adapted to and connected to the server. The client group includes several clients, each client having local private data, and the local private data of all clients has the characteristic of non-independent and identically distributed.

[0072] Configuring a federated learning system for federated learning includes at least configuring each client to perform teacher model generation processing and configuring the server to perform student model generation processing.

[0073] When any client performs teacher model generation processing, it searches for and generates a neural network local teacher model that best matches the client based on the client's local private data, and sends the generated neural network local teacher model to the connected server.

[0074] After the server performs the student model generation process, it generates at least a globally shared student model. The generation of the globally shared student model includes:

[0075] Based on all received neural network local teacher models, a pseudo-supervised training dataset is generated. The pseudo-supervised training dataset includes several pseudo-supervised training samples. Each pseudo-supervised training sample includes a pseudo-training image that can reproduce the knowledge representation of the corresponding client and a fused soft label corresponding to the pseudo-training image. When generating each pseudo-supervised training sample, the server does not access the local private data of the corresponding client in the client group.

[0076] A basic student model is constructed by distillation training using the pseudo-supervised training dataset, and a globally shared student model is generated after distillation training of the basic student model.

[0077] Figure 1 The diagram shows a flowchart of an embodiment of the federated learning of the present invention. As can be seen from the diagram, the federated learning of the present invention should be based on a federated learning system, that is, federated learning should be performed through the provided federated learning system. Figure 2 The diagram illustrates one embodiment of a federated learning system. As shown in the diagram, the federated learning system should include a server and a client group, with the client group including at least one client. Figure 2 The diagram shows that the client group includes four clients. All clients in the client group should be compatible with the server. Specifically, the compatibility between the client and the server means that data communication can be realized between the client and the server. The specific method of realizing data communication can be selected as needed.

[0078] It should be noted that the server and client can adopt existing commonly used forms, depending on whether they can meet the requirements for performing federated learning, which will not be elaborated here. In specific implementation, each client stores local private data, and the local private data of all clients has the characteristic of non-independent and identically distributed, that is, the local private data in different clients are significantly different. For example, one client mainly contains local private data of a few categories such as "cat" and "dog", while another client contains local private data of completely different categories such as "airplane" and "car", and the number of categories, image resolution, texture complexity and other characteristics of the local private data are also inconsistent.

[0079] Depend on Figure 1 Therefore, when conducting federated learning, each client should be configured to perform teacher model generation processing, such as... Figure 2When there are four clients in the client group, all four clients execute the corresponding teacher model generation process. Specifically, when each client executes the teacher model generation process, it should search for and generate the optimal matching neural network local teacher model for that client based on the local private data within the client. For example, for clients with local private data in a few categories such as "cat" and "dog", the corresponding neural network local teacher model should be searched for and generated based on the category data of "cat" and "dog" in the corresponding local private data. It should be understood that the generated neural network local teacher model can perform task processing on the local private data within the corresponding client. The task processing capability can be classification task processing, regression task processing, etc. In this invention, classification task processing is used as an example for explanation. Other cases can be referred to the corresponding explanation, and will not be illustrated one by one here.

[0080] It should be understood that, due to the non-independent and identically distributed nature of the local private data of different clients, and since each client only uses its own local private data to search and generate a neural network local teacher model, the neural network local teacher models generated by different clients can have different network architectures. That is, this invention can support client groups generating heterogeneous neural network local teacher models. Therefore, the federated learning of this invention can support the search for heterogeneous model architectures. The method and process of searching and generating neural network local teacher models based on local private data will be described in detail below.

[0081] Since the client and server have data communication capabilities, after generating the local teacher model of the neural network within the client, the generated local teacher model should be sent to the server. Each time the client sends the local teacher model to the server, a data communication is completed between the client and the server. It should be noted that when executing the federated learning method of this invention, each client only needs to communicate with the server once; that is, data communication is only required when sending the local teacher model of the neural network to the server. Therefore, this invention can effectively reduce network overhead and communication costs in federated learning.

[0082] Once all clients have sent their corresponding local neural network teacher models to the server, the server should be configured to perform student model generation processing. After performing student model generation processing, at least a globally shared student model can be generated. That is, after generating the globally shared student model within the server, the federated learning of this invention is completed.

[0083] Because the globally shared student model is generated based on all local teacher models of the neural networks, it has the advantages of unified structure, strong generalization ability, and no reliance on local private data within the client. In practical deployments, the globally shared student model can be used in the following two typical scenarios:

[0084] 1) Deploy the globally shared model directly to new clients: For newly joined clients, the server can directly distribute the globally shared student model to the newly joined clients to provide an immediately usable neural network model, avoid retraining, and improve deployment efficiency. The newly joined clients specifically refer to clients that did not participate in the process of generating the globally shared student model, such as clients that did not participate in the above-mentioned teacher model generation process.

[0085] 2) As an initial model for personalized fine-tuning: For clients with existing local private data, the globally shared student model can be used as the initial weight model. Subsequently, the globally shared student model, which is the initial weight model, can be further fine-tuned using local private data, thereby quickly adapting to the local data distribution and improving the performance of the client.

[0086] In addition, the globally shared student model can also be used in other scenarios. The specific applications are consistent with existing technologies, and will not be listed here.

[0087] To generate a globally shared student model, after receiving all the local teacher models of the neural network, a pseudo-supervised training dataset should be generated based on the received local teacher models, such as... Figure 2 When the client group includes four clients, the four clients will send their corresponding neural network local teacher models to the server. Subsequently, the server will generate a pseudo-supervised training dataset based on the four neural network local teacher models. The method and process of generating the pseudo-supervised training dataset will be explained in detail below.

[0088] The pseudo-supervised training dataset may include several pseudo-supervised training samples, the number of which can be selected as needed. Specifically, each pseudo-supervised training sample includes at least one pseudo-training image that can reproduce the knowledge representation of the corresponding client and a fused soft label corresponding to the pseudo-training image. The fused soft label is the label of the pseudo-training image. For example, if the task processing capability mentioned above is a classification task, the fused soft label should represent the classification category state within the pseudo-training image. It should be noted that reproducing the knowledge representation of the corresponding client specifically refers to the characteristics of the pseudo-training image including the knowledge of the corresponding client; that is, when generating the pseudo-training image, the knowledge features of the corresponding client can be captured. Furthermore, when generating pseudo-supervised training samples on the server, the server does not access the local private data of the corresponding clients within the client group, thereby improving personalization capabilities and privacy protection. Specifically, the corresponding client here refers to the client whose knowledge representation has been reproduced by the image trained in the pseudo-supervised training sample.

[0089] To generate a globally shared student model, a basic student model should first be built on the server. The details of the basic student model will be explained below. After building the basic student model, it can be trained using the pseudo-supervised training dataset through distillation. The globally shared student model is then generated after distillation training of the basic student model. The method and process of distilling the basic student model using the pseudo-supervised training dataset will be explained in detail below.

[0090] In one embodiment of the present invention, when each client performs teacher model generation processing, it includes:

[0091] Construct a network particle swarm, wherein the network particle swarm consists of several network particles, each of which contains a network encoding sequence and represents a candidate convolutional neural network based on the corresponding network encoding sequence, and the length of the network particle is the network depth of the candidate convolutional neural network it represents.

[0092] Based on the local private data of the current client, the above-constructed network particle swarm is optimized by PSO (Particle Swarm Optimization) to obtain the optimal candidate convolutional neural network corresponding to the optimal network particle.

[0093] The optimal candidate convolutional neural network is trained based on the local private data of the current client, and a local teacher model of the neural network is generated after the model is trained.

[0094] To search for and generate a local teacher model for a neural network based on local private data within each client, a network particle swarm should be constructed. Generally, a network particle swarm consists of several network particles, each containing a network encoding sequence. This network encoding sequence is used to represent a candidate convolutional neural network, and the length of the network particle is equal to the network depth of the candidate convolutional neural network it represents. Thus, a corresponding candidate convolutional neural network can be obtained based on the network encoding sequence of each network particle. In other words, the network encoding sequence mainly records the layer type (such as convolution, pooling, or fully connected layers), layer state information (if the layer type is a convolutional layer, the layer state information may include the kernel size), and the number of output channels of the candidate convolutional neural network.

[0095] It should be noted that when the above-mentioned task processing capability is a classification task, the candidate convolutional neural network can perform classification processing on the local private data within the client. Generally, the local private data is image data, that is, the candidate convolutional neural network can identify the category of each image within the client.

[0096] To ensure efficient operation of the search on the client side, this invention provides a set of representative search space parameter configurations, specifically:

[0097] 1) Given that edge clients generally have limited computing power and storage constraints, this invention controls the layer type, layer state, and high number of output channels of candidate convolutional neural networks within the network encoding sequence. Specifically, the maximum number of output channels of a convolutional layer is limited to 128, and the maximum number of neurons in a fully connected layer is set to 300, in order to avoid generating network structures with excessively high computational complexity that are not conducive to deployment. It is understood that the control implemented in this invention should ensure that the candidate convolutional neural network can still classify local private data.

[0098] 2) To guide network particles in searching within an effective architectural space, the number of network layers is limited to between 3 and 20 layers to balance the expressive power and training difficulty of the neural network. Furthermore, in terms of layer type sampling probabilities, convolutional layers, pooling layers, and fully connected layers are set at ratios of 0.6, 0.3, and 0.1, respectively, which enhances architectural diversity and maintains the stability of information abstraction capabilities; the size of the convolution sum does not exceed 7*7 to capture image features at different scales.

[0099] 3) The population size of the network particle swarm optimization is set to 20 (at this time, the number of network particles is 20), and the total number of search rounds for PSO optimization is set to 10, with 50 iterations per round. The aim is to reduce the training burden and search time on the client side while ensuring the convergence of the network particle swarm optimization search. In addition, after each network particle update, the candidate convolutional neural network is trained locally for only 10 epochs to evaluate fitness, while the final optimal candidate convolutional neural network is trained for 50 epochs to fully learn local knowledge, after which a local teacher model of the neural network can be generated.

[0100] It should be noted that the search space parameter configurations described above demonstrated good convergence speed and accuracy in actual tests. Of course, these search space parameter configurations can be adjusted as needed, dynamically based on factors such as the client's computing power and memory capacity in the deployment environment. For example, on GPU nodes with more abundant resources, the population size of the network particle swarm optimization, the depth of the candidate convolutional neural network, and the upper limit of the number of channels can all be expanded accordingly; further examples will not be provided here.

[0101] In specific implementation, after determining the above-mentioned search space parameter configuration, PSO optimization can be performed on the constructed network particle swarm based on the local private data of the current client to obtain the optimal candidate convolutional neural network corresponding to the optimal network particle. Subsequently, the optimal candidate convolutional neural network is trained to generate a local teacher model of the neural network. The process of PSO optimization and model training of the optimal candidate convolutional neural network is explained in detail below.

[0102] In one embodiment of the present invention, at least an optimized local training set and an optimized local validation set are constructed based on local private data within each client;

[0103] First, the network encoding sequence corresponding to the network particles in the network particle swarm is converted into the corresponding candidate convolutional neural network. Then, PSO optimization is performed on the network particle swarm. The PSO optimization includes at least several sequential network training optimization processes, wherein each network training optimization process includes sequential optimization training processes and particle update processes.

[0104] During the optimization training process, the candidate convolutional neural networks are trained on the target round using the local optimization training set, so as to generate the corresponding candidate convolutional neural networks after model training.

[0105] When performing particle update processing, each candidate convolutional neural network is validated using the optimization validation set, and the network accuracy calculated by the network validation is used as the fitness value of the current candidate convolutional neural network.

[0106] Based on the fitness value of each candidate convolutional neural network, the individual optimal convolutional neural network corresponding to each candidate convolutional neural network is determined, and the global optimal convolutional neural network is determined based on all the individual optimal convolutional neural networks.

[0107] When the PSO optimization of the network particle swarm fails to reach the target state, each network particle is updated based on the global optimal convolutional neural network and the individual optimal convolutional neural network of each network particle to update the network encoding sequence of each network particle and perform the next network training optimization process.

[0108] When the PSO optimization of the network particle swarm reaches the target state, the network particle corresponding to the globally optimal convolutional neural network is taken as the optimal network particle, and the globally optimal convolutional neural network configuration is taken as the optimal candidate convolutional neural network.

[0109] As explained above, local private data generally consists of image data. Furthermore, local private data includes labels corresponding to each image. For example, when the network task mentioned above is a classification task, the image label represents the category corresponding to the image. An image and its label constitute a local private sample. To meet the PSO optimization requirements, the local private data can be divided into an optimization local training set and an optimization local validation set. Both the optimization local training set and the training local validation set can include several local private samples. The local private samples within the optimization local training set and the training local validation set are completely unique. The specific local private sample configuration can be determined as needed, and will not be elaborated further here.

[0110] When the total number of search rounds in the above search space parameter configuration is 10, the PSO optimization should include 10 sequential network training optimization processes. Each network training optimization process may include sequential optimization training and particle update processing. The statement "the total number of search rounds for PSO optimization is set to 10, with 50 iterations per round" specifically means that the total number of PSO optimization rounds is 10, and within each PSO optimization round, the above-mentioned network training optimization process should be executed 50 times, that is, 50 rounds of optimization training. In each round of optimization training, the model training for the target round should be performed for each candidate convolutional neural network, and the target round can be 10 rounds. The above description explains how to perform 10 rounds of optimization training.

[0111] Understandably, when performing the first network training optimization process, the initial value of the network encoding sequence for each network particle should be given. In order to ensure that the candidate convolutional neural network has a certain diversity and meets the constraints such as size and number of layers, the initial value of the network encoding sequence can be given by random initialization, which can determine the initial state of each network particle.

[0112] The following example illustrates the method and process of randomly initializing the network coding sequence of network particles. Specifically:

[0113] 1) Initialization parameter planning,

[0114] min_layer, max_layer: used to constrain the number of layers in the candidate convolutional neural network. Specifically, based on the search space parameter configuration above, min_layer is 3 and max_layer is 20.

[0115] conv_prob: The probability of generating a convolutional layer, set to 0.6, is used to guide the candidate convolutional neural network to prioritize the use of convolutional layers to extract image features;

[0116] pool_prob: The probability of generating a pooling layer, set to 0.3, used to achieve feature map size compression and dimensionality reduction;

[0117] fc_prob: The probability of generating a fully connected layer, set to 0.1, used for high-dimensional feature integration and classification at the end of the network;

[0118] max_conv_kernel: Maximum convolutional kernel size, set to 7, used to limit the receptive field size of convolution operations;

[0119] max_out_ch: The maximum number of output channels for the convolutional layer, set to 256, is used to control the model capacity and complexity;

[0120] `max_pool_layers`: The maximum number of pooling layers, dynamically determined based on the input image size, ensuring that the feature map retains effective spatial resolution after multiple pooling operations.

[0121] max_fc_neurons: The maximum number of neurons in a single fully connected layer. Set to 300 to limit the expression scale of the fully connected layer.

[0122] 2) Network particle initialization mechanism

[0123] The initialization process of each network particle follows the principle of hierarchical progression and probability-driven structural sampling. Specifically, within a given network depth range (controlled by min_layer and max_layer), a hybrid neural network architecture consisting of convolutional layers (Conv), pooling layers (Pooling), and fully connected layers (FC) is generated layer by layer, ultimately forming a candidate convolutional neural network with a valid structure and classification capabilities.

[0124] First, the first network layer of the network particles is forced to be a convolutional layer to ensure that the initial model has basic feature extraction capabilities. The number of output channels (ou_c) of the initial convolutional layer is randomly generated in the range of [16, max_out_ch], and the kernel size (kernel) is randomly sampled in the range of [3, max_conv_kernel].

[0125] Subsequently, in the remaining layers, the structure type is determined layer by layer using a probability-driven approach. Specifically, a random number in the interval [0,1] should be generated. The specific strategy is as follows:

[0126] If the sampled random number If so, add a new convolutional layer with random values ​​for both the number of channels and the kernel size;

[0127] like If so, add a pooling layer (max pooling or average pooling, with a kernel size of 2×2), provided that the current number of pooling layers does not exceed the set maximum number of pooling layers max_pool_layers;

[0128] like If so, add a fully connected layer, with the number of neurons randomly determined within the range of [1, max_fc_neurons].

[0129] Specifically, in order to maintain the logical coherence of the network structure, in one embodiment of the present invention, when a fully connected layer is generated for the first time in the candidate convolutional neural network, it indicates that the candidate convolutional neural network has entered the tail stage, and the type of subsequent layers is limited to fully connected layers until the candidate convolutional neural network is generated.

[0130] Furthermore, this invention introduces a mechanism for controlling the number of pooling layers. Specifically, considering that the feature map size will gradually shrink under continuous pooling operations, in order to prevent the generation of too many pooling layers from causing the feature map size to be too small and the network to be untrainable, continuous 2×2 pooling operations are simulated for the initial width value of the input image until the feature map width is lower than the minimum threshold (e.g., 4 pixels). This determines the maximum allowed number of pooling layers, max_pool_layers. This value will be embedded as a constraint condition during the network particle initialization stage to limit the number of pooling operations contained in each particle.

[0131] The candidate convolutional neural networks generated during the entire network particle initialization process are stored as a list in the `layers` attribute of each network particle. The `layers` attribute records the particle's type (convolutional, pooling, or fully connected) and key structural parameters. Simultaneously, "velocity" information corresponding to each layer is initialized to support structural evolution operations in subsequent particle swarm optimization. As explained above, the final layer of each network particle is ultimately forced to be a fully connected output layer, with its output dimension consistent with the classification task's objective dimension, thus ensuring the integrity and usability of the model structure.

[0132] It should be noted that the "velocity" information mentioned above specifically refers to the trend of changes that may occur in the candidate convolutional neural network during the subsequent PSO optimization process. It is usually represented by a set of structural change markers, such as keep, replace, remove, or add. By assigning initial velocity information to network particles, the swarm of network particles can have a certain direction of exploration during the first round of optimization, preventing all network particles from remaining in the initial structure and lacking diversity, thereby improving the coverage of the search space and convergence efficiency. During the PSO optimization process, the velocity information will be dynamically updated based on the difference between the individual best (pBest) and the swarm best (gBest) to continuously guide the particles towards a better network architecture.

[0133] Specifically, velocity information initialization is required to set the velocity information, with the aim of assigning initial evolution instruction labels to each layer of each network particle. The initialization strategy can be: marking convolutional and pooling layers as `keep` (maintain current structure), and fully connected layers as `keep_fc`, meaning their structure is not adjusted by default. When the last layer of a network particle is forcibly set as an output layer (FC), its output dimension is consistent with the number of task classification targets, ensuring the functional integrity and usability of the entire candidate convolutional neural network structure; the number of task classification targets is generally related to local private data within the client and can be determined based on that data.

[0134] Understandably, the initialization of the candidate convolutional neural network is now complete. For example, the initialized network particle structure is printed as follows:

[0135] Particle 0: conv | conv | pool | conv | fc |, where “conv | conv | pool | conv | fc |” is the network encoding sequence of a network particle, where conv represents a convolutional layer, pool represents a pooling layer, and fc represents a fully connected layer;

[0136] Particle 1: conv | conv | fc |, where “conv | conv | fc |” is the network encoding sequence of a network particle.

[0137] 3) Particle compilation mechanism

[0138] After initializing the network particles, the network encoding sequence (i.e., the list of layers attributes) of each network particle is converted into a trainable candidate convolutional neural network model ("structure compilation process"). In one embodiment, the above structure compilation can be implemented based on Keras; specifically, during structure compilation, the structure description of each layer is read sequentially, and the candidate convolutional neural network is constructed according to the following logic:

[0139] First, if the current layer is a convolutional layer (conv), add a Conv2D layer according to the recorded number of output channels ou_c and the kernel size. The first convolutional layer needs to specify the input shape (input_shape=(w, h, c)), while subsequent layers omit this. Each convolutional layer is followed by a ReLU activation function, BatchNormalization, and Dropout to enhance training stability and prevent overfitting.

[0140] If the current layer is a pooling layer (max_pool or avg_pool), then add MaxPooling2D or AveragePooling2D layers respectively, using a 2x2 kernel size and stride by default to maintain the consistency of feature map downsampling.

[0141] For a fully connected layer (fc), first determine whether a flattened layer needs to be added before it (if the previous layer is not fully connected, the feature map needs to be flattened); then add a dense layer, the number of neurons of which is given by ou_c, and introduce Dropout and L2 regularization to improve generalization ability. If the current layer is the last layer of the network, the Softmax activation function is used to output the classification probability; otherwise, the ReLU activation function is used, and BatchNormalization is added.

[0142] Finally, the candidate convolutional neural network was compiled using the Adam optimizer (learning rate 0.001), with categorical crossentropy as the loss function and accuracy as the evaluation metric, thus completing the formal construction and training preparation of the candidate convolutional neural network.

[0143] It should be noted that the Keras implementation described above can be achieved through the Keras interface within the TensorFlow framework. Furthermore, when compiling using the Adam optimizer (learning rate of 0.001), compilation can be performed through the Adam module within the Keras interface.

[0144] The above provides an example of initializing each network particle and generating a candidate convolutional neural network based on the network particle's network encoding sequence. Of course, there are other ways to convert the network encoding sequence and the candidate convolutional neural network, which will not be illustrated here.

[0145] In practice, during the optimization training process, the candidate convolutional neural networks generated by each transformation should be pre-trained using the optimization training set, and the cross-entropy loss function should be used as the pre-training loss function. The pre-training loss function is as follows:

[0146]

[0147] in, For pre-training loss, To find the optimal training set, The number of optimal training samples within the optimal training set; To optimize the number of categories in the training set; For candidate convolutional neural networks to optimize training samples Prediction belongs to the first The predicted probability of a class; Let be the indicator function, representing the search for optimal training samples. tags Is it the first kind.

[0148] Understandably, the optimization training set includes several optimization training samples, which consist of optimization training images and corresponding training image labels. The candidate convolutional neural network optimizes the training samples. Prediction, specifically, refers to the process by which candidate convolutional neural networks optimize training samples. Internal optimization training image prediction, optimization of training samples tags Specifically, this refers to optimizing training samples. The training image labels within.

[0149] Generally, within each client, candidate convolutional neural networks generated from each network particle in the network particle swarm are pre-trained. To adapt to the resource constraints of edge clients, as described above, this invention employs a lightweight local training strategy for each candidate convolutional neural network model: the number of training rounds is fixed at 10, the batch size is set to 32, and the optimizer can be either Adam or SGD, to quickly evaluate the performance of candidate convolutional neural network models in scenarios with limited computational resources. After each round of pre-training, candidate convolutional neural networks are generated. Subsequently, the parameters of these candidate convolutional neural networks are updated based on local task data for use in subsequent fitness evaluation and structural evolution.

[0150] After pre-training, the accuracy of the candidate convolutional neural networks after pre-training is calculated using the optimization validation set, and used as the fitness score of the candidate convolutional neural network corresponding to the current network particle. The fitness function is as follows:

[0151]

[0152] in, This represents the search for the optimal validation set. This represents the number of optimization validation samples within the optimization validation set. For the first Each network particle corresponds to a fitness value of the candidate convolutional neural network. Indicates the first Each network particle corresponds to a candidate convolutional neural network for optimizing and validating samples. The prediction results This is an indicator function that indicates whether the prediction result is label b.

[0153] The number of optimization verification samples can be referenced from the number of optimization training samples mentioned above. It is understood that the optimization verification samples should be completely different from the optimization training samples, that is, they should be formed based on different data content of local private data within the client.

[0154] In one embodiment of the present invention, based on the fitness score of each candidate convolutional neural network, the individual optimal convolutional neural network corresponding to each candidate convolutional neural network can be determined, and the global optimal convolutional neural network can be determined based on all the individual optimal convolutional neural networks. The specific methods for determining the individual optimal convolutional neural network and the global optimal convolutional neural network can be consistent with the prior art. For example, when performing the first network training optimization process, each candidate convolutional neural network after pre-training is used as the individual optimal convolutional neural network for each network particle, and the candidate convolutional neural network with the highest fitness score is used as the global optimal convolutional neural network. The methods for determining the individual optimal convolutional neural network and the global optimal convolutional neural network after subsequent network training optimization processes can be referred to the description here.

[0155] When performing particle update processing, the network encoding sequence within each network particle should be updated based on the individual optimal convolutional neural network and the global optimal convolutional neural network. During the update, the common update calculation method of the PSO algorithm can be used to ensure that each network particle can both move towards its previously best-performing convolutional neural network and be guided by the currently best convolutional neural network in the group, thereby achieving a balance between personalized and collaborative search. The following provides an example illustrating the method and process of network particle updating:

[0156] The core idea of ​​network particle updates is to gradually guide network particles toward a better structure by simulating swarm intelligence behavior, thereby achieving adaptive optimization of the network architecture. Specifically, in each iteration, the structural change rate of each network particle needs to be calculated first. This rate vector is determined by the difference between the current network particle and its individual best (pBest) and swarm best (gBest), and is used to guide the structural adjustment of the candidate convolutional neural network.

[0157] When calculating structural differences, the network is divided into convolutional / pooling layers and fully connected layers, which are compared separately to generate difference vectors. Specifically: for convolutional / pooling layers, if the current network particle and the reference structure (such as individual best pBest or group best gBest) both exist at the same position and have the same layer type, it is marked as "keep" in the difference vector; if both have layers at that position but of different types, it is marked as "replace", indicating that the entire layer needs to be replaced with the corresponding layer in the reference structure during subsequent updates; if the current network particle has an extra layer at that position but the reference structure does not have a corresponding layer, it is marked as "remove"; if the current network particle is missing the layer but the reference structure exists, it is marked as "add", indicating that the layer needs to be added.

[0158] For fully connected layers, since the structure is usually located at the end of the network, a reverse alignment method is used for comparison: if there are fully connected layers at corresponding positions, it is denoted as "keep_fc"; if the current particle has an extra layer, it is denoted as "remove_fc"; if the current particle is missing a layer, it is denoted as "add_fc". It should be noted that the output layer is always forcibly retained, and its output dimension remains consistent with the number of task categories, and it does not participate in the difference operation. The difference vector obtained in the above way is essentially a set of structure evolution instructions (such as keep, replace, delete, supplement, etc.) used to guide subsequent structure velocity generation and particle updates.

[0159] Furthermore, if the structure of the current network particles is completely consistent with both the individual optimal pBest and the swarm optimal gBest, no further difference calculation is needed; instead, a random selection is made between the two to generate a preservation update instruction. If a difference exists, the structural information of the corresponding layer is selected from either the swarm optimal gBest or the individual optimal pBest with a certain probability, based on a preset global learning factor (e.g., 0.7). If a layer does not exist in the target structure, a deletion instruction is generated. Through this method, a set of structural evolution instructions (i.e., structural velocity) can be formed, including operations such as preservation, replacement, or pruning.

[0160] Specifically, the target structure mentioned above refers to the set of structural features of the individual optimal pBest and the group optimal gBest, i.e., the reference structure that the current network particle needs to align with or approach. If a layer does not exist in the target structure but exists in the current network particle, the system generates a deletion instruction (e.g., {type:remove} or {type:remove_fc}) indicating that the layer should be pruned. Conversely, if a layer exists in the target structure but is missing in the current network particle, an addition instruction is generated to supplement the layer structure at the corresponding position. The final structure evolution instruction (i.e., structure velocity) may include operations such as maintaining, replacing, adding, or deleting.

[0161] After receiving the structural evolution instructions, they are applied to the structural encoding sequence of the current network particles to obtain the updated network structure, and new candidate convolutional neural networks are generated accordingly. Thus, after performing optimization training, the candidate convolutional neural networks, after model training, can generate candidate convolutional neural networks. After particle updates, the candidate convolutional neural networks are regenerated into corresponding candidate convolutional neural networks. Subsequently, the updated candidate networks are trained based on local training data on the client side to obtain accuracy as the fitness value. If the fitness is better than its historical best performance, the individual optimal value of that particle is updated; if it also exceeds the swarm optimal value, the swarm optimal value is updated and used as the candidate convolutional neural network for the next round of optimization.

[0162] After updating the new convolutional neural network using the above method, the network particle structure is converted into the corresponding candidate convolutional neural network. As explained above, the converted candidate convolutional neural network should be trained for 10 epochs based on the client's local training data to obtain its accuracy as the fitness value. If the accuracy of the current network particle is better than its historical best performance, its individual optimal pBest structure and fitness are updated; if its accuracy also exceeds the current group optimal gBest fitness, the globally optimal structure is updated. Ultimately, this mechanism guides the network particles to continuously evolve towards a better neural network architecture in the structure space.

[0163] It should be noted that when the preset number of iterations is reached, or when the network particle swarm converges as a whole, the client will use the network particle corresponding to the globally optimal convolutional neural network as the optimal network particle, and the globally optimal convolutional neural network configuration as the optimal candidate convolutional neural network. It should be understood that once the optimal candidate convolutional neural network for the current client is obtained, model training should be performed on the optimal candidate convolutional neural network so that the corresponding local teacher model of the neural network can be generated after model training.

[0164] Specifically, when training the optimal candidate convolutional neural network, a teacher model training set and a teacher model validation set should be formed based on the local private data within the current client. The optimal candidate convolutional neural network can be trained using the teacher model training set, and the optimal candidate convolutional neural network can be validated using the teacher model validation set. The details of the teacher model training set and the teacher model validation set can be referred to the corresponding descriptions of the optimization of the local training set and the optimization of the local validation set mentioned above, which will not be repeated here.

[0165] It should be understood that when training the optimal candidate convolutional neural network using the teacher model training set, the specific training process can be consistent with existing technologies, such as using supervised learning to perform standard training on the model. The model training process includes conventional steps such as forward propagation, loss calculation, gradient backpropagation, and parameter updates. In one feasible embodiment, the parameters for supervised learning may include: the loss function used for training is the cross-entropy loss function; the optimizer is the Adam optimizer; parameters such as the learning rate and batch size can be flexibly configured according to the client's resource availability; and the number of training rounds is set by the client. As explained above, the number of iterations can be 50 rounds to ensure that the model fully converges and accurately learns the distribution characteristics of the local private data.

[0166] In practice, after training the optimal candidate convolutional neural network (CNN) once using the teacher model training set, the trained CNN is validated using the teacher model validation set to obtain the corresponding teacher model validation accuracy. Once the number of training rounds for the optimal candidate CNN reaches a specified number, the CNN with the highest teacher model validation accuracy is selected as the local teacher model for the neural network. When the local teacher model is used for a classification task, the teacher model validation accuracy, specifically the probability of accurately classifying the teacher model validation set, is calculated using methods consistent with existing techniques and will not be elaborated here.

[0167] As explained above, the process of generating the local neural network teacher model within each client does not involve communication with other clients or servers, ensuring that local private data is always protected on the client's local machine. After training, the client obtains a local neural network teacher model with an optimal local architecture and fully trained weights, capable of representing the client's local private data. After obtaining the local neural network teacher model, the client saves it as a model file and uploads it to the server in one go via network transmission.

[0168] As explained above, each neural network local teacher model has an independent network architecture and parameters. Therefore, neural network local teacher models uploaded by different clients may be heterogeneous. Furthermore, each neural network local teacher model maintains an optimal match with its respective client; that is, the best-performing network model is trained only on the local private data of its client. For details on the optimal matching process, please refer to the above explanation of the neural network local teacher model generation process.

[0169] In one embodiment of the present invention, generating a pseudo-supervised training dataset includes:

[0170] Within the server, based on the local teacher model of each neural network, a pseudo-training image generation mechanism optimized by random noise input is used to generate several pseudo-training images, wherein each pseudo-training image satisfies the requirement to reproduce the knowledge representation of the corresponding client.

[0171] Each pseudo-training image is loaded into each neural network local teacher model within the server to configure the neural network local teacher model to perform inference prediction on the loaded pseudo-training images and generate personalized model inference output information.

[0172] The server will perform consistency-aware fusion processing on all corresponding personalized model inference output information for each pseudo-training image to generate fused soft labels after consistency-aware fusion processing.

[0173] The generated fused soft labels and corresponding pseudo-training images are used as pseudo-supervised training samples in the pseudo-supervised training dataset.

[0174] It should be understood that generating a pseudo-supervised training dataset specifically refers to generating pseudo-supervised training samples within the pseudo-supervised training dataset. In one embodiment of the present invention, when generating pseudo-supervised training samples, the server first generates pseudo-training images based on the received neural network local teacher model, and then generates corresponding fused soft labels based on the pseudo-training images. The method and process of generating pseudo-training images and fused soft labels are described in detail below.

[0175] Since the server cannot access the local private data on the client, in order to reproduce and extract the local private data on the client, this invention designs a pseudo-image generation mechanism based on random noise input optimization to reconstruct the semantic feature information contained in the local teacher model of the corresponding uploaded neural network on the client.

[0176] It should be noted that when generating pseudo-training images using a pseudo-image generation mechanism optimized with random noise input, a set of random noise images should first be initialized for the corresponding neural network local teacher model. Specifically, each random noise image is generated by sampling from a normal distribution with a mean of 0 and a variance of 1. The normal distribution has the "maximum entropy" property in statistics, representing the most neutral distribution of random variables in the absence of prior knowledge, making it very suitable as the initial state for the pseudo-training image generation process. Random noise images generated using the normal distribution do not contain any specific structure and only appear as pure noise, but they will be gradually adjusted to images with semantic features during subsequent optimization. Compared to a uniform distribution, the normal distribution can generate more pixels concentrated near the image's gray-level mean, more closely resembling the basic statistical characteristics of natural images.

[0177] It should be noted that the feature dimensions of local private data across different clients should be consistent, and the feature dimensions of the generated random noise image should be consistent with the feature dimensions of data across all clients. Therefore, after determining the feature dimensions, a set of random noise images can be generated using the method described above. The feature dimensions of the random noise image are the feature size and the number of channels.

[0178] In one embodiment of the present invention, when generating pseudo-training images using a pseudo-image generation mechanism optimized with random noise input for each neural network local teacher model, the process includes:

[0179] Extract the task layer weight information of the local teacher model of the neural network, and construct a category similarity matrix based on the extracted task layer weight information;

[0180] Based on the constructed category similarity matrix, a category-aware pseudo-label modeling method is used to construct the corresponding soft label distribution group. The soft label distribution group includes several soft label distribution vectors. When the neural network local teacher model makes a classification prediction on the pseudo-training image, the number of distribution probabilities in the soft label distribution vector is consistent with the number of categories predicted by the neural network local teacher model.

[0181] The server initializes a set of random noise images for the current neural network local teacher model, and optimizes and updates the random noise images within the set of random noise images using each soft label distribution vector, so as to generate a corresponding pseudo training image with class discrimination after optimization and update.

[0182] Specifically, the task layer of a neural network local teacher model refers to the network layer directly used to complete task prediction. For example, when a neural network local teacher model performs a classification task, the task layer of the neural network local teacher model specifically refers to the last fully connected classification layer of the neural network local teacher model. When the network task is other, the situation of the task layer within the neural network local teacher model can be determined, which will not be elaborated here.

[0183] The extracted task-level weight information can be in matrix form, such as a weight matrix. ,in, The number of categories that can be identified when performing a classification task. This represents the input feature dimension for the classification task, which is also the dimension of the feature vector input to the layer preceding the task layer. In fact, it is related to the structure of the neural network local teacher model. More specifically, it is the dimension of the feature vector output of the penultimate layer (i.e., the layer before the classification layer) in the current neural network local teacher model. Therefore, for a given neural network local teacher model, the input feature dimension can be determined. Specifically, the weight matrix The first OK Vectors can be viewed as categories The representation vector is used to express the category. Semantic orientation in feature space.

[0184] When constructing a category similarity matrix based on task-level weight information, specifically referring to constructing a similarity matrix based on the weight matrix... Constructing the category similarity matrix, we have:

[0185]

[0186] in, Category similarity matrix Inner Line 1 The category similarity element of the column, Weight matrix The The vector of rows, Weight matrix The The vector of rows, Representing vectors with vector Dot product, For vectors The L2 norm, For vectors L2 norm and vector Modular multiplication of the L2 norm.

[0187] As can be seen from the above explanation, the category similarity matrix is: That is, the category similarity matrix is ​​an L-order square matrix, and the category similarity matrix can be determined using the above method. Each element within the matrix can be used to construct a category similarity matrix. The category similarity matrix measures the directional similarity between different categories. For the element within the category similarity matrix... Rows represent categories Semantic similarity with all other categories.

[0188] After constructing the category similarity matrix, a category-aware pseudo-label modeling method can be used to construct the corresponding soft-label distribution group. Since the category similarity matrix corresponds to the local teacher model of the neural network, the corresponding soft-label distribution group here specifically refers to the soft-label distribution group that corresponds to the local teacher model of the neural network. The following provides an example illustrating the method and process of constructing the soft-label distribution group.

[0189] The Dirichlet distribution is a multidimensional continuous distribution defined on a probability simplex, often used to model the probability distribution of multiple classes. To generate pseudo-labels with discriminative and diverse characteristics, this invention introduces a diversity adjustment coefficient. Among them, the diversity adjustment coefficient The possible values ​​for are: Of course, the diversity adjustment coefficient The value of can also be other cases, when the diversity adjustment coefficient For other values, please refer to this explanation for how to generate soft label distribution groups.

[0190] In practice, it is based on the diversity adjustment coefficient. and category similarity matrix One row is used as the concentration parameter of the Dirichlet distribution, from which the Dirichlet distribution can be constructed. ,in, The concentration parameter of the Dirichlet distribution. Concentration parameters This determines the sparsity of the sampling labels. When the concentration parameter... When the values ​​of each dimension are small, the sampling results exhibit a sharp unimodal structure, and the label vector is closer to the one-hot format; while when the concentration parameter... When the values ​​of each dimension are large, the distribution of the sampling results is smoother, indicating that the label vector has uncertainty for multiple categories. Specifically, when the diversity adjustment coefficient is large... When the value is small (e.g., 0.1), the constructed soft labels are sparser, reflecting a strong preference for the target category, making them suitable for generating pseudo-images with strong classification signals; the diversity adjustment coefficient... When the value is large (e.g., 1.0), the soft label becomes smoother, which can induce the generation of pseudo-training images that fuse features from multiple categories through the diversity adjustment coefficient, thereby improving the diversity and generalization ability of the pseudo-training images.

[0191] As can be seen from the above description, this invention uses the Dirichlet distribution as the base distribution for generating soft-label distribution groups, and adjusts the diversity adjustment coefficient. It can achieve controllable sparsity and diversity of soft labels, thereby improving the diversity and expressiveness of pseudo-training images. For the first element in each category similarity matrix... OK and each diversity adjustment coefficient The server is distributed from the corresponding Dirichlet. A soft label distribution vector (soft labelY) is obtained through mid-sampling, which serves as the supervision target for subsequent image optimization and update stages.

[0192] In practical implementation, when the diversity adjustment coefficient The value is When, for each value, combine with the corresponding A corresponding Dirichlet distribution can be constructed. Afterwards, a soft label distribution vector (soft label Y∈R) can be obtained by sampling. L ), such as for When the diversity adjustment coefficient When the scalar value is 0.1, a Dirichlet distribution can be constructed. After this, a soft label Y can be obtained by sampling, and when the diversity adjustment coefficient... When the value is 1.0, a Dirichlet distribution can be constructed. After that, another soft label Y can be sampled. For other cases, please refer to the description. Examples will not be given one by one.

[0193] As explained above, the number of soft-label distribution vectors in a soft-label distribution group can be correlated with the diversity adjustment coefficient. The value of is related to the number of rows in the category similarity matrix. Furthermore, the number of probabilities distributed within the soft label distribution vector corresponds to the number of categories predicted by the local teacher model in the neural network, such as soft label Y∈R. L The number of rows and columns of the category similarity matrix are both L. For example, the soft label Y can be [0.05, 0.10, 0.70, 0.10, 0.05]. In this case, L is 5. Each element in the soft label Y represents the probability distribution of the corresponding category. For example, 0.70 represents the probability distribution of the third category. For details, please refer to this explanation.

[0194] As can be seen from the above explanation, based on the concentration parameter Construct a Dirichlet distribution, which is a multivariate distribution defined on an L-dimensional probability simplex. When sampling to generate a soft label Y, it represents the "mixed representation weights" of the current category on different semantic categories. The sampling result is a non-negative real vector (i.e., a probability distribution vector) that sums to 1.

[0195] After generating the soft label distribution group in the above manner, the server can initialize a set of random noise images for the current neural network local teacher model. The situation of each random noise image in the set of random noise images can be referred to the above description. After that, each random noise image can be optimized and updated using each soft label distribution vector, so as to generate a corresponding pseudo training image with class discrimination after optimization and update. That is, a random noise image can be optimized and updated using a soft label distribution vector, and a pseudo training image can be generated. The pseudo training image generated here corresponds exactly to the soft label distribution vector.

[0196] In one embodiment of the present invention, optimizing and updating each random noise image includes:

[0197] Random noise images are loaded into a neural network local teacher model, and the corresponding noise image logits output information is obtained through inference and prediction by the neural network local teacher model.

[0198] The logits output information of the noisy image is temperature scaled and then normalized by softmax to generate the predicted probability distribution of the noisy image.

[0199] The cross-entropy loss between each soft label distribution vector and the predicted probability distribution of the noisy image is used as the optimization objective. During the optimization process, the random noise image is updated by gradient descent until the cross-entropy loss, which is the optimization objective, is minimized, and then the corresponding pseudo-training image is generated.

[0200] Specifically, after generating a random noise image using the above method, a random noise image is loaded into a neural network local teacher model. Subsequently, the neural network local teacher model is used for inference prediction. As explained above, the inference prediction performed by the neural network local teacher model on the received random noise image can be a classification prediction of the random noise image. After inference prediction, the noise image logits output information can be obtained. It should be noted that the noise image logits output information is the raw prediction score generated by the neural network local teacher model at the output layer, which has not been normalized. It is usually transformed into a probability distribution through softmax normalization in the subsequent process. That is, the noise image prediction probability distribution can be generated after softmax normalization.

[0201] In one embodiment of the present invention, the logits output information of the noisy image is temperature scaled, and then input into a softmax function to convert it into a predicted probability distribution of the noisy image, wherein the predicted probability distribution of the noisy image can be expressed as:

[0202]

[0203] in, For the first A neural network local teacher model for random noise images The output information of the noisy image logits generated after inference and prediction. For temperature parameters, Predict the probability distribution for the corresponding noisy image.

[0204] It should be noted that when performing classification and prediction tasks, the temperature parameter... This is used to scale the logits output information of noisy images, thereby controlling the distribution characteristics of the softmax output probability. Specifically, a larger temperature parameter... This can effectively reduce the differences in predicted probabilities between different categories, thereby enhancing the inter-class structural information contained in the soft label and improving the stability and gradient smoothness of the model during pseudo-image generation. Temperature parameter The selection of is generally based on experience and can be flexibly adjusted according to task complexity and data characteristics, with a common value range of 2 to 10. It is understandable that after temperature scaling and conversion to a noisy image prediction probability distribution via the softmax function, the pseudo-training images can be more stably guided towards semantic space evolution when the uncertainty of the local teacher model output in the neural network is high. Therefore, temperature scaling is one of the key mechanisms in this invention for improving the quality and diversity of pseudo-training images.

[0205] After obtaining the predicted probability distribution of each random noise image, the cross-entropy loss between a soft label distribution vector and the predicted probability distribution of the noise image is used as the optimization objective. Gradient descent is then applied to update the random noise images. Specifically, with the corresponding model parameters of the local teacher model of the neural network frozen, the random noise image itself is used as the optimization variable. By minimizing the cross-entropy loss between its predicted probability distribution and the soft label Y, the gradient is backpropagated and the pixel values ​​within the random noise image are updated, thereby guiding it to gradually approach the semantic target direction represented by the soft label Y in the feature space. Then:

[0206]

[0207] in, Given a soft-label distribution vector, Cross-entropy loss is used to measure the predicted probability distribution between the soft label distribution vector Y and the output of the k-th neural network local teacher model. The differences between them; The final optimized and updated pseudo-training image is the pseudo-training image that conforms to the semantic prior.

[0208] Specifically, the optimization process of the cross-entropy loss between the soft label distribution vector and the predicted probability distribution of the noisy image is implemented using the standard backpropagation algorithm. After gradient descent updates, the pixel values ​​in the random noise image gradually shift towards the current soft label distribution vector in each iteration. When the corresponding semantic directions are close, and the cross-entropy loss, which is the optimization objective, is minimized, pseudo-training images with class discriminative power are "induced". .

[0209] As explained above, the generated pseudo-training image is associated with a random noise image, a soft label distribution vector, and a neural network local teacher model. Changing any one of these elements will generate a corresponding pseudo-training image. In other words, the neural network local teacher models generated by different clients are independent of each other when generating pseudo-training images. Repeating this generation process can produce multiple corresponding pseudo-training images. The server then aggregates all the pseudo-training images to generate a global pseudo-training image set.

[0210] It should be understood that although the server does not directly access the local private data within any client, these pseudo-training images are generated by reverse-engineering neural network local teacher models generated from different clients. The model parameters of these neural network local teacher models implicitly contain the statistical characteristics and classification decision boundaries of their training data. Therefore, the generated pseudo-training images can effectively reproduce the client's knowledge representation, providing a high-quality input foundation for subsequent distillation training. Furthermore, "its training data" here specifically refers to the local private data within the client used to generate the neural network local teacher model.

[0211] As can be seen from the generation process of each pseudo-training image, each pseudo-training image satisfies the reproduction of the knowledge representation of the corresponding client. Here, "the corresponding client" specifically refers to the client corresponding to the local teacher model of the neural network used to generate the current pseudo-training image.

[0212] As explained above, after generating pseudo-training images using the above method, fused soft labels can be generated using the pseudo-training images. Specifically, when generating a fused soft label, a pseudo-training image needs to be loaded into each neural network local teacher model in the server. That is, the pseudo-training image is loaded into the neural network local teacher models uploaded by all clients to the server. After that, each neural network local teacher model can be used to perform inference and prediction on the current pseudo-training image, and corresponding personalized model inference output information can be generated after inference and prediction.

[0213] As explained above, the inference prediction made by the neural network local teacher model on the pseudo-training image can be a classification prediction. When the inference prediction is a classification prediction, the generated personalized model inference output information is the classification prediction information for the current pseudo-training image, and the dimension of each personalized model output inference information is 1×L. It should be understood that multiple different personalized model inference output information can be generated based on a single pseudo-training image, and the number of personalized model inference output information is consistent with the number of neural network local teacher models received on the server.

[0214] For each pseudo-training image, multiple personalized model inference outputs can be generated through the above inference prediction. In view of the possibility that the local teacher model of the non-neural network may produce prediction discrepancies on specific pseudo-training images, in order to improve the reliability and adaptability of the subsequent generation of fused soft labels, the server performs consistency-aware fusion processing on all corresponding personalized model inference outputs to generate fused soft labels. The consistency-aware fusion processing method and process are explained in detail below.

[0215] In one embodiment of the present invention, when the server performs consistency-aware fusion processing on the personalized model inference output information corresponding to each pseudo-training image, it includes:

[0216] The server calculates the average output information of the personalized model corresponding to the personalized model inference output information of the current pseudo-training image, and generates a teacher consistency metric based on the calculated average output information of the personalized model.

[0217] When the teacher consistency metric is not greater than the consistency judgment threshold, the average output information of the personalized model is used as the fusion soft label. Otherwise, based on the main teacher fusion mechanism, all personalized model inference output information is output fused to generate the corresponding fusion soft label after output fusion.

[0218] Within the server, each pseudo-training image is input into all the received neural network local teacher models to obtain K corresponding personalized model inference outputs. Each neural network local teacher model generates a corresponding personalized model inference output after inference prediction. The personalized model inference output is a logits vector, representing its original prediction score for each category of the pseudo-training image. Given that differences among the neural network local teacher models may lead to inconsistent output scales, the server performs temperature scaling on all personalized model inference outputs. This involves dividing each set of logits by a positive temperature parameter and then feeding it into a softmax function to obtain K normalized prediction probability distributions. Each prediction distribution is considered a personalized model inference output. The method and process of temperature scaling are explained above and will not be repeated here.

[0219] Specifically, the server first calculates the teacher consistency metric. In one embodiment of the present invention, when calculating and generating the teacher consistency metric, the following is true:

[0220]

[0221] in, Pseudo-training images Teacher consistency metrics This represents the number of local teacher models of the neural network received by the server. For the first A neural network local teacher model uses pseudo-training images Personalized model inference output information generated through inference and prediction. The average output information of the personalized model To calculate the personalized model inference output information based on the average output information of the personalized model The divergence.

[0222] Understandably, teacher consistency metrics It can measure the difference between the inference predictions of the local teacher model and the average output information of the personalized model for each neural network, reflecting the difference between pseudo-training images. The consistency of inference predictions across all neural network local teacher models. If the distribution of personalized model inference output information is similar across all neural network local teacher models, then the teacher consistency metric is... Smaller indicates a pseudo-training image. Inference predictions are relatively consistent across multiple neural network local teacher models; conversely, if inference predictions differ significantly across multiple neural network local teacher models, then the teacher consistency metric... Relatively large.

[0223] A consistency judgment threshold can be preset within the server. This threshold can be an empirical value, such as 0.05. After calculating the teacher consistency metric, it can be compared with the consistency judgment threshold. If the output of the personalized model is positive, the average output information of the personalized model will be used as the fusion soft label; otherwise, the output information of all personalized models will be fused based on the main teacher fusion mechanism to generate the corresponding fusion soft label after output fusion. The threshold for consistency judgment.

[0224] It is understandable that when generating fused soft labels corresponding to each pseudo-training image in the above manner, it can adapt to the consistent performance of different pseudo-training images in inference prediction under all neural network local teacher models, thereby improving the semantic quality and training stability of fused soft labels.

[0225] In one embodiment of the present invention, when fusing all personalized model inference output information based on the master teacher fusion mechanism, the following steps are included:

[0226] The server selects one neural network local teacher model as the master teacher model and uses the remaining neural network local teacher models as assistant teacher models. The master teacher model is at least the neural network local teacher model with the most complex network structure.

[0227] When performing output fusion, we have:

[0228]

[0229] in, Pseudo-training images The integration of soft tags, The weighting coefficients for the main teacher model. The main teacher model uses pseudo-training images Personalized model inference output information generated through inference and prediction. This represents the number of local teacher models of the neural network received by the server. For the first A neural network local teacher model uses pseudo-training images Personalized model inference output information generated through inference and prediction.

[0230] After receiving all the local teacher models of the neural networks sent by the clients, the server can use commonly used techniques in this field to determine the network structure corresponding to each local teacher model, thereby identifying the local teacher model with the most complex network structure. In one embodiment of the present invention, the local teacher model with the most complex network structure can be configured as the master teacher model, so that the master teacher model has strong expressive and feature modeling capabilities, can provide more comprehensive semantic information, and guides the fusion of more representative labels. Once the master teacher model is determined, the remaining local teacher models of the neural networks are all used as auxiliary teacher models. The weight coefficient of the master teacher model is... The value is typically set to 0.7, which is used to emphasize the leading teacher's dominant role in the integration process.

[0231] As can be seen from the above description of generating fused soft labels, the present invention can achieve knowledge fusion without accessing any client's local private data and local private data labels. In other words, without using any real labels, it can rely entirely on the personalized model inference output of all neural network local teacher model inference predictions to generate corresponding fused soft labels through information fusion, which are then used as supervision labels for the corresponding pseudo-training images.

[0232] Furthermore, when there are significant discrepancies in inference prediction among all neural network local teacher models, this invention dynamically assigns higher weights to the master teacher model, which has a more complex structure and stronger expressive power, to guide the generation of more reliable supervision labels. At the same time, it retains the auxiliary information of its assistant teacher model, avoids overfitting to a single neural network local teacher model, and achieves the alignment and unified expression of knowledge across structures and multi-source neural network local teacher models. The fused soft label distribution is smoother, which helps the basic student model learn more generalized category relationships through distillation training.

[0233] In one embodiment of the present invention, when generating a globally shared student model after distillation training of a basic student model, the process includes:

[0234] The basic student model is trained using a pseudo-supervised training dataset. During model training, the pseudo-training images in each pseudo-supervised training sample are loaded into the basic student model. After the basic student model performs inference and prediction, the inference output information of the student model is generated.

[0235] After each round of model training, for each student model inference output information, the training divergence between the student model inference output information and the corresponding fused soft label is calculated, and the calculated training divergence is used as the distillation loss. Thereafter, the base student model is updated with the optimization objective of minimizing the distillation loss.

[0236] The updated basic student model was tested using the student training test set, and the corresponding test accuracy was determined after the test.

[0237] After training the basic student model to the target number of training rounds, determine the highest test accuracy and use the basic student model corresponding to the highest test accuracy as the globally shared student model.

[0238] It should be noted that when generating a globally shared student model through distillation training, a basic student model should first be constructed. This basic student model can be a standardized model, maintaining consistency across all experimental scenarios for easy deployment and migration, and not relying on the architecture of any client's local neural network teacher model. The basic student model is typically a moderately complex convolutional neural network, possessing good expressive power and generalization ability. For example, a moderately complex basic student model structure can employ four convolutional modules, each containing two 3×3 convolutional layers (both using Batch Normalization and ReLU activation functions) and one 2×2 max-pooling layer, finally connected to a three-layer fully connected network (containing 256 and 64-dimensional hidden layers and a final classification output layer) via a Flatten layer. In this case, the total number of network parameters of the entire basic student model is controlled within the millions, combining feature extraction capability and computational efficiency, suitable for multi-class image recognition tasks. It should be understood that the basic student model constructed in this invention is not limited to the examples given here; the model depth and width can be appropriately adjusted according to the computing resources of the deployment platform, which will not be elaborated further here.

[0239] After constructing the basic student model, the pseudo-supervised training dataset generated above is used to train the basic student model, thereby achieving knowledge fusion and generalization of all neural network local teacher models. During training, the server inputs each pseudo-training image into the basic student model and obtains the standard output information of the student model after inference and prediction by the basic student model. Generally, the standard output information of the student model is in logits form, and the standard output information of the student model can be represented as follows: Subsequently, Divide by the set temperature coefficient The prediction probability distribution of the ensemble student model is obtained by applying a softmax function for normalization. This distribution is the student model's inference output information. For the student model's inference output information, we have:

[0240]

[0241] in, To use pseudo-training images The corresponding student model inference output information, Temperature coefficient For details, please refer to the explanation of the temperature coefficient above.

[0242] As explained above, for each pseudo-training image, the fused soft label generated based on the pseudo-training image serves as the supervision signal for training. In one embodiment of the present invention, the server uses Kullback-Leibler divergence as the distillation loss function to measure the difference between the student model inference output information of the base student model and the corresponding fused soft label, then:

[0243]

[0244] in, For distillation losses, For pseudo-supervised training set, To use pseudo-training images The corresponding fusion soft tag, To calculate the training divergence between the fused soft labels and the student model's inference output information.

[0245] During model training, the base student model is updated with the goal of minimizing distillation loss. The specific method for updating the base student model can be consistent with existing techniques, such as updating the base student model through backpropagation. The specific update method and process will not be elaborated here.

[0246] After training the basic student model once using a pseudo-supervised training dataset, the updated basic student model can be tested using a student training test set, and the corresponding test accuracy can be determined after the test. It should be noted that the student training test set can be generated using data with supervised information on the server. Of course, the test samples in the student training test set should correspond to the types of the training samples in the aforementioned pseudo-supervised training dataset to meet the corresponding testing requirements. After testing, the corresponding test accuracy can be calculated and determined using methods commonly used in this technical field. The method for obtaining the test accuracy is consistent with existing technologies and will not be elaborated here.

[0247] In practice, the number of training rounds of the base student model using the pseudo-supervised training dataset can be configured, i.e., the target distillation training rounds can be configured. The specific number of training rounds can be selected as needed. Each training round generally means that all training samples in the pseudo-supervised training dataset are used to train the base student model. After the base student model has reached the target distillation training rounds, the highest test accuracy is determined, and the base student model corresponding to the highest test accuracy is used as the globally shared student model.

[0248] It should be understood that the federated learning process of this invention ends when the globally shared student model is trained. As can be seen from the aforementioned distillation training process, the server does not introduce any real label supervision during training, nor does it involve gradient updates of the local teacher model of the neural network. Instead, it only relies on the pseudo-supervised training dataset to perform standard backpropagation on the server side to minimize distillation loss.

[0249] The globally shared student model can effectively absorb the output distribution characteristics of the local teacher models of the neural networks from various heterogeneous clients, achieving the fusion of distributed heterogeneous knowledge while maintaining a unified structure. The federated learning method of this invention does not require access to the local private data of the clients, does not require a unified architecture of the local teacher models of the neural networks, and only requires one communication to obtain a globally shared student model with strong generalization ability, thereby significantly improving the flexibility and efficiency of federated learning in practical deployment scenarios.

Claims

1. A federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion, characterized in that, The federated learning method includes: A federated learning system is provided for federated learning, wherein the federated learning system includes a server and a client group adapted to and connected to the server. The client group includes several clients, each client having local private data, and the local private data of all clients has the characteristic of non-independent and identically distributed. Configuring a federated learning system for federated learning includes at least configuring each client to perform teacher model generation processing and configuring the server to perform student model generation processing. When any client performs teacher model generation processing, it searches for and generates a neural network local teacher model that best matches the client based on the client's local private data, and sends the generated neural network local teacher model to the connected server. After the server performs the student model generation process, it generates at least a globally shared student model. The generation of the globally shared student model includes: Based on all received neural network local teacher models, a pseudo-supervised training dataset is generated. The pseudo-supervised training dataset includes several pseudo-supervised training samples. Each pseudo-supervised training sample includes a pseudo-training image that can reproduce the knowledge representation of the corresponding client and a fused soft label corresponding to the pseudo-training image. When generating each pseudo-supervised training sample, the server does not access the local private data of the corresponding client in the client group. A basic student model is constructed by distillation training using the pseudo-supervised training dataset, and a globally shared student model is generated after distillation training of the basic student model.

2. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 1, characterized in that, Each client performing teacher model generation processing includes: Construct a network particle swarm, wherein the network particle swarm consists of several network particles, each of which contains a network encoding sequence and represents a candidate convolutional neural network based on the corresponding network encoding sequence, and the length of the network particle is the network depth of the candidate convolutional neural network it represents. Based on the local private data of the current client, the above-constructed network particle swarm is optimized by PSO, so as to obtain the optimal candidate convolutional neural network corresponding to the optimal network particle. The optimal candidate convolutional neural network is trained based on the local private data of the current client, and a local teacher model of the neural network is generated after the model is trained.

3. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 2, characterized in that, At least one local training set and one local validation set should be constructed based on the local private data within each client. When performing PSO optimization on a network, it includes at least several sequential network training optimization processes, where each network training optimization process includes sequential optimization training processes and particle update processes. During the optimization training process, the network encoding sequence of each network particle is converted into a corresponding candidate convolutional neural network, and the model training of each converted candidate convolutional neural network is performed on the target round using the optimization local training set, so as to generate the corresponding candidate convolutional neural network after model training. When performing particle update processing, each candidate convolutional neural network is validated using the optimization validation set, and the network accuracy calculated by the network validation is used as the fitness value of the current candidate convolutional neural network. Based on the fitness value of each candidate convolutional neural network, the individual optimal convolutional neural network corresponding to each candidate convolutional neural network is determined, and the global optimal convolutional neural network is determined based on all the individual optimal convolutional neural networks. When the PSO optimization of the network particle swarm fails to reach the target state, each network particle is updated based on the global optimal convolutional neural network and the individual optimal convolutional neural network of each network particle to update the network encoding sequence of each network particle and perform the next network training optimization process. When the PSO optimization of the network particle swarm reaches the target state, the network particle corresponding to the globally optimal convolutional neural network is taken as the optimal network particle, and the globally optimal convolutional neural network configuration is taken as the optimal candidate convolutional neural network.

4. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to any one of claims 1 to 3, characterized in that, When generating a pseudo-supervised training dataset, the following are included: Within the server, based on the local teacher model of each neural network, a pseudo-training image generation mechanism optimized by random noise input is used to generate several pseudo-training images, wherein each pseudo-training image satisfies the requirement to reproduce the knowledge representation of the corresponding client. Each pseudo-training image is loaded into each neural network local teacher model within the server to configure the neural network local teacher model to perform inference prediction on the loaded pseudo-training images and generate personalized model inference output information. The server will perform consistency-aware fusion processing on all corresponding personalized model inference output information for each pseudo-training image to generate fused soft labels after consistency-aware fusion processing. The generated fused soft labels and corresponding pseudo-training images are used as pseudo-supervised training samples in the pseudo-supervised training dataset.

5. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 4, characterized in that, For each neural network local teacher model, when generating pseudo-training images using a pseudo-image generation mechanism optimized with random noise input, the following steps are included: Extract the task layer weight information of the local teacher model of the neural network, and construct a category similarity matrix based on the extracted task layer weight information; Based on the constructed category similarity matrix, a category-aware pseudo-label modeling method is used to construct the corresponding soft label distribution group. The soft label distribution group includes several soft label distribution vectors. When the neural network local teacher model makes a classification prediction on the pseudo-training image, the number of distribution probabilities in the soft label distribution vector is consistent with the number of categories predicted by the neural network local teacher model. The server initializes a set of random noise images for the current neural network local teacher model, and optimizes and updates each random noise image using each soft label distribution vector, so as to generate a corresponding pseudo training image with class discrimination after optimization and update.

6. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 5, characterized in that, When optimizing and updating each image with random noise, the following steps are included: Random noise images are loaded into a neural network local teacher model, and the corresponding noise image logits output information is obtained through inference and prediction by the neural network local teacher model. Temperature scaling is applied to the logits output information of the noisy image, and the prediction probability distribution of the noisy image is generated by softmax normalization. The cross-entropy loss between each soft label distribution vector and the predicted probability distribution of the noisy image is used as the optimization objective. During the optimization process, the random noise image is updated by gradient descent until the cross-entropy loss, which is the optimization objective, is minimized, and then the corresponding pseudo-training image is generated.

7. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 4, characterized in that, When the server performs consistency-aware fusion processing with all corresponding personalized model inference output information for each pseudo-training image, it includes: The server calculates the average output information of the personalized model corresponding to the personalized model inference output information of the current pseudo-training image, and generates a teacher consistency metric based on the calculated average output information of the personalized model. When the teacher consistency metric is not greater than the consistency judgment threshold, the average output information of the personalized model is used as the fusion soft label. Otherwise, based on the main teacher fusion mechanism, all personalized model inference output information is output fused to generate the corresponding fusion soft label after output fusion.

8. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 7, characterized in that, When calculating the teacher consistency metric, we have: in, Pseudo-training images Teacher consistency metrics This represents the number of local teacher models of the neural network received by the server. For the first A neural network local teacher model uses pseudo-training images Personalized model inference output information generated through inference and prediction. The average output information of the personalized model To calculate the personalized model inference output information based on the average output information of the personalized model The divergence.

9. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 7, characterized in that, When fusing all personalized model inference output information based on the master teacher fusion mechanism, it includes: The server selects one neural network local teacher model as the master teacher model and uses the remaining neural network local teacher models as assistant teacher models. The master teacher model is at least the neural network local teacher model with the most complex network structure. When performing output fusion, we have: in, Pseudo-training images The integration of soft tags, The weighting coefficients for the main teacher model. The main teacher model uses pseudo-training images Personalized model inference output information generated through inference and prediction. This represents the number of local teacher models of the neural network received by the server. For the first A neural network local teacher model uses pseudo-training images Personalized model inference output information generated through inference and prediction.

10. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to any one of claims 1 to 3, characterized in that, When generating a globally shared student model after distillation training of the basic student model, the following steps are included: The basic student model is trained using a pseudo-supervised training dataset. During model training, the pseudo-training images in each pseudo-supervised training sample are loaded into the basic student model. After the basic student model performs inference and prediction, the inference output information of the student model is generated. After each round of model training, for each student model inference output information, the training divergence between the student model inference output information and the corresponding fused soft label is calculated, and the calculated training divergence is used as the distillation loss. Thereafter, the base student model is updated with the optimization objective of minimizing the distillation loss. The updated basic student model was tested using the student training test set, and the corresponding test accuracy was determined after the test. After training the basic student model to the target number of training rounds, determine the highest test accuracy and use the basic student model corresponding to the highest test accuracy as the globally shared student model.

Images