Federal learning method supporting heterogeneous model architecture search and zero sample knowledge fusion

Abstract

The invention relates to a federated learning method supporting heterogeneous model architecture search and zero sample knowledge fusion. The method comprises the following steps: providing a federated learning system to be subjected to federated learning, and when the federated learning system is configured to perform federated learning and any client executes teacher model generation processing, searching and generating a neural network local teacher model optimally matched with the client based on local private data in the client, and sending the generated neural network local teacher model to a connected server, after the server executes student model generation processing, at least generating a global shared student model, when the global shared student model is generated, training the constructed basic student model by using the pseudo-supervised training data set, and generating the global shared student model by using the pseudo-supervised training data set. And after the basic student model is subjected to distillation training, a global shared student model is generated. According to the method, the heterogeneous model of the client can be effectively supported, the personalized capability and privacy protection are improved, the communication cost is reduced, and the model generalization is excellent.

Description

TECHNICAL FIELD

[0001] The present application relates to a federated learning method, in particular to a federated learning method supporting heterogeneous model architecture search and zero-shot knowledge distillation. BACKGROUND

[0002] With the rapid development of big data and artificial intelligence technology, deep neural networks have made significant progress in image recognition, natural language processing and other tasks. However, the success of these tasks relies on centralized data collection and model training processes, which poses serious challenges to user privacy and data security. In this context, federated learning (FL) is proposed. As a new type of distributed collaborative learning paradigm, federated learning supports multiple terminal devices to collaboratively train models without sharing local private data, effectively alleviating the problem of data privacy leakage. Each terminal device acts as a client in federated learning.

[0003] Although federated learning has good privacy protection mechanism, it still faces many challenges in practical application; specifically: first, the data distribution between clients often differs significantly (i.e., non-identically distributed), making it difficult for a global model with a unified model architecture to achieve ideal performance on local data; second, different clients differ in computing power, memory bandwidth, etc., making it difficult for a unified model architecture to run efficiently on all clients. In addition, to improve model performance, traditional federated learning methods often rely on multiple rounds of communication to iteratively synchronize model parameters, resulting in high system load and communication overhead, and even causing additional privacy risks.

[0004] To solve the above problems, personalized federated learning (PFL) has become a research hotspot in recent years, which aims to provide each client with a customized model that better fits its local data characteristics while ensuring privacy. Currently, in the field of personalized federated learning (PFL), there are various methods trying to improve the adaptability of the model in the scenario of heterogeneous data distribution and resource constraints. Specifically, the methods can be mainly classified into the following two categories: The first type is a personalized federated optimization method based on a unified model architecture. Representative methods include pre-FedAvg, FedBN, Ditto, FedPer, LG-FedAvg, etc. These methods generally assume that all clients share a unified model structure and achieve personalized training through local fine-tuning, regularization constraints, or module division. For example, the FedBN method alleviates feature distribution deviation by localizing batch normalization layer parameters, and the FedPer method divides the model into shared layers and personalized layers for separate processing. However, this type of method still essentially uses a static model structure and is difficult to dynamically adapt to the significant differences in data distribution and computing resources among clients, resulting in limited performance and efficiency.

[0005] The second type is a personalized federated learning method that introduces an automatic structure optimization mechanism. This type of method usually introduces neural architecture search (NAS) or model pool mechanisms. FedNAS is a representative method that optimizes the model through a distributed NAS collaboration process, where the control and decision of the architecture search are mainly completed by the server side, and the clients only participate in performance evaluation. The final optimal model is selected for global sharing. Although FedNAS considers client differences in form, its search process still highly depends on server coordination, making it difficult to truly realize structure heterogeneity and local adaptation.

[0006] In addition, there are some methods that use pre-defined models to achieve model personalization, such as DENSE and FedMMD. These methods use existing excellent architectures such as ResNet and MobileNet to select and deploy models as needed to achieve model heterogeneity. FedMMD requires real public data for distillation, and DENSE requires an external generator to synthesize images, both of which have privacy leakage risks and high system complexity. In addition, FedMHO uses a model pool for structure difference deployment, using discriminative models on resource-rich clients and generative models on resource-constrained clients to achieve one-time distillation collaboration under heterogeneous models.

[0007] Although personalized federated learning has made some progress in supporting personalization and model heterogeneity, existing methods still have the following limitations, including: 1) Client model structure homogeneity limits personalized performance. Specifically, most federated learning methods assume that all clients use a unified model architecture, lack the ability to adapt to heterogeneous hardware resources and non-independent and identically distributed data, and cannot meet the individual modeling needs of different clients.

[0008] 2) The structure depends on a pre-defined model set or manual selection of a model architecture, lacking automatic search capability. Specifically, some methods support heterogeneous modeling, usually relying on a pre-defined model set or manual selection of a model structure, lacking an automatic search mechanism, making it difficult to dynamically optimize the model architecture for the client's local data, and the degree of personalization is limited.

[0009] 3) Large communication overhead, difficult to deploy in low-bandwidth scenarios. Specifically, existing personalized methods based on neural architecture search generally rely on multiple rounds of communication and intermediate structure transmission, resulting in high communication cost and increased privacy risk, making it difficult to apply to communication-limited scenarios.

[0010] 4) Existing federated distillation methods rely on public data, making it difficult to meet privacy constraints. Specifically, although distillation can be used for knowledge transfer between heterogeneous models, most federated distillation methods (such as FedDF, FedMD, etc.) require the introduction of a portion of pre-set public data as a distillation medium for unified training of student models. However, in practical applications, such public data is difficult to obtain or cannot cover the non-independent data distribution of the client, and the introduction of public data itself may pose new privacy risks. Therefore, under the premise that the server cannot access any real data, how to achieve efficient and secure knowledge fusion is still a major difficulty in current federated distillation research.

[0011] In summary, the existing personalized federated learning methods still have obvious deficiencies in model heterogeneity adaptation, automatic architecture optimization, communication efficiency, and knowledge distillation without public data, making it difficult to meet the actual application requirements. SUMMARY

[0012] The purpose of the present application is to overcome the deficiencies in the prior art and provide a federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion, which can effectively support heterogeneous models of clients, improve personalization and privacy protection, reduce communication costs, and improve model generalization.

[0013] According to the technical solution provided by the present application, a federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion is provided, which comprises: A federated learning system to be subjected to federated learning is provided, wherein the federated learning system comprises a server and a client group connected to the server, the client group comprises a plurality of clients, each client has local private data, and the local private data of all clients has a non-independent and identically distributed characteristic; When the federated learning system is configured for federated learning, at least each client is configured to perform teacher model generation processing and the server is configured to perform student model generation processing, wherein, When any client performs the teacher model generation processing, a neural network local teacher model that is most optimally matched with the client is searched and generated based on the local private data in the client, and the generated neural network local teacher model is sent to the connected server, After the server performs the student model generation processing, at least a global shared student model is generated, wherein when the global shared student model is generated, the following is included: Based on all the received neural network local teacher models, a pseudo-supervised training data set is generated, wherein the pseudo-supervised training data set includes a plurality of pseudo-supervised training samples, each pseudo-supervised training sample includes a pseudo-training image that can reproduce the knowledge representation of the corresponding client and a fusion soft label corresponding to the pseudo-training image, and when each pseudo-supervised training sample is generated, the server does not access the local private data of the corresponding client in the client group; The base student model constructed by distillation training is distilled using the pseudo-supervised training data set, and the global shared student model is generated after distillation training of the base student model.

[0014] When each client performs the teacher model generation processing, the following is included: A network particle swarm is constructed, wherein the network particle swarm includes a plurality of network particles, each network particle has a network code sequence, and a candidate convolutional neural network is represented based on the corresponding network code sequence, and the length of the network particle is the network depth of the represented candidate convolutional neural network; The network particle swarm constructed above is subjected to PSO optimization based on the local private data of the current client, so that the optimal candidate convolutional neural network corresponding to the optimal network particle is obtained by PSO optimization; The optimal candidate convolutional neural network obtained is subjected to model training based on the local private data of the current client, so that a neural network local teacher model is generated after model training.

[0015] Based on the local private data in each client, at least an optimization local training set and an optimization local validation set are constructed; When the network particle swarm is subjected to PSO optimization, at least a plurality of times of network training optimization processing are sequentially performed, wherein each time of network training optimization processing includes sequential optimization training processing and particle updating processing; When the optimization training processing is performed, the network code sequence of each network particle is converted into a corresponding candidate convolutional neural network, and each converted generated candidate convolutional neural network is subjected to target round model training using the optimization local training set, so that a corresponding candidate convolutional neural network is generated after model training; When performing the particle update process, the network verification is performed on each candidate convolutional neural network by using the optimization verification set, and the network accuracy calculated by the network verification is taken as the fitness value of the current candidate convolutional neural network. Based on the fitness value of each candidate convolutional neural network, the individual optimal convolutional neural network corresponding to each candidate convolutional neural network is determined, and the global optimal convolutional neural network is determined based on all the individual optimal convolutional neural networks. When the PSO optimization on the network particle swarm does not reach the target state, each network particle is updated based on the global optimal convolutional neural network and the individual optimal convolutional neural network of each network particle to update the network coding sequence of each network particle, and the next network training optimization process is performed. When the PSO optimization on the network particle swarm reaches the target state, the network particle corresponding to the global optimal convolutional neural network is taken as the optimal network particle, and the global optimal convolutional neural network is configured as the optimal candidate convolutional neural network.

[0016] When generating the pseudo-supervised training data set, the following steps are included: In the server, based on each neural network local teacher model, a pseudo image generation mechanism optimized by random noise input is used to generate a plurality of pseudo training images, wherein each pseudo training image satisfies the requirement of reproducing the knowledge representation of the corresponding client. Each pseudo training image is loaded into each neural network local teacher model in the server to configure the neural network local teacher model to perform inference prediction on the loaded pseudo training image and generate personalized model inference output information. The server performs consistency perception fusion processing on all the corresponding personalized model inference output information of each pseudo training image to generate a fusion soft label after the consistency perception fusion processing. The fusion soft label generated above and the corresponding pseudo training image are taken as a pseudo-supervised training sample in the pseudo-supervised training data set.

[0017] When generating pseudo training images for each neural network local teacher model by using a pseudo image generation mechanism optimized by random noise input, the following steps are included: The task layer weight information of the neural network local teacher model is extracted, and a category similarity matrix is constructed based on the extracted task layer weight information. Based on the constructed category similarity matrix, a category perception type pseudo label modeling method is used to construct a corresponding soft label distribution group, which includes a plurality of soft label distribution vectors, wherein when the inference prediction of the neural network local teacher model on the pseudo training image is a classification prediction, the number of distribution probabilities in the soft label distribution vector is consistent with the number of categories of the classification prediction performed by the neural network local teacher model. The server initializes a set of random noise images for the current neural network local teacher model, and optimizes and updates each random noise image using each soft label distribution vector to generate a corresponding pseudo training image after optimization and update.

[0018] When optimizing and updating each random noise image, the following steps are included: The random noise image is loaded into the neural network local teacher model to obtain corresponding noise image logits output information through inference prediction of the neural network local teacher model. The noise image logits output information is temperature scaled and normalized by softmax to generate noise image prediction probability distribution. The cross-entropy loss between each soft label distribution vector and the noise image prediction probability distribution is taken as the optimization objective, and the random noise image is updated by gradient descent during the optimization process until the cross-entropy loss as the optimization objective is minimized, thereby generating the corresponding pseudo training image.

[0019] When the server performs consistency perception fusion processing on all corresponding personalized model inference output information of each pseudo training image, the following steps are included: The server calculates the personalized model average output information of the personalized model inference output information corresponding to the current pseudo training image, and calculates the teacher consistency metric based on the calculated personalized model average output information. When the teacher consistency metric is not greater than the consistency judgment threshold, the personalized model average output information is taken as the fusion soft label, otherwise, all personalized model inference output information is output fused based on the main teacher fusion mechanism to generate the corresponding fusion soft label after output fusion.

[0020] When calculating the teacher consistency metric, the following steps are included:

[0021] Wherein, is the teacher consistency metric of the pseudo training image , is the number of neural network local teacher models received by the server, is the i-th neural network local teacher model, is the personalized model inference output information generated by the i-th neural network local teacher model for the pseudo training image , is the personalized model average output information, is the divergence of the personalized model inference output information based on the personalized model average output information.

[0022] ​​When all the personalized model inference output information is output fused based on the main teacher fusion mechanism, it includes: The server selects one neural network local teacher model as a main teacher model, and the remaining neural network local teacher models as auxiliary teacher models, wherein the main teacher model is at least the neural network local teacher model with the most complex network structure; When output fusion is performed, then:

[0023] Among them, is a pseudo-training image a fusion soft label, is a main teacher model weight coefficient, is the personalized model inference output information generated by the main teacher model for inference prediction on the pseudo-training image , the number of neural network local teacher models received by the server is , the personalized model inference output information generated by the first neural network local teacher model for inference prediction on the pseudo-training image is the personalized model inference output information generated by the first neural network local teacher model for inference prediction on the pseudo-training image

[0024] When the global shared student model is generated after distillation training of the basic student model, it includes: The basic student model is trained using the pseudo-supervised training dataset, wherein during model training, the pseudo-training image in each pseudo-supervised training sample is loaded into the basic student model, and the student model inference output information is generated after inference prediction by the basic student model; After each round of model training, the training divergence of each student model inference output information and the corresponding fusion soft label is calculated, and the calculated training divergence is used as a distillation loss, and thereafter, the basic student model is updated with the minimization of the distillation loss as the optimization goal; The updated basic student model is tested using the student training test set, and the corresponding test accuracy is determined after testing; After the model training of the basic student model reaches the target distillation training round, the highest test accuracy is determined, and the basic student model corresponding to the highest test accuracy is used as the global shared student model.

[0025] The advantages of the present application are that the client searches and generates the optimal matching neural network local teacher model through PSO optimization, the server constructs a pseudo-supervised training dataset based on the neural network local teacher model uploaded by the client, and trains the basic student model using the pseudo-supervised training dataset, so that the global shared student model can be obtained; the federal learning method of the present application achieves the following technical effects: Enhance personalization ability: Since each client searches and generates the optimal matching neural network local teacher model according to its own local private data distribution, it can more accurately fit the feature distribution of the local private data, significantly improve the expressiveness and generalization ability of the neural network local teacher model, avoid the performance bottleneck caused by the homogeneous architecture, and effectively support the search and generation of heterogeneous model architecture.

[0026] Significantly reduce communication cost: In federated learning, the "one-time communication" mechanism is adopted, and only the client needs to send the neural network local teacher model to the server for data communication. Thereafter, the server can independently complete the training and generation of the global shared student model, saving the process of multiple rounds of parameter synchronization and iteration required by traditional federated learning, greatly reducing communication overhead and training period.

[0027] Achieve knowledge aggregation under the condition of no real data: The server does not need to access any local private data of the client, nor does it need to rely on public data sets. It only uses the neural network local teacher model uploaded by the client to generate pseudo-training images and construct a fusion soft label corresponding to each pseudo-training image, avoiding the risk of real data leakage and enhancing the privacy protection ability of the system.

[0028] In summary, the present application realizes the efficient completion of cross-structure knowledge integration and basic student model training under the premise of heterogeneous neural network local teacher model structure and server not accessing any real local private data of the client, has the technical advantages of low communication cost, strong personalization ability, good privacy protection, and excellent model generalization, and solves the technical problems of model heterogeneity, knowledge fusion, and privacy constraints in existing federated learning methods. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 An embodiment flowchart of the federated learning method of the present application.

[0030] Figure 2 An embodiment schematic diagram of the federated learning system of the present application. DETAILED DESCRIPTION

[0031] The present application will be further described below in conjunction with specific drawings and embodiments.

[0032] In order to effectively support the heterogeneous model of the client, improve the personalization ability and privacy protection, and reduce the communication cost, the present application provides a federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion. Specifically, the federated learning method comprises: A federated learning system is provided for federated learning, wherein the federated learning system comprises a server and a client group connected to the server, the client group comprises a plurality of clients, each client has local private data, and the local private data of all clients has a non-independent and identically distributed characteristic; When the federated learning system is configured for federated learning, at least the following are included: configuring each client to perform teacher model generation processing, and configuring the server to perform student model generation processing, wherein, When any client performs teacher model generation processing, a neural network local teacher model that optimally matches the client is searched and generated based on the local private data in the client, and the generated neural network local teacher model is sent to the server connected to the client, After the server performs student model generation processing, a global shared student model is generated, wherein when the global shared student model is generated, the following is included: Based on all received neural network local teacher models, a pseudo-supervised training data set is generated, wherein the pseudo-supervised training data set includes a plurality of pseudo-supervised training samples, each pseudo-supervised training sample includes a pseudo-training image that can reproduce the knowledge representation of the corresponding client and a fusion soft label corresponding to the pseudo-training image, and when each pseudo-supervised training sample is generated, the server does not access the local private data of the corresponding client in the client group; The pseudo-supervised training data set is used to distill and train a basic student model, and a global shared student model is generated after distilling and training the basic student model.

[0033] Figure 1 An embodiment flowchart of the federated learning of the present application is shown in the figure, and as shown in the figure, the federated learning of the present application should be based on a federated learning system, that is, the federated learning is performed through the provided federated learning system. Figure 2 An embodiment of the federated learning system is shown in the figure, and as shown in the figure, the federated learning system should include a server and a client group, the client group includes at least one client, Figure 2 The client group includes four clients in the embodiment, and the clients in the client group should be connected to the server, wherein the connection between the client and the server is an adaptive connection, that is, the client and the server can realize data communication, and the specific way of realizing data communication can be selected as needed.

[0034] It should be noted that the server and the client can adopt the existing common form, and the specific implementation can meet the execution of federated learning. Here, details are not repeated. In specific implementation, the local private data is stored in each client, and the local private data of all clients has the non-identically distributed characteristic, that is, the local private data in different clients presents significant difference. For example, one client mainly contains local private data of a few categories such as "cat" and "dog", and another client contains local private data of completely different categories such as "airplane" and "car", and the number of categories, image resolution, texture complexity and other characteristics of the local private data are also inconsistent.

[0035] By Figure 1 It can be seen that when performing federated learning, each client should be configured to perform teacher model generation processing, such as Figure 2 When there are four clients in the client group, the four clients all perform corresponding teacher model generation processing. Specifically, when each client performs teacher model generation processing, it should search and generate a neural network local teacher model that is the best match for the client based on the local private data in the client. For example, the client with local private data of a few categories such as "cat" and "dog" should search and generate a corresponding neural network local teacher model based on the "cat", "dog" and other category data of the corresponding local private data. It should be understood that the generated neural network local teacher model can perform task processing on the local private data in the corresponding client. The task processing capability can be classification task processing, regression task processing, etc. In the present application, the task capability is taken as classification task processing for example. Other cases can refer to the corresponding description. Here, details are not repeated.

[0036] It should be understood that since the local private data of different clients has the non-identically distributed characteristic, and each client only searches and generates a neural network local teacher model using its own local private data, the neural network local teacher models generated by different clients can be different network architectures. That is, the present application can support the client group to generate heterogeneous neural network local teacher models. Therefore, it can be seen that the federated learning of the present application can support heterogeneous model architecture search. The way and process of searching and generating a neural network local teacher model based on local private data will be described in detail below.

[0037] Since the client and the server have data communication capability, after generating the neural network local teacher model in the client, the generated neural network local teacher model should be sent to the server. After the client sends the neural network local teacher model to the server, the client and the server complete one data communication. It should be noted that when performing federated learning of the present application, each client only needs to perform data communication with the server once, that is, only when the neural network local teacher model is sent to the server, data communication is required. Therefore, it can be seen that the present application can effectively reduce the network overhead in federated learning and reduce the communication cost.

[0038] When all the clients send the corresponding neural network local teacher model to the server, the server should be configured to perform student model generation processing, and after performing the student model generation processing, at least a global shared student model can be generated, that is, after generating the global shared student model in the server, the federated learning of the present application is completed.

[0039] Since the global shared student model is generated based on all the neural network local teacher models, the global shared student model has the advantages of unified structure, strong generalization ability and no need to rely on local private data in the client. In actual deployment, the global shared student model can be used in the following two typical scenarios: 1) As a global shared model directly deployed in a new client: For newly added clients, the server can directly issue the global shared student model to the newly added clients to provide a neural network model that can be used immediately, avoiding retraining and improving deployment efficiency. Among them, the newly added client specifically refers to a client that does not participate in the training process of generating the global shared student model, such as a client that does not participate in the above teacher model generation processing; 2) As an initial model for personalized fine-tuning: For clients with local private data, the global shared student model can be used as an initial weight model, and then the local private data can be used to further fine-tune the global shared student model as an initial weight model, thereby quickly adapting to the local data distribution and improving the performance of the client.

[0040] In addition, the global shared student model can also be used in other scenarios, and the specific application can be consistent with the prior art, which will not be illustrated one by one here.

[0041] In order to generate a global shared student model, after receiving all the neural network local teacher models, a pseudo-supervised training data set should be generated based on the received neural network local teacher models, such as Figure 2When the number of clients in the client group is four, the four clients send the corresponding neural network local teacher model to the server, and thereafter, the server generates a pseudo-supervised training dataset according to the four neural network local teacher models. The manner and process of generating the pseudo-supervised training dataset will be described below.

[0042] The pseudo-supervised training dataset can include a plurality of pseudo-supervised training samples, and the number of pseudo-supervised training samples can be selected as needed. Specifically, each pseudo-supervised training sample includes at least one pseudo-training image that can reproduce the knowledge representation of the corresponding client and one fusion soft label corresponding to the pseudo-training image, wherein the fusion soft label is the label of the pseudo-training image. When the task processing capability is a classification task, as mentioned above, the fusion soft label should represent the classification category state in the pseudo-training image. It should be noted that reproducing the knowledge representation of the corresponding client specifically refers to including the characteristics of the knowledge of the corresponding client in the pseudo-training image, that is, when generating the pseudo-training image, the knowledge characteristics of the corresponding client can be captured. In addition, when generating the pseudo-supervised training sample in the server, the server does not access the local private data of the corresponding client in the client group, thereby improving the ability of personalization and privacy protection. Specifically, the corresponding client here refers to the client whose knowledge representation is reproduced by the training image in the pseudo-supervised training sample.

[0043] In order to generate a globally shared student model, a basic student model should be constructed in the server, and the basic student model will be described below. After the basic student model is constructed, the pseudo-supervised training dataset can be used to distill train the constructed basic student model, and a globally shared student model can be generated after distill training the basic student model. The manner and process of distill training the basic student model using the pseudo-supervised training dataset will be described in detail below.

[0044] In an embodiment of the present application, when each client performs the teacher model generation process, it includes: constructing a network particle swarm, wherein the network particle swarm includes a plurality of network particles, each network particle has a network code sequence, and a candidate convolutional neural network is represented based on the corresponding network code sequence, and the length of the network particle is the network depth of the represented candidate convolutional neural network; performing PSO optimization on the constructed network particle swarm based on the local private data of the current client, to obtain an optimal candidate convolutional neural network corresponding to an optimal network particle through PSO (Particle Swarm Optimization) optimization; performing model training on the obtained optimal candidate convolutional neural network based on the local private data of the current client, to generate a neural network local teacher model after model training.

[0045] In order to search and generate a neural network local teacher model based on local private data in each client, a network particle swarm should be constructed. Generally, the network particle swarm includes a plurality of network particles, each of which has a network code sequence to represent a candidate convolutional neural network, and the length of the network particle is the network depth of the represented candidate convolutional neural network. Therefore, based on the network code sequence of each network particle, a corresponding candidate convolutional neural network can be obtained, that is, the network code sequence mainly records the layer type (such as convolution, pooling, and fully connected layer) of the candidate convolutional neural network, the layer state information (such as the size of the convolution kernel when the layer type is a convolution layer), and the output channel number information.

[0046] It should be noted that when the task processing capability is a classification task, the candidate convolutional neural network can perform classification processing on the local private data in the client. Generally, the local private data is image data, that is, the candidate convolutional neural network can identify the category of each image in the client.

[0047] In order to ensure that the search runs efficiently on the client, the present application provides a set of representative search space parameter configurations, specifically: 1) In view of the limited computing power and storage of edge clients, the present application controls the layer type, layer state, and output channel number information of the candidate convolutional neural network in the network code sequence. Specifically, the maximum output channel number of the convolution layer is limited to 128, and the maximum number of neurons of the fully connected layer is set to 300, to avoid generating a network structure with high computational complexity that is not conducive to deployment. It can be understood that the control performed by the present application should ensure that the candidate convolutional neural network can still classify the local private data.

[0048] 2) In order to guide the network particles to search in an effective architecture space, the number of network layers is limited to between 3 and 20 layers to balance the expression ability and training difficulty of the neural network. In addition, in the sampling probability of the layer type, the convolution layer, the pooling layer, and the fully connected layer are set to a proportion of 0.6, 0.3, and 0.1 respectively, which can improve the architecture diversity and maintain the stability of the information abstraction ability; the size of the convolution and is not more than 7*7, which is used to capture image features of different scales.

[0049] 3) The population size of the network particle swarm is set to 20 (at this time, the number of network particles is 20), the total number of search rounds of PSO optimization is set to 10, and the iteration is 50 times per round, which ensures the search convergence of the network particle swarm under the premise of reducing the training burden and search time of the client. In addition, after each network particle update, the candidate convolutional neural network is only trained for 10 epochs locally for fitness evaluation, and the final optimal candidate convolutional neural network is trained for 50 epochs (generations) to fully learn local knowledge, and then the neural network local teacher model can be generated.

[0050] It should be noted that the search space parameter configuration described above exhibits good convergence speed and accuracy performance in actual tests. Of course, the search space parameter configuration described above can also be adjusted as needed, and can be dynamically adjusted according to factors such as the computing power and memory capacity of the client in the deployment environment, for example, on a more resourceful GPU node, the population size of the network particle swarm, the depth of the candidate convolutional neural network, and the upper limit of the number of channels can be correspondingly expanded, which will not be illustrated one by one here.

[0051] In specific implementation, after determining the search space parameter configuration described above, the network particle swarm constructed based on the local private data of the current client can be optimized by PSO to obtain an optimal candidate convolutional neural network corresponding to the optimal network particle through PSO optimization, and then the optimal candidate convolutional neural network is trained to generate a neural network local teacher model after model training; the process of PSO optimization and model training of the optimal candidate convolutional neural network is described in detail below.

[0052] In an embodiment of the application, at least an optimization local training set and an optimization local validation set are constructed based on the local private data in each client; First, the network encoding sequence corresponding to the network particle in the network particle swarm is converted into the corresponding candidate convolutional neural network, and then the network particle swarm is optimized by PSO, and the PSO optimization includes at least a plurality of network training optimization processes performed in turn, wherein each network training optimization process includes optimization training process and particle update process performed in turn; When performing the optimization training process, the candidate convolutional neural network is trained by the optimization local training set for a target number of rounds to generate a corresponding candidate convolutional neural network after model training; When performing the particle update process, each candidate convolutional neural network is verified by the optimization validation set, and the network accuracy calculated by the network verification is used as the fitness value of the current candidate convolutional neural network; Based on the fitness value of each candidate convolutional neural network, determine the individual optimal convolutional neural network corresponding to each candidate convolutional neural network, and determine the global optimal convolutional neural network based on all the individual optimal convolutional neural networks; When the PSO optimization of the network particle swarm fails to reach the target state, each network particle is updated based on the global optimal convolutional neural network and the individual optimal convolutional neural network of each network particle to update the network coding sequence of each network particle and perform the next network training optimization process; When the PSO optimization of the network particle swarm reaches the target state, the network particle corresponding to the global optimal convolutional neural network is taken as the optimal network particle, and the global optimal convolutional neural network configuration is taken as the optimal candidate convolutional neural network.

[0053] As can be seen from the above description, local private data can generally be image data. In addition, local private data also includes labels corresponding to each image. For example, when the network task mentioned above is a classification task, the image label is the category corresponding to the annotated image. The image-label can constitute a local private sample. In order to meet the requirements of PSO optimization, the local private data can be divided into a local training set and a local validation set. The local training set and the local validation set can include several local private samples. The local private samples in the local training set and the local validation set are completely non-repeated. The corresponding local private samples can be divided as needed and will not be described here.

[0054] When the total number of search rounds in the above search space parameter configuration is 10, the PSO optimization process should include 10 consecutive network training optimization processes, and each network training optimization process may include sequential optimization training processes and particle update processes. In the above description, "the total number of search rounds for PSO optimization is set to 10, and each round is iterated 50 times" specifically means that the total number of search rounds for PSO optimization is 10 rounds, and in each round of PSO optimization, the above network training optimization process should be performed 50 times, that is, 50 rounds of optimization training process are performed. In each round of optimization training process, each candidate convolutional neural network should be trained for the target round model, and the target number of rounds can be 10 rounds. For the case of performing 10 rounds of optimization training process, please refer to the above description.

[0055] It is understandable that when the first network training optimization process is performed, the initial value of the network coding sequence should be given to each network particle. In order to ensure that the candidate convolutional neural network has a certain diversity and meets the constraints such as size and number of layers, the initial value of the network coding sequence can be given by random initialization, which can also determine the initial state of each network particle.

[0056] The following illustrates the random initialization of the network coding sequence of the network particle and the process, specifically: 1) initialization parameter planning, min_layer, max_layer: used to constrain the number of layers of the generated candidate convolutional neural network, specifically, based on the above search space parameter configuration, min_layer is 3 and max_layer is 20; conv_prob: probability of generating a convolutional layer, set to 0.6, used to guide the priority use of convolutional layers in the candidate convolutional neural network to extract image features; pool_prob: probability of generating a pooling layer, set to 0.3, used to realize feature map size compression and dimension reduction; fc_prob: probability of generating a fully connected layer, set to 0.1, used for high-dimensional feature integration and classification at the tail of the network; max_conv_kernel: maximum convolution kernel size, set to 7, used to limit the receptive field size of convolution operation; max_out_ch: maximum output channel number of convolutional layer, set to 256, used to control the model capacity and complexity; max_pool_layers: maximum number of pooling layers, determined dynamically according to the input image size, to ensure that the feature map still has effective spatial resolution after multiple pooling max_fc_neurons: maximum number of neurons in a single layer of a fully connected layer, set to 300, used to limit the expression size of the fully connected layer.

[0057] 2) network particle initialization mechanism The initialization process of each network particle follows the principles of hierarchical progression and probability-driven structure sampling, specifically, within the given network depth range (controlled by min_layer and max_layer), a mixed neural network architecture composed of convolutional layers (Conv), pooling layers (Pooling) and fully connected layers (FC) is generated layer by layer, and finally a structurally legal candidate convolutional neural network with classification ability is formed.

[0058] First, the first network layer of the network particle is forced to be a convolutional layer to ensure that the initial model has basic feature extraction capability, where the output channel number (ou_c) of the initial convolutional layer is randomly generated in the interval [16, max_out_ch], and the convolution kernel size (kernel) is randomly sampled in the interval [3, max_conv_kernel].

[0059] Subsequently, in the remaining layers, the structure type is determined layer by layer in a probability-driven manner. Specifically, a random number in the interval [0, 1] should be generated: the specific strategy is as follows: If the sampled random number is less than the preset probability threshold value p_conv, a new convolutional layer is added, and the number of channels and the size of the convolution kernel are both random values. If the sampled random number is less than the preset probability threshold value p_pool, a pooling layer (maximum pooling or average pooling, kernel size 2x2) is added, provided that the number of current pooling layers does not exceed the preset maximum number of pooling layers max_pool_layers. If the sampled random number is less than the preset probability threshold value p_fc, a fully connected layer is added, and the number of neurons is randomly determined in the range [1, max_fc_neurons].

[0060] Specifically, to maintain the logical coherence of the network structure, in an embodiment of the present application, when the first fully connected layer is generated in the candidate convolutional neural network, it indicates that the candidate convolutional neural network enters the tail phase, and thereafter the type of the layer is limited to a fully connected layer until the generation of the candidate convolutional neural network is completed.

[0061] In addition, the present application particularly introduces a number control mechanism for the pooling layer. Specifically, considering that the feature map size will gradually decrease under successive pooling operations, to prevent the generation of too many pooling layers from causing the feature map size to be too small and the network to be untrainable, the initial width value of the input image is simulated to perform successive 2x2 pooling operations until the feature map width is below a minimum threshold (such as 4 pixels), to determine the maximum allowed number of pooling layers max_pool_layers, which will be embedded as a constraint condition in the network particle initialization stage, to limit the number of pooling operations contained in each particle.

[0062] The candidate convolutional neural network generated by the entire network particle initialization process will be stored in the layers attribute of the network particle in the form of a list, wherein the layers attribute records the type (convolution, pooling, or fully connected) and key structure parameters. At the same time, the "speed" information corresponding to each layer will be initialized to support the subsequent structure evolution operation in the particle swarm optimization. As can be seen from the above description, the last layer of the network particle is finally forced to be a fully connected output layer, and the output dimension is consistent with the target dimension of the classification task, thereby ensuring the integrity and usability of the model structure.

[0063] ​​​It should be noted that the above-mentioned "speed" information specifically refers to describing the possible change trend of the candidate convolutional neural network in the subsequent PSO optimization process, which usually exists in the form of a set of structure change markers, such as keep, replace, remove or add. By assigning initial speed information to network particles, the network particle swarm has a certain exploration direction in the optimization of a round of optimization, avoiding all network particles staying in the initial structure and lacking diversity, thereby improving the coverage of the search space and the convergence efficiency. In the PSO optimization process, the speed information will be dynamically updated according to the difference between the individual optimal (pBest) and the group optimal (gBest) to constantly guide the particles to evolve towards a better network architecture.

[0064] Specifically, the speed information needs to be initialized to set the speed information, the purpose of which is to assign initial evolution instruction markers to each layer of each network particle. The initialization strategy can be that the convolution and pooling layers are marked as keep (keep the current structure), and the fully connected layers are marked as keep_fc, that is, the structure of the fully connected layers is not adjusted by default. When the last layer of the network particle is forcibly set as an output layer (FC), the output dimension is consistent with the number of task classification targets, ensuring the functional integrity and usability of the entire candidate convolutional neural network structure; wherein the number of task classification targets is generally related to the local private data in the client, and can be determined according to the local private data in the client.

[0065] It can be understood that at this point, the initialization of the candidate convolutional neural network is complete. For example, the initialized network particle structure is printed as follows: Particle 0: conv | conv | pool | conv | fc |, at this time, "conv | conv | pool | conv | fc |" is the network code sequence of a network particle, where "conv" represents a convolutional layer, "pool" represents a pooling layer, and "fc" represents a fully connected layer; Particle 1: conv | conv | fc |, at this time, "conv | conv | fc |" is the network code sequence of a network particle.

[0066] 3) Particle compilation mechanism After completing the structure initialization of the network particles, the network code sequence (i.e. the layers attribute list) of each network particle is converted into a trainable candidate convolutional neural network model ("structure compilation process"). In an embodiment, the above-mentioned structure compilation can be implemented based on Keras; specifically, during structure compilation, the structure description of each layer is read in sequence, and the candidate convolutional neural network is constructed according to the following logic, specifically: First, if the current layer is a convolutional layer (conv), a Conv2D layer is added according to the recorded output channel number ou c and the kernel size kernel; the first convolutional layer needs to specify the input shape (input_shape=(w, h, c)), and the subsequent layers are omitted; after each convolutional layer, a ReLU activation function, BatchNormalization, and Dropout are followed to enhance the training stability and prevent overfitting; If the current layer is a pooling layer (max_pool or avg_pool), MaxPooling2D or AveragePooling2D layers are added, respectively, with a kernel size and stride of 2x2 to maintain the consistency of feature map downsampling.

[0067] For the fully connected layer (fc), first determine whether to add a Flatten layer before it (if the previous layer is not a fully connected layer, the feature map needs to be flattened); then add a Dense layer, the number of neurons of which is given by ou c, and introduce Dropout and L2 regularization to improve the generalization ability. If the current layer is the last layer of the network, use the Softmax activation function to output the classification probability; otherwise, use the ReLU activation function and add BatchNormalization.

[0068] Finally, the candidate convolutional neural network is compiled using the Adam optimizer (learning rate is 0.001), the loss function is categorical_crossentropy, and the evaluation index is accuracy, thereby completing the formal construction and training preparation of the candidate convolutional neural network.

[0069] It should be noted that the Keras mentioned above can be implemented through the Keras interface under the TensorFlow framework. In addition, when using the Adam optimizer (learning rate is 0.001) for compilation, the Adam module in the Keras interface can be used to compile.

[0070] The above gives an embodiment of each network particle initialization and network encoding sequence conversion to generate a candidate convolutional neural network. Of course, the network encoding sequence and the way of converting into a candidate convolutional neural network can also be other ways, which will not be illustrated one by one here.

[0071] In specific implementation, when performing the optimization training process, each converted candidate convolutional neural network should be pre-trained using the optimization training set, and the cross-entropy loss function is used as the pre-training loss function for pre-training. For the pre-training loss function, it has:

[0072] wherein, is the pre-training loss, To find the optimal training set, is the number of optimal training samples in the optimal training set; To optimize the number of categories in the training set; Optimize training samples for candidate convolutional neural network pairs Prediction belongs to The predicted probability of the class; Is the indicator function, indicating the optimal training sample Tags Is it the first kind.

[0073] It can be understood that the optimization training set includes several optimization training samples, and the optimization training samples include optimization training images and training image labels corresponding to the training images. The candidate convolutional neural network optimizes the optimization training samples. Prediction, specifically refers to the candidate convolutional neural network to optimize the training samples Internal optimization training image prediction, optimization training samples Tags , specifically refers to optimizing training samples The training image labels within .

[0074] Generally, in each client, the candidate convolutional neural network generated by converting each network particle in the network particle group is pre-trained. In order to adapt to the resource limitations of the edge client, it can be seen from the above description that the present invention adopts a lightweight local training strategy for each candidate convolutional neural network model: the number of training rounds is fixed to 10 rounds, the batch size (batchsize) is set to 32, and the optimizer can use Adam or SGD to quickly evaluate the performance of the candidate convolutional neural network model in scenarios with limited computing resources. After each round of pre-training is completed, a candidate convolutional neural network can be generated. Thereafter, the parameters of the candidate convolutional neural network will be updated based on the local task data for subsequent fitness evaluation and structural evolution.

[0075] After pre-training is completed, the accuracy of the candidate convolutional neural network after pre-training is calculated using the optimization validation set, which is used as the fitness score of the current network particle corresponding to the candidate convolutional neural network. The fitness function is as follows:

[0076] in, represents the optimal validation set, is the number of optimal validation samples in the optimal validation set, For the The network particles correspond to the fitness values ​​of the convolutional neural network to be selected. Indicates the each network particle corresponds to a to-be-selected convolutional neural network the prediction result of the to-be-selected convolutional neural network, is an indicator function, indicating whether the prediction result is the label b.

[0077] The number of the optimization verification samples can refer to the number of the optimization training samples. It can be understood that the optimization verification samples and the optimization training samples should be completely different, that is, different data contents based on the local private data in the client.

[0078] In an embodiment of the present application, based on the fitness score of each to-be-selected convolutional neural network, the individual optimal convolutional neural network corresponding to each to-be-selected convolutional neural network can be determined, and the global optimal convolutional neural network can be determined based on all individual optimal convolutional neural networks. The specific determination of the individual optimal convolutional neural network and the global optimal convolutional neural network can be consistent with the prior art. For example, when performing the first network training optimization process, each to-be-selected convolutional neural network after pre-training is used as the individual optimal convolutional neural network of each network particle, and the to-be-selected convolutional neural network with the highest fitness score is used as the global optimal convolutional neural network. The way of determining the individual optimal convolutional neural network and the global optimal convolutional neural network after performing the subsequent network training optimization process can refer to the description herein.

[0079] When performing the particle update process, the network coding sequence in each network particle should be updated according to the individual optimal convolutional neural network and the global optimal convolutional neural network. During the update, the general update calculation method of the PSO algorithm can be used to ensure that each network particle can not only approach the convolutional neural network that has ever performed best, but also be guided by the current optimal convolutional neural network in the group, so as to achieve the balance between individualization and collaborative search. The way and process of network particle update are described below. The core idea of network particle update is to gradually guide the network particle to approach a better structure by simulating the behavior of group intelligence, so as to realize the adaptive optimization of network architecture. Specifically, in each iteration, first, the structure change speed of each network particle needs to be calculated. The speed vector is determined by the difference between the current network particle and its individual optimal (pBest) and the group optimal (gBest), which is used to guide the structure adjustment of the to-be-selected convolutional neural network.

[0080] In calculating the structural difference, the network is divided into convolution / pooling layer section and fully connected layer section, and the difference vector is generated by comparing respectively. Specifically: for the convolution / pooling layer section, if the current network particle and the reference structure (such as individual optimal pBest or group optimal gBest) exist in the same position and have the same layer type, it is marked as "keep" in the difference vector; if both have layers in the position but have different types, it is marked as "replace", indicating that the corresponding layer in the reference structure needs to be replaced as a whole in the subsequent update; if the current network particle has an extra layer in the position and the reference structure has no corresponding layer, it is marked as "remove"; if the current network particle lacks the layer and the reference structure exists, it is marked as "add", indicating that the layer needs to be supplemented.

[0081] For the fully connected layer section, since the structure is usually located at the tail of the network, the reverse alignment method is used for comparison: if there is a fully connected layer in the corresponding position, it is marked as "keep_fc"; if the current particle has an extra layer, it is marked as "remove_fc"; if the current particle lacks the layer, it is marked as "add_fc". It should be noted that the output layer is always forced to be retained, and its output dimension is consistent with the number of task categories, and does not participate in the difference operation. The difference vector obtained by the above method is essentially a set of structure evolution instructions (such as keep, replace, delete, supplement, etc.), which is used to guide the subsequent structure speed generation and particle update.

[0082] In addition, if the structure of the current network particle is consistent with the individual optimal pBest and the group optimal gBest, no further difference calculation is needed, but a random selection is made between the two to generate a retention update instruction. If there is a difference, the structure information of the corresponding layer is selected from the group optimal gBest or the individual optimal pBest with a certain probability according to the preset global learning factor (such as 0.7); if a layer does not exist in the target structure, a delete instruction is generated. Through the above method, a set of structure evolution instructions (i.e. structure speed) can be formed, including keep, replace or cut, etc.

[0083] Specifically, the above-mentioned target structure specifically refers to the structure feature set of the individual optimal pBest and the group optimal gBest, i.e. the reference structure that the current network particle needs to align or approach. If a layer does not exist in the target structure but exists in the current network particle, the system generates a delete instruction (for example, {type: remove} or {type: remove_fc}), indicating that the layer should be cut. Conversely, if the target structure has a layer and the current network particle lacks it, an add instruction is generated to supplement the layer structure at the corresponding position. The final structure evolution instruction (i.e. structure speed) may include keep, replace, add or delete, etc.

[0084] After obtaining the structure evolution instruction, the structure evolution instruction is applied to the structure code sequence of the current network particle, so that an updated network structure is obtained, and a new candidate convolutional neural network is generated accordingly. As can be seen, after performing the optimization training process, the candidate convolutional neural network can generate a to-be-selected convolutional neural network after model training, and after performing particle updating, the to-be-selected convolutional neural network is re-generated to correspond to the candidate convolutional neural network. Thereafter, the updated candidate network is trained based on the client local training data to obtain the accuracy as the fitness value; if the fitness is better than its historical best performance, the individual optimal of the particle is updated; if it also exceeds the group optimal, the group optimal is updated, and the candidate convolutional neural network is used as the candidate convolutional neural network for the next round of optimization storage process.

[0085] After updating to the new convolutional neural network in the above manner, the network particle structure is converted into the corresponding candidate convolutional neural network. As can be seen from the above description, for the converted candidate convolutional neural network, 10 epochs of training should be performed based on the client local training data to obtain the accuracy as the fitness value. If the accuracy of the current network particle is better than its historical best performance, the individual optimal pBest structure and fitness are updated; if the accuracy also exceeds the current group optimal gBest fitness, the global optimal structure is updated. Finally, the network particle is guided to evolve towards a better neural network architecture in the structure space through the above mechanism.

[0086] It should be noted that when the preset number of iterations is reached or the network particle group converges as a whole, the client takes the network particle corresponding to the global optimal convolutional neural network as the optimal network particle, and configures the global optimal convolutional neural network as the optimal candidate convolutional neural network. It should be understood that when the optimal candidate convolutional neural network of the current client is obtained, the optimal candidate convolutional neural network should be model trained to generate a corresponding neural network local teacher model after model training.

[0087] Specifically, when the optimal candidate convolutional neural network is model trained, the local private data in the current client is divided to form a teacher model training set and a teacher model validation set. The teacher model training set can be used to train the optimal candidate convolutional neural network, and the teacher model validation set can be used to verify the trained optimal candidate convolutional neural network. The conditions of the teacher model training set and the teacher model validation set can be respectively referred to the corresponding descriptions of the above optimization local training set and optimization local validation set, which will not be described here.

[0088] It should be understood that when the optimal candidate convolutional neural network is trained by using the teacher model training set, the specific training process mode and process can be consistent with the prior art, such as standard training process of the model by using a supervised learning mode. The model training process includes conventional steps such as forward propagation, loss calculation, gradient back propagation, and parameter updating. In a feasible embodiment, the parameters of the supervised learning mode can include: the loss function used in the training is a cross-entropy loss function, the optimizer is an Adam optimizer, and the learning rate and batch_size parameters can be flexibly configured according to the client resource status, and the training rounds are set by the client. As described above, the number of iterations of the training can be 50 rounds to ensure that the model is fully converged and accurately learns the distribution characteristics of the local private data.

[0089] In a specific implementation, after one round of training of the optimal candidate convolutional neural network by using the teacher model training set, the trained optimal candidate convolutional neural network is verified by using the teacher model verification set to obtain a corresponding teacher model verification accuracy. When the number of training rounds of the optimal candidate convolutional neural network reaches a specified number of rounds, the optimal candidate convolutional neural network corresponding to the highest teacher model verification accuracy is taken as the neural network local teacher model. When the neural network local teacher model is used for a classification task, the teacher model verification accuracy specifically refers to the probability of accurately classifying the teacher model verification set. The method for calculating the teacher model verification accuracy can be consistent with the prior art, which will not be described here.

[0090] As described above, in each client, the process of generating the neural network local teacher model does not involve communication with other clients or servers, which can ensure that the local private data is always protected locally in the client. After the training is completed, the client obtains a neural network local teacher model with a locally optimal architecture and fully trained weights. The neural network local teacher model has the ability to express the local private data in the client. After obtaining the neural network local teacher model, the client saves the neural network local teacher model as a model file and uploads the neural network local teacher model to the server at one time through network transmission.

[0091] As described above, each neural network local teacher model has an independent network architecture and parameters, so the neural network local teacher models uploaded by different clients can be in a heterogeneous state. At the same time, each neural network local teacher model is optimally matched with its own client, that is, the network model with the best performance is trained based on the local private data of the client, and the optimal matching condition can be referred to the above description of the generation process of the neural network local teacher model.

[0092] In an embodiment of the present application, when generating the pseudo-supervised training data set, the following steps are included: In the server, based on each neural network local teacher model, a pseudo image generation mechanism optimized by random noise input is used to generate a plurality of pseudo training images, wherein each pseudo training image satisfies the reproduction of the knowledge representation of the corresponding client; Each pseudo training image is loaded into each neural network local teacher model in the server to configure the neural network local teacher model to perform inference prediction on the loaded pseudo training image and generate personalized model inference output information; The server performs consistency perception fusion processing on the personalized model inference output information corresponding to each pseudo training image to generate a fusion soft label after consistency perception fusion processing; The generated fusion soft label and the corresponding pseudo training image are used as a pseudo supervision training sample in the pseudo supervision training data set.

[0093] It should be understood that when generating the pseudo supervision training data set, specifically, generating the pseudo supervision training sample in the pseudo supervision training data set, in an embodiment of the present application, the server first generates a pseudo training image based on the received neural network local teacher model, and then generates a corresponding fusion soft label based on the pseudo training image. The way and process of generating the pseudo training image and the fusion soft label are described below.

[0094] Since the server cannot access the local private data on the client, in order to be able to reproduce and extract the local private data in the client, the present application designs a pseudo image generation mechanism optimized by random noise input to reconstruct the semantic feature information contained in the corresponding uploaded neural network local teacher model of the client.

[0095] It should be noted that when generating the pseudo training image by using the pseudo image generation mechanism optimized by random noise input, a set of random noise images should be initialized for the corresponding neural network local teacher model. Specifically, each random noise image is generated by sampling from a normal distribution with a mean of 0 and a variance of 1, wherein the normal distribution has the property of "maximum entropy" in statistics, indicating the most neutral random variable distribution without prior knowledge, and is therefore very suitable for the starting state of the pseudo training image generation process. The random noise image generated using the normal distribution does not contain any specific structure and only represents pure noise, but will be gradually adjusted to an image with semantic features in the subsequent optimization process. Compared with the uniform distribution, the normal distribution can generate more pixels concentrated around the mean value of the image gray scale, which is closer to the basic statistical characteristics of natural images.

[0096] It should be noted that the feature dimensions of the local private data in different clients should be consistent, and the feature dimensions of the generated random noise images should be consistent with the feature dimensions of the data in all clients, so that after the feature dimensions are determined, a group of random noise images can be generated in the above manner. The feature dimensions of the random noise images are the feature size and the number of channels.

[0097] In an embodiment of the present application, for each neural network local teacher model, when generating pseudo training images using the pseudo image generation mechanism optimized by random noise input, the following steps are included: extracting the task layer weight information of the neural network local teacher model, and constructing a category similarity matrix based on the extracted task layer weight information; Based on the constructed category similarity matrix, a category-aware pseudo-label modeling method is used to construct a corresponding soft label distribution group, which includes a plurality of soft label distribution vectors, wherein when the inference prediction of the neural network local teacher model on the pseudo training image is a classification prediction, the number of distribution probabilities in the soft label distribution vector is consistent with the number of categories of the classification prediction performed by the neural network local teacher model. The server initializes a group of random noise images for the current neural network local teacher model, and optimizes and updates the random noise images in the group of random noise images using each soft label distribution vector to generate a corresponding and class-discriminative pseudo training image after optimization and update.

[0098] Specifically, the task layer of the neural network local teacher model specifically refers to the network layer directly used to complete the task prediction, such as when the neural network local teacher model implements a classification task, the task layer of the neural network local teacher model specifically refers to the last fully connected classification layer of the neural network local teacher model, and when the network task is other, the condition of the task layer in the neural network local teacher model can be determined, which will not be described here.

[0099] The extracted task layer weight information can be in the form of a matrix, such as the task layer weight information can be a weight matrix , wherein, is the number of categories that can be recognized when performing a classification task, represents the input feature dimension of the classification task, that is, the dimension of the feature vector input to the previous layer of the task layer, and the input feature dimension is actually related to the structure of the neural network local teacher model, and more specifically, it is the dimension of the feature vector output by the second last layer (i.e., the layer before the classification layer) of the current neural network local teacher model, so for a determined neural network local teacher model, the input feature dimension , specifically, the th row Vectors can be viewed as categories The representation vector is used to express the category Semantic orientation in feature space.

[0100] When constructing the category similarity matrix based on the task layer weight information, it specifically refers to the weight matrix Construct the category similarity matrix. When constructing the category similarity matrix, we have:

[0101] in, is the category similarity matrix Neidi Rank The category similarity element of the column, is the weight matrix No. A vector of rows, is the weight matrix No. A vector of rows, Represents a vector With vector dot product, is a vector The L2 norm of is a vector The L2 norm of the vector The modular multiplication of the L2 norm of .

[0102] From the above description, we can see that the category similarity matrix is , that is, the category similarity matrix is ​​an L-order square matrix. The above method can be used to determine the category similarity matrix Each element in the category similarity matrix can be constructed. The category similarity matrix can be used to measure the degree of similarity between different categories. For the first element in the category similarity matrix, Row, indicating category Semantic similarity with all other categories.

[0103] After constructing the category similarity matrix, we can use the category-aware pseudo-label modeling method to build the corresponding soft label distribution group. Since the category similarity matrix corresponds to the neural network local teacher model, the corresponding soft label distribution group here specifically refers to the soft label distribution group that corresponds to the neural network local teacher model. The following example illustrates the method and process of constructing the soft label distribution group.

[0104] Dirichlet distribution is a multidimensional continuous distribution defined on a probability simplex, which is often used to model the probability distribution of multiple categories. In order to generate pseudo labels with discrimination and diversity, this paper introduces the diversity adjustment coefficient , where the diversity adjustment coefficient The value of can be: , of course, the diversity adjustment coefficient The value of can also be other cases, when the diversity adjustment coefficient For information on generating soft label distribution groups when other values ​​are used, refer to the instructions here.

[0105] In specific implementation, based on the diversity adjustment coefficient and the category similarity matrix A row as the concentration parameter of the Dirichlet distribution, from which the Dirichlet distribution can be constructed ,in, is the concentration parameter of the Dirichlet distribution, the concentration parameter , concentration parameter Determines the sparsity of the sampled labels. When the concentration parameter When the values ​​of each dimension in are small, the sampling result shows a sharp single-peak structure, and the label vector is closer to the one-hot format; when the concentration parameter When the values ​​of each dimension in are large, the sampling result distribution is smoother, indicating that the label vector has uncertainty for multiple categories. Specifically, when the diversity adjustment coefficient When it is smaller (such as 0.1), the constructed soft label is more sparse, reflecting a strong preference for the target category, which is suitable for generating pseudo images with strong classification signals; in the diversity adjustment coefficient When it is larger (such as 1.0), the soft label is smoother, so that the diversity adjustment coefficient can be used to induce the generation of pseudo training images that fuse multiple category features, thereby improving the diversity and generalization ability of pseudo training images.

[0106] From the above description, it can be seen that the present invention uses Dirichlet distribution as the basic distribution for generating soft label distribution group, and adjusts the diversity adjustment coefficient It can achieve controllable sparsity and diversity of soft labels, thereby improving the diversity and expressiveness of pseudo training images. OK and each diversity adjustment coefficient , the server is distributed from the corresponding Dirichlet A soft label distribution vector (soft labelY) is obtained by sampling from the image to serve as the supervision target for the subsequent image optimization update stage.

[0107] In specific implementation, when the diversity adjustment coefficient The value of , then under each value, combined with the corresponding corresponding Dirichlet distribution can be constructed , and then a soft label Y can be sampled L , such as for , when the diversity adjustment coefficient is 0.1, a Dirichlet distribution can be constructed, and then a soft label Y can be sampled , when the diversity adjustment coefficient is 1.0, a Dirichlet distribution can be constructed, and then another soft label Y can be sampled, and other cases are not repeated here.

[0108] As can be seen from the above description, the number of soft label distribution groups including soft label distribution vectors can be related to the value of the diversity adjustment coefficient and the number of rows of the category similarity matrix. In addition, the number of distribution probabilities in the soft label distribution vector is consistent with the number of categories predicted by the neural network local teacher model, such as soft label Y L , and the number of rows and columns of the category similarity matrix is L, such as soft label Y can be [0.05, 0.10, 0.70, 0.10, 0.05], at this time, L is 5, and each element in soft label Y represents the probability distribution of the corresponding category, such as 0.70 representing the probability distribution of the third category, and the case can be referred to the description here.

[0109] As can be seen from the above description, the Dirichlet distribution is constructed based on the concentration parameter , and the Dirichlet distribution is a multivariate distribution defined on an L-dimensional probability simplex. When sampling to generate soft label Y, it represents the "mixed representation weight" of the current category in different semantic categories, and the sampling result is a real number vector (i.e. probability distribution vector) that satisfies non-negativity and sum of 1.

[0110] After generating the soft label distribution group in the above manner, the server can initialize a set of random noise images for the current neural network local teacher model, and the case of each random noise image in the set of random noise images can be referred to the above description. Then, each soft label distribution vector can be used to optimize and update each random noise image to generate a corresponding pseudo training image with category discriminability, that is, a soft label distribution vector can be used to optimize and update a random noise image to generate a pseudo training image, and the pseudo training image generated here corresponds to the soft label distribution vector.

[0111] In one embodiment of the present invention, optimizing and updating each random noise image includes: The random noise image is loaded into the local teacher model of the neural network, and the corresponding noise image logits output information is obtained through inference and prediction by the local teacher model of the neural network; The noise image logits output information is temperature scaled and normalized by softmax to generate the noise image prediction probability distribution; The cross entropy loss between each soft label distribution vector and the predicted probability distribution of the noise image is used as the optimization target. During the optimization process, the random noise image is updated by gradient descent until the cross entropy loss as the optimization target is minimized, and the corresponding pseudo training image is generated.

[0112] Specifically, after generating a random noise image using the above method, a random noise image is loaded into a neural network local teacher model, and then the neural network local teacher model is used to perform inference prediction. From the above description, it can be seen that the inference prediction performed by the neural network local teacher model on the received random noise image can be a classification prediction of the random noise image. After inference prediction, the noise image logits output information can be obtained. It should be noted that the noise image logits output information is the original prediction score generated by the neural network local teacher model in the output layer, which has not been normalized. It is usually converted into a probability distribution through softmax normalization later, that is, after softmax normalization, a noise image prediction probability distribution can be generated.

[0113] In one embodiment of the present invention, the noise image logits output information is temperature scaled and then input into a softmax function to convert it into a noise image prediction probability distribution, wherein the noise image prediction probability distribution can be expressed as:

[0114] in, For the A neural network local teacher model for random noise images The noise image logits output information generated after inference prediction, is the temperature parameter, Predict the probability distribution for the corresponding noise image.

[0115] It should be noted that when performing classification prediction tasks, the temperature parameter It is used to scale the output information of the noise image logits, thereby controlling the distribution characteristics of the softmax output probability. Specifically, a larger temperature parameter The difference between the prediction probabilities of various categories can be effectively reduced, thereby enhancing the inter-class structure information contained in the soft label, and improving the stability and gradient smoothness of the model in the pseudo-image generation process. The temperature parameter is generally set empirically and can be flexibly adjusted according to the task complexity and data characteristics. The common value range is 2-10. It can be understood that after the temperature scaling processing and the conversion to the noise image prediction probability distribution by the softmax function, the pseudo-training image can be more stably guided to evolve in the semantic space when the uncertainty of the neural network local teacher model output is high. Therefore, it can be known that the temperature scaling is one of the key mechanisms of the present application for improving the quality and diversity of the pseudo-training image.

[0116] After obtaining the noise image prediction probability distribution of each random noise image, the cross-entropy loss between a soft label distribution vector and the noise image prediction probability distribution is taken as the optimization objective, and the gradient descent update is performed on the random noise image. Specifically, under the premise of freezing the corresponding model parameters of the neural network local teacher model, the random noise image itself is taken as the optimization variable, the cross-entropy loss between its prediction probability distribution and the soft label Y is minimized, the gradient is back-propagated and the pixel value in the random noise image is updated, so as to guide it to gradually approach the semantic target direction represented by the soft label Y in the feature space. Then,

[0117] wherein, is a soft label distribution vector, is a cross-entropy loss, which is used to measure the difference between the soft label distribution vector soft label Y and the prediction probability distribution output by the kth neural network local teacher model; is the final pseudo-training image after optimization and update, that is, the pseudo-training image conforming to the semantic prior.

[0118] Specifically, the cross-entropy loss optimization process between the above-mentioned soft label distribution vector and the noise image prediction probability distribution is realized by using the standard back-propagation algorithm. After the gradient descent update, the pixel value in the random noise image gradually approaches the corresponding semantic direction of the current soft label distribution vector in each iteration, and the cross-entropy loss as the optimization objective is minimized, thereby "inducing" the pseudo-training image with class discriminability.

[0119] As can be seen from the above description, the generated pseudo training image is associated with a random noise image, a soft label distribution vector, and a neural network local teacher model. When one of the random noise image, soft label distribution vector, or neural network local teacher model is replaced, a corresponding pseudo training image can be generated. In other words, when generating pseudo training images, the neural network local teacher models generated by different clients are independent of each other. Repeating this generation operation can generate multiple corresponding pseudo training images. The server aggregates all pseudo training images to generate a global pseudo training image set.

[0120] It should be understood that although the server does not directly access the local private data within any client, since these pseudo-training images are generated by reverse-guiding the neural network local teacher models generated by different clients, the model parameters of the neural network local teacher models already implicitly reflect the statistical characteristics and classification decision boundaries of their training data. Therefore, the generated pseudo-training images can effectively reproduce the client's knowledge representation, providing a high-quality input foundation for subsequent distillation training. In addition, the "training data" here specifically refers to the local private data within the client used to generate the neural network local teacher model.

[0121] From the description of the generation process of each pseudo training image, it can be seen that each pseudo training image satisfies the knowledge representation of the corresponding client to be reproduced, where the "corresponding client" specifically refers to the client corresponding to the local teacher model of the neural network used to generate the current pseudo training image.

[0122] From the above description, it can be seen that after the pseudo training image is generated in the above manner, the pseudo training image can be used to generate a fused soft label. Specifically, when generating a fused soft label, a pseudo training image needs to be loaded into each neural network local teacher model in the server respectively, that is, the pseudo training image is loaded into the neural network local teacher model uploaded to the server by all clients. After that, each neural network local teacher model can be used to perform inference prediction on the current pseudo training image, and the corresponding personalized model inference output information can be generated after the inference prediction.

[0123] As can be seen from the above description, the inference prediction performed by the neural network local teacher model on the pseudo training image can be a classification prediction. When the inference prediction is a classification prediction, the generated personalized model inference output information is the classification prediction information for the current pseudo training image, and the dimension of each personalized model output inference output information is 1×L. It should be understood that based on a pseudo training image, multiple different personalized model inference output information can be generated, and the number of personalized model inference output information is consistent with the number of neural network local teacher models received in the server.

[0124] For each pseudo-training image, the inference prediction can generate multiple personalized model inference output information. In order to improve the reliability and adaptability of the subsequent generated fusion soft label, the server performs consistency perception fusion processing on all corresponding personalized model inference output information, and generates a fusion soft label after consistency perception fusion processing. The consistency perception fusion processing method and process are described below.

[0125] In an embodiment of the present application, when the server performs consistency perception fusion processing on all corresponding personalized model inference output information of each pseudo-training image, it includes: The server calculates the personalized model average output information of the personalized model inference output information corresponding to the current pseudo-training image, and calculates the teacher consistency measure index based on the calculated personalized model average output information. When the teacher consistency measure index is not greater than the consistency judgment threshold, the personalized model average output information is taken as the fusion soft label, otherwise, all personalized model inference output information is output fused based on the main teacher fusion mechanism to generate the corresponding fusion soft label after output fusion.

[0126] In the server, each pseudo-training image is input into all received neural network local teacher models to obtain K corresponding personalized model inference output information, wherein one corresponding personalized model inference output information is generated after inference prediction by each neural network local teacher model. The personalized model inference output information is a logits vector, which represents the original prediction score of the pseudo-training image belonging to each category. Due to the difference between each neural network local teacher model, the output scale may not be consistent. The server performs temperature scaling processing on all personalized model inference output information, that is, each set of logits is divided by a positive temperature parameter and then input into a softmax function to obtain K normalized prediction probability distributions. Each prediction distribution is regarded as a personalized model inference output information. The temperature scaling processing method and process can refer to the above description, which will not be repeated here.

[0127] Specifically, the server first calculates the teacher consistency measure index. In an embodiment of the present application, when the teacher consistency measure index is calculated, it has:

[0128] Wherein, is the teacher consistency measure index of the pseudo-training image is the number of neural network local teacher models received by the server is the number of neural network local teacher models received by the server is the first The teacher consistency measurement index The personalized model inference output information generated by performing inference prediction, The personalized model average output information, The divergence of the personalized model inference output information calculated based on the personalized model average output information .

[0129] It can be understood that the teacher consistency measurement index measures the difference between the inference prediction of each neural network local teacher model and the personalized model average output information, reflecting the consistency of the inference prediction of the pseudo training image under all neural network local teacher models. If the personalized model inference output information distribution corresponding to all neural network local teacher models is similar, the teacher consistency measurement index is small, indicating that the inference prediction of the pseudo training image is consistent in multiple neural network local teacher models; on the contrary, if the inference prediction of multiple neural network local teacher models is significantly different, the teacher consistency measurement index is large.

[0130] A consistency judgment threshold can be preset in the server, which can be an empirical value, such as 0.05. After the teacher consistency measurement index is calculated, the teacher consistency measurement index can be compared with the consistency judgment threshold. If , the personalized model average output information is taken as the fusion soft label, otherwise, based on the main teacher fusion mechanism, all personalized model inference output information is output fused to generate the corresponding fusion soft label after output fusion; wherein, is the consistency judgment threshold.

[0131] It can be understood that when generating the fusion soft label corresponding to each pseudo training image in the above manner, the consistency of the inference prediction of different pseudo training images under all neural network local teacher models can be adapted, so that the semantic quality and training stability of the fusion soft label can be improved.

[0132] In an embodiment of the present application, when all personalized model inference output information is output fused based on the main teacher fusion mechanism, it includes: The server selects one neural network local teacher model as the main teacher model, and the remaining neural network local teacher models as auxiliary teacher models, wherein the main teacher model is at least the neural network local teacher model with the most complex network structure; When output fusion is performed, then:

[0133] wherein, is a pseudo training image , a fusion soft label of the pseudo training image, is a main teacher model weight coefficient, is a personalized model inference output information generated by the main teacher model performing inference prediction on the pseudo training image , is a number of neural network local teacher models received by the server, is the first neural network local teacher model, is the first neural network local teacher model,

[0134] After receiving all the neural network local teacher models sent by the clients, the server can determine the network structure corresponding to each neural network local teacher model by using the technical means commonly used in the technical field, and thus the neural network local teacher model with the most complex network structure can be determined. In an embodiment of the present application, the neural network local teacher model with the most complex network structure can be configured as the main teacher model, so that the main teacher model has strong expression ability and feature modeling ability, can provide more comprehensive semantic information, and guide the fusion label to be more representative. After the main teacher model is determined, the remaining neural network local teacher models are all auxiliary teacher models. The main teacher model weight coefficient is generally 0.7, which is used to emphasize the leading position of the main teacher in the fusion.

[0135] As can be known from the above description of generating the fusion soft label, the present application can realize knowledge fusion without accessing any local private data and local private data label of the client, in other words, without using any real label, the personalized model inference output of all neural network local teacher models can be completely relied on to generate the corresponding fusion soft label through information fusion, which is used as the supervision label of the corresponding pseudo training image.

[0136] Further, when there is a large difference in inference prediction among all the neural network local teacher models, the present application dynamically gives a higher weight to the main teacher model with more complex structure and stronger expression ability, guides to generate a more reliable supervision label, while retaining the auxiliary information of the auxiliary teacher model, avoids overfitting to a single neural network local teacher model, and realizes the alignment and unified expression of the knowledge of the cross-structure and multi-source neural network local teacher models. The distribution of the fusion soft label after fusion is more smooth, which is helpful for the basic student model to learn more generalizable class relationships through distillation training.

[0137] In an embodiment of the present application, when generating a global shared student model after distillation training of the basic student model, the method comprises: The basic student model is trained using the pseudo-supervised training dataset. During model training, the pseudo-training images in each pseudo-supervised training sample are loaded into the basic student model respectively, and the student model inference output information is generated after inference prediction by the basic student model. After each round of model training, the training divergence between the inference output information of each student model and the corresponding fusion soft label is calculated, and the calculated training divergence is used as the distillation loss. Thereafter, the basic student model is updated with the minimization of the distillation loss as the optimization goal; Use the student training test set to test the updated basic student model and determine the corresponding test accuracy after testing; After the model training of the base student model reaches the target number of distillation training rounds, the highest test accuracy is determined, and the base student model corresponding to the highest test accuracy is used as the global shared student model.

[0138] It should be noted that when distillation training generates a globally shared student model, a base student model should first be constructed. This base student model can be a standardized student model that remains consistent across all experimental scenarios, facilitating deployment and migration, independent of the architecture of any client's local neural network teacher model. The base student model can typically be a convolutional neural network of medium complexity, exhibiting good expressiveness and generalization. For example, a base student model of medium complexity can employ four convolutional modules, each containing two 3×3 convolutional layers (both using Batch Normalization and ReLU activation functions) and a 2×2 max pooling layer, ultimately connected via a Flatten layer to a three-layer fully connected network (including 256- and 64-dimensional hidden layers and a final classification output layer). In this case, the total number of network parameters for the entire base student model is kept below one million, achieving both feature extraction capabilities and computational efficiency, making it suitable for multi-class image recognition tasks. It should be understood that the base student model constructed in the present invention is not limited to the examples presented here; the model depth and width can also be appropriately adjusted based on the computing resources of the deployment platform, and detailed examples will not be provided here.

[0139] After building the basic student model, the pseudo-supervised training dataset generated above is used to train the basic student model, thereby achieving knowledge fusion and generalization of all neural network local teacher models. During the training process, the server inputs each pseudo-training image into the basic student model and obtains the standard output information of the student model predicted by the basic student model. Generally, the standard output information of the student model is in the form of logits, and the standard output information of the student model can be expressed as . Then, Divide by the set temperature coefficient and a softmax function is applied for normalization to obtain a prediction probability distribution of the integrated student model. The prediction probability distribution obtained is the student model inference output information. For the student model inference output information, there is:

[0140] wherein, is a pseudo training image corresponding to the student model inference output information, is a temperature coefficient, and the temperature coefficient The case can be referred to the above description of the temperature coefficient.

[0141] As can be known from the above description, for each pseudo training image, the fusion soft label generated based on the pseudo training image is used as a supervision signal for training. In an embodiment of the present application, the server uses Kullback-Leibler divergence as a distillation loss function to measure the difference between the student model inference output information of the base student model and the corresponding fusion soft label, and there is:

[0142] wherein, is a distillation loss, is a pseudo supervised training set, is a pseudo training image corresponding to the fusion soft label, is a training divergence between the fusion soft label and the student model inference output information.

[0143] During model training, the base student model is updated with the optimization goal of minimizing the distillation loss. The specific way of updating the base student model can be consistent with the prior art, for example, the base student model can be updated by back propagation. The specific updating method and process are not described here.

[0144] After one round of model training of the base student model using the pseudo supervised training data set, the updated base student model can be tested using the student training test set, and the corresponding test accuracy can be determined after testing. It should be noted that the student training test set can be generated using the data with supervision information in the server. Of course, the test samples in the student training test set should correspond to the corresponding types of the training samples in the pseudo supervised training data set, so as to meet the corresponding test requirements. When testing, the corresponding test accuracy can be calculated and determined using the commonly used method in the technical field. The way of obtaining the test accuracy can be consistent with the prior art, and is not described here.

[0145] In implementation, the number of rounds of model training of the base student model using the pseudo-supervised training dataset can be configured, i.e., the target distillation training round number is configured. The number of rounds of model training configured can be selected according to needs. In each round of model training, generally, all training samples in the pseudo-supervised training dataset participate in model training of the base student model. When the model training of the base student model reaches the target distillation training round number, the highest test accuracy is determined, and the base student model corresponding to the highest test accuracy is taken as the globally shared student model.

[0146] It should be understood that when the globally shared student model is trained, it indicates that the federated learning process of the present application ends. As can be known from the foregoing distillation training process, the server does not introduce any real label supervision in the training, nor does it involve gradient update of the neural network local teacher model, but only relies on the pseudo-supervised training dataset to perform standard back propagation at the server end to minimize the distillation loss.

[0147] The globally shared student model can effectively absorb the output distribution features of the neural network local teacher model from each heterogeneous client, and realize fusion of distributionally heterogeneous knowledge while maintaining a unified structure. The federated learning method of the present application does not need to access the local private data of the client, does not need to unify the architecture of the neural network local teacher model, and only needs one communication to obtain a globally shared student model with strong generalization ability, thereby significantly improving the flexibility and efficiency of the federated learning in actual deployment scenarios.

Claims

1. A federated learning method that supports heterogeneous model architecture search and zero-shot knowledge fusion, characterized by: The federated learning method includes: A federated learning system is provided for performing federated learning, wherein the federated learning system includes a server and a client group adapted to be connected to the server, the client group includes a plurality of clients, each client having local private data, and the local private data of all clients has a non-independent and identically distributed property; When configuring a federated learning system for federated learning, it at least includes configuring each client to perform teacher model generation processing and configuring the server to perform student model generation processing, wherein, When any client executes the teacher model generation process, it searches and generates the neural network local teacher model that best matches the client based on the local private data in the client, and sends the generated neural network local teacher model to the connected server. After the server executes the student model generation process, it generates at least a globally shared student model. The generation of the globally shared student model includes: Based on all received neural network local teacher models, a pseudo-supervised training dataset is generated, wherein the pseudo-supervised training dataset includes a plurality of pseudo-supervised training samples, each pseudo-supervised training sample includes a pseudo-training image that can reproduce the knowledge representation of the corresponding client and a fused soft label corresponding to the pseudo-training image, and when generating each pseudo-supervised training sample, the server does not access the local private data of the corresponding client in the client group; The pseudo-supervised training dataset is used to distill and train the constructed basic student model, and a global shared student model is generated after distilling and training the basic student model.

2. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 1 is characterized in that: Each client performs the teacher model generation process, including: Constructing a network particle swarm, wherein the network particle swarm includes a plurality of network particles, each network particle has a network coding sequence, and represents a candidate convolutional neural network based on the corresponding network coding sequence, and the length of the network particle is the network depth of the represented candidate convolutional neural network; Based on the local private data of the current client, the PSO optimization is performed on the network particle swarm constructed above to obtain the optimal candidate convolutional neural network corresponding to the optimal network particle through PSO optimization; The optimal candidate convolutional neural network is trained based on the local private data of the current client to generate a neural network local teacher model after model training.

3. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 2 is characterized in that: Based on the local private data in each client, at least an optimized local training set and an optimized local validation set are constructed; When performing PSO optimization on a network particle swarm, it includes at least several network training optimization processes performed in sequence, wherein each network training optimization process includes sequential optimization training processing and particle update processing; When performing the optimization training process, the network coding sequence of each network particle is converted into the corresponding candidate convolutional neural network, and the target round model training is performed on each candidate convolutional neural network generated by the conversion using the optimization local training set, so as to generate the corresponding candidate convolutional neural network after model training; When executing particle update processing, each candidate convolutional neural network is verified using the optimal verification set, and the network accuracy calculated by the network verification is used as the fitness value of the current candidate convolutional neural network; Based on the fitness value of each candidate convolutional neural network, determine the individual optimal convolutional neural network corresponding to each candidate convolutional neural network, and determine the global optimal convolutional neural network based on all the individual optimal convolutional neural networks; When the PSO optimization of the network particle swarm fails to reach the target state, each network particle is updated based on the global optimal convolutional neural network and the individual optimal convolutional neural network of each network particle to update the network coding sequence of each network particle and perform the next network training optimization process; When the PSO optimization of the network particle swarm reaches the target state, the network particle corresponding to the global optimal convolutional neural network is taken as the optimal network particle, and the global optimal convolutional neural network configuration is taken as the optimal candidate convolutional neural network.

4. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to any one of claims 1 to 3, characterized in that: When generating a pseudo-supervised training dataset, include: In the server, based on each neural network’s local teacher model, a pseudo-image generation mechanism optimized with random noise input is used to generate several pseudo-training images, where each pseudo-training image satisfies the requirements of reproducing the knowledge representation of the corresponding client. Loading each pseudo training image into each neural network local teacher model in the server, respectively, to configure the neural network local teacher model to perform inference prediction on the loaded pseudo training image and generate personalized model inference output information; The server performs consistency-aware fusion processing on all the personalized model inference output information corresponding to each pseudo training image to generate a fused soft label after consistency-aware fusion processing; The generated fusion soft labels and the corresponding pseudo training images are used as a pseudo supervised training sample in the pseudo supervised training dataset.

5. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 4 is characterized in that: For each neural network local teacher model, a pseudo image generation mechanism optimized with random noise input is used to generate pseudo training images, including: Extracting task-layer weight information of the neural network local teacher model, and constructing a category similarity matrix based on the extracted task-layer weight information; Based on the constructed category similarity matrix, a category-aware pseudo-label modeling method is used to construct a corresponding soft label distribution group, wherein the soft label distribution group includes a number of soft label distribution vectors, wherein when the inference prediction made by the neural network local teacher model on the pseudo training image is a classification prediction, the number of distribution probabilities in the soft label distribution vector is consistent with the number of categories predicted by the neural network local teacher model; The server initializes a set of random noise images for the current neural network local teacher model, and optimizes and updates each random noise image using each soft label distribution vector to generate a corresponding pseudo training image with category discrimination after optimization and updating.

6. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 5 is characterized in that: When optimizing and updating each random noise image, it includes: The random noise image is loaded into the local teacher model of the neural network, and the corresponding noise image logits output information is obtained through inference and prediction by the local teacher model of the neural network; The noise image logits output information is temperature scaled and normalized by softmax to generate the noise image prediction probability distribution; The cross entropy loss between each soft label distribution vector and the predicted probability distribution of the noise image is used as the optimization target. During the optimization process, the random noise image is updated by gradient descent until the cross entropy loss as the optimization target is minimized, and the corresponding pseudo training image is generated.

7. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 4 is characterized in that: The server performs consistency-aware fusion processing on all the personalized model inference output information corresponding to each pseudo training image, including: The server calculates personalized model average output information of the personalized model inference output information corresponding to the current pseudo training image, and generates a teacher consistency measurement indicator based on the calculated personalized model average output information; When the teacher consistency metric is not greater than the consistency judgment threshold, the average output information of the personalized model is used as the fusion soft label. Otherwise, all the personalized model reasoning output information is output fused based on the main teacher fusion mechanism to generate the corresponding fusion soft label after output fusion.

8. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 7 is characterized in that: When calculating the teacher consistency metric, we have: in, Pseudo training images Teacher consistency metrics, is the number of neural network local teacher models received by the server, For the A neural network local teacher model for pseudo training images Perform inference prediction to generate personalized model inference output information, is the average output information of the personalized model, Calculate the personalized model inference output information based on the average output information of the personalized model The divergence of .

9. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to claim 7 is characterized in that: When all personalized model inference output information is output and fused based on the master teacher fusion mechanism, it includes: The server selects a neural network local teacher model as a main teacher model and uses the remaining neural network local teacher models as auxiliary teacher models, wherein the main teacher model is at least the neural network local teacher model with the most complex network structure; When output fusion is performed, there are: in, Pseudo training images Fusion soft label, is the weight coefficient of the main teacher model, Pseudo training images for the main teacher model Perform inference prediction to generate personalized model inference output information, is the number of neural network local teacher models received by the server, For the A neural network local teacher model for pseudo training images Perform inference prediction to generate personalized model inference output information.

10. The federated learning method supporting heterogeneous model architecture search and zero-shot knowledge fusion according to any one of claims 1 to 3, characterized in that: When generating a global shared student model after distillation training of the base student model, it includes: The basic student model is trained using the pseudo-supervised training dataset. During model training, the pseudo-training images in each pseudo-supervised training sample are loaded into the basic student model respectively, and the student model inference output information is generated after inference prediction by the basic student model. After each round of model training, the training divergence between the inference output information of each student model and the corresponding fusion soft label is calculated, and the calculated training divergence is used as the distillation loss. Thereafter, the basic student model is updated with the minimization of the distillation loss as the optimization goal; Use the student training test set to test the updated basic student model and determine the corresponding test accuracy after testing; After the model training of the base student model reaches the target number of distillation training rounds, the highest test accuracy is determined, and the base student model corresponding to the highest test accuracy is used as the global shared student model.

Images