Information Processing Method, Apparatus, Device, Storage Medium, and Program Product

Abstract

The present invention discloses an information processing method, apparatus, device, storage medium and program product. The method includes: obtaining the N-th fused classification model and the N-th fused feature generation model sent by a server; training the N-th data generation model and the N-th feature extraction model to obtain the (N + 1)-th data generation model and the first updated feature extraction model in the (N + 1)-th time; training the first updated feature extraction model in the (N + 1)-th time and the N-th fused classification model to obtain the (N + 1)-th classification model and the second updated feature extraction model in the (N + 1)-th time; training the N-th fused feature generation model through an adversarial network so that the output result of the feature generation model approximates the result output by the feature extraction model. The present invention realizes improving the performance of the local model on the premise of effectively protecting privacy, making the output result more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of information processing, and in particular, to an information processing method, apparatus, device, storage medium, and program product. Background Art

[0002] Deep Neural Networks (DNNs) have achieved great success in many fields, including computer vision, natural language processing, and recommendation systems. Their success largely depends on massive amounts of training data. However, in many scenarios, the training data is usually scattered across different organizations. Due to privacy and regulatory issues, these organizations are usually reluctant to directly share their data. To alleviate these problems, Federated Learning (FL) has been proposed. It enables multiple clients (organizations or terminal devices, which can be regarded as data providers) to collaboratively build machine learning models without revealing the private data of the clients.

[0003] Federated Learning has the ability to protect privacy. Among them, to prevent data leakage and still enjoy the benefits of federated learning, split learning can be combined with federated learning to propose federated split learning to protect privacy. Specifically, federated split learning divides the neural network model of the client into two parts, namely a private feature extractor and a public classifier, and only shares the public classifier with a coordinator (such as a server). However, federated split learning will have a non-negligible performance degradation because the private model is not enhanced through federated learning. Therefore, to achieve a high level of privacy protection, the feature extractor can be retained locally to protect privacy. At the same time, it shares the generator of the client with the server to replace the extractor.

[0004] However, in the above method during the local general knowledge fusion process of the client, since the training data (including features and labels) corresponding to each client belongs to non-independent and identically distributed data, therefore, under the premise of protecting privacy, there is a problem of non-independent and identically distributed data in the above method, such that each client can only be applicable to the local distributed data and cannot cover more data, thereby resulting in a low performance of the local model of the client and a low accuracy of the output result. Summary of the Invention

[0005] The main objective of the present invention is to provide an information processing method, apparatus, device, storage medium, and program product, aiming to improve the performance of the local model and make the output result more accurate under the premise of effectively protecting privacy.

[0006] To achieve the above object, the present invention provides an information processing method, which is applied to a data provider among multiple participants participating in federated learning, and is used to train a feature generation model and a classification model. After sequentially inputting an image to be predicted into the feature generation model and the classification model, a corresponding classification result is obtained; the method includes:

[0007] Obtain the Nth fused classification model and the Nth fused feature generation model sent by the server; wherein, the feature generation model is used to generate image features;

[0008] According to the Nth fused feature generation model and the Nth fused classification model, train the Nth data generation model and the Nth feature extraction model to obtain the (N + 1)th data generation model and the first updated feature extraction model in the (N + 1)th time. The data generation model is used to generate pseudo-images, and the feature extraction model is used to extract image features;

[0009] According to the pseudo-images generated by the (N + 1)th data generation model and the real images, train the first updated feature extraction model in the (N + 1)th time and the Nth fused classification model to obtain the (N + 1)th classification model and the second updated feature extraction model in the (N + 1)th time. The classification model is used to determine the corresponding result according to the image features, and the second updated feature extraction model in the (N + 1)th time is used as the feature extraction model in the (N + 1)th time;

[0010] Use the image features extracted by the (N + 1)th feature extraction model as real data, and use the image features generated by the Nth fused feature generation model as pseudo data. Through an adversarial network, train the Nth fused feature generation model so that the output result of the feature generation model approximates the result output by the feature extraction model.

[0011] Optionally, the step of training the Nth data generation model and the Nth feature extraction model according to the Nth fused feature generation model and the Nth fused classification model to obtain the (N + 1)th data generation model and the first updated feature extraction model in the (N + 1)th time includes:

[0012] Input the first target random number in the random numbers and the first preset class label corresponding to the first target random number into the Nth data generation model to obtain a first pseudo-image, and the real label corresponding to the first pseudo-image is the first preset class label;

[0013] Input the first pseudo-image into the Nth feature extraction model to obtain first image features;

[0014] Input the second target random number among the random numbers and the corresponding second preset category label into the feature generation model after the Nth fusion to obtain a second image feature. The true label corresponding to the second image feature is the second preset category label, and the first preset category label is the same as the second preset category label.

[0015] According to the first image feature and the second image feature, through the classification model after the Nth fusion, determine the data generation model for the (N + 1)th time and the feature extraction model updated for the first time in the (N + 1)th time; wherein, the feature extraction model updated for the first time in the (N + 1)th time is used to support the update of the classification model.

[0016] Optionally, the step of determining the data generation model for the (N + 1)th time and the feature extraction model updated for the first time in the (N + 1)th time according to the first image feature and the second image feature through the classification model after the Nth fusion includes:

[0017] Input the first image feature and the second image feature into the classification model after the Nth fusion to obtain a first image category probability distribution and a second image category probability distribution;

[0018] Calculate the distance loss value between the first image category probability distribution and the second image category probability distribution;

[0019] Based on the distance loss value, calculate the gradient value of the data generation model for the Nth time and the first gradient value of the feature extraction model for the Nth time respectively;

[0020] Update the data generation model for the Nth time according to the gradient value of the data generation model for the Nth time to obtain the data generation model for the (N + 1)th time, and update the feature extraction model for the Nth time according to the first gradient value to obtain the feature extraction model updated for the first time in the local training for the (N + 1)th time.

[0021] Optionally, the step of training the feature extraction model updated for the first time in the (N + 1)th time and the classification model after the Nth fusion according to the pseudo-image and the real image generated by the data generation model for the (N + 1)th time to obtain the classification model for the (N + 1)th time and the feature extraction model updated for the second time in the (N + 1)th time includes:

[0022] Input the first target random number and the first preset category label into the data generation model for the (N + 1)th time to obtain a second pseudo-image;

[0023] Obtain a first original image containing the true label from the real images, and input the first original image and the second pseudo-image into the feature extraction model updated for the first time in the (N + 1)th time to obtain a third image feature;

[0024] Input the third image feature into the classification model after the Nth fusion to obtain a first predicted class label, and calculate a first loss value between the first preset class label and the true label corresponding to the first original data and the corresponding first predicted class label;

[0025] According to the first loss value, calculate a second gradient value of the feature extraction model after the first update in the (N + 1)th time and a gradient value of the classification model after the Nth fusion respectively, and update the feature extraction model after the first update in the (N + 1)th time according to the second gradient value to obtain the feature extraction model after the second update in the (N + 1)th time, and update the classification model after the Nth fusion according to the gradient value of the classification model after the Nth fusion to obtain the classification model in the (N + 1)th time.

[0026] Optionally, using the image features extracted by the feature extraction model in the (N + 1)th time as true data and the image features generated by the feature generation model after the Nth fusion as pseudo data, training the feature generation model after the Nth fusion through an adversarial network includes:

[0027] Input the real image into the feature extraction model in the (N + 1)th time to obtain a fourth image feature;

[0028] Input the random number and the preset class label corresponding to the random number into the feature generation model after the Nth fusion to obtain a fifth image feature;

[0029] According to the fourth image feature and the fifth image feature, calculate a first target loss value of the adversarial network through the feature discrimination model in the Nth time, and update the feature discrimination model in the Nth time and the feature generation model after the Nth fusion respectively according to the first target loss value to obtain the feature discrimination model in the (N + 1)th time and the feature generation model in the (N + 1)th time;

[0030] Among them, the feature discrimination model is used to distinguish whether the image feature is true data or pseudo data.

[0031] Optionally, before obtaining the classification model after the Nth fusion and the feature generation model after the Nth fusion sent by the server, the method further includes:

[0032] Train an initial classification model and an initial feature extraction model according to the real image to obtain the classification model in the first time and the feature extraction model in the first time, and send the classification model in the first time to the server so that the server fuses each classification model in the first time to obtain the classification model after the first fusion;

[0033] Taking the image features extracted by the first feature extraction model as real data and the image features generated by the initial feature generation model as pseudo data, training the initial feature generation model through an adversarial network to obtain the first feature generation model, and sending the first feature generation model to the server so that the server fuses the first feature generation models to obtain the first fused feature generation model.

[0034] Optionally, training the initial classification model and the initial feature extraction model according to the real image to obtain the first classification model and the first feature extraction model includes:

[0035] Inputting the real image into the initial feature extraction model to obtain the sixth image features;

[0036] Inputting the sixth image features into the initial classification model to obtain the third predicted class label, and calculating the third loss value between the third predicted class label and the real label corresponding to the real image;

[0037] Calculating the gradient value of the initial feature extraction model and the gradient value of the initial classification model respectively according to the third loss value;

[0038] Updating the initial feature extraction model according to the gradient value of the initial feature extraction model to obtain the first feature extraction model, and updating the initial classification model according to the gradient value of the initial classification model to obtain the first classification model;

[0039] Among them, the first feature extraction model is used to support the operation of updating the feature generation model.

[0040] Optionally, after obtaining the first classification model and the first feature extraction model, the method further includes:

[0041] Inputting the random number and the preset class label corresponding to the random number into the initial data generation model to obtain the third pseudo image;

[0042] Inputting the third pseudo image into the first feature extraction model to obtain the seventh image features;

[0043] Inputting the seventh image features into the first classification model to obtain the fourth predicted class label, and calculating the fourth loss value between the fourth predicted class label and the preset class label;

[0044] Updating the initial data generation model according to the fourth loss value to obtain the first data generation model.

[0045] Optionally, using the image features extracted by the first feature extraction model as real data and the image features generated by the initial feature generation model as pseudo data, training the initial feature generation model through an adversarial network to obtain the first feature generation model, including:

[0046] Input the real image into the first feature extraction model to obtain the eighth image features;

[0047] Input the random number and the preset class label into the initial feature generation model to obtain the ninth image features;

[0048] Input the eighth image features and the ninth image features into the initial feature discrimination model respectively, calculate the second target loss value of the adversarial network, and update the initial feature discrimination model and the initial feature generation model respectively according to the second target loss value to obtain the first feature discrimination model and the first feature generation model.

[0049] The present invention also provides an information processing method, which is applied to a server among multiple participating parties participating in federated learning; the method includes:

[0050] Obtain the (N + 1)-th classification model and the (N + 1)-th feature generation model sent by each data provider among multiple participating parties; wherein, each data provider is used to train a feature generation model and a classification model, and after sequentially inputting the image to be predicted into the feature generation model and the classification model, obtain the corresponding classification result; the feature generation model is used to generate image features, and the classification model is used to output the corresponding classification result;

[0051] Fuse each (N + 1)-th classification model to obtain the (N + 1)-th fused classification model, and send the (N + 1)-th fused classification model to each data provider respectively, so that each data provider conducts the (N + 2)-th classification model training;

[0052] Fuse each (N + 1)-th feature generation model to obtain the (N + 1)-th fused feature generation model, and send the (N + 1)-th fused feature generation model to each data provider respectively, so that each data provider conducts the (N + 2)-th feature generation model training;

[0053] Among them, the (N + 1)-th feature generation model is obtained by each data provider using the image features extracted by the (N + 1)-th feature extraction model as real data and the image features generated by the feature generation model after the N-th fusion as pseudo data, and training the feature generation model after the N-th fusion through an adversarial network; the (N + 1)-th classification model and the (N + 1)-th feature extraction model are obtained by each data provider training the first updated feature extraction model in the (N + 1)-th time and the classification model after the N-th fusion according to the pseudo images and real images generated by the (N + 1)-th data generation model; the (N + 1)-th data generation model and the first updated feature extraction model in the (N + 1)-th time are both obtained by each data provider training the N-th data generation model and the N-th feature extraction model according to the feature generation model after the N-th fusion and the classification model after the N-th fusion, where the data generation model is used to generate pseudo images and the feature extraction model is used to extract image features.

[0054] The present invention also provides an information processing device, which is applied to a data provider among multiple participants participating in federated learning, and is used to train a local feature generation model and a common classification model, and obtain corresponding classification results after sequentially inputting an image to be predicted into the feature generation model and the classification model; the device includes:

[0055] An acquisition module, configured to acquire the classification model after the N-th fusion and the feature generation model after the N-th fusion sent by the server; among them, the feature generation model is used to generate image features;

[0056] A first processing module, configured to train the N-th data generation model and the N-th feature extraction model according to the feature generation model after the N-th fusion and the classification model after the N-th fusion, to obtain the (N + 1)-th data generation model and the first updated feature extraction model in the (N + 1)-th time, where the data generation model is used to generate pseudo images and the feature extraction model is used to extract image features;

[0057] A second processing module, configured to train the first updated feature extraction model in the (N + 1)-th time and the classification model after the N-th fusion according to the pseudo images and real images generated by the (N + 1)-th data generation model, to obtain the (N + 1)-th classification model and the second updated feature extraction model in the (N + 1)-th time, where the classification model is used to determine a corresponding result according to the image features, and the second updated feature extraction model in the (N + 1)-th time is used as the (N + 1)-th feature extraction model;

[0058] A third processing module, which uses the image features extracted by the (N + 1)-th feature extraction model as real data, and the image features generated by the feature generation model after the N-th fusion as pseudo data, and trains the feature generation model after the N-th fusion through an adversarial network, so that the output result of the feature generation model approximates the result output by the feature extraction model.

[0059] The present invention also provides an information processing device, which is applied to a server among multiple participating parties participating in federated learning. The device includes:

[0060] An acquisition module, which is used to acquire the (N + 1)-th classification model and the (N + 1)-th feature generation model sent by each data provider among the multiple participating parties; wherein, each data provider is used to train a feature generation model and a classification model, and after sequentially inputting the image to be predicted into the feature generation model and the classification model, obtain the corresponding classification result; the feature generation model is used to generate image features, and the classification model is used to output the corresponding classification result;

[0061] A first processing module, which is used to fuse each (N + 1)-th classification model to obtain a (N + 1)-th fused classification model, and send the (N + 1)-th fused classification model to each data provider respectively, so that each data provider conducts the (N + 2)-th classification model training;

[0062] A second processing module, which is used to fuse each (N + 1)-th feature generation model to obtain a (N + 1)-th fused feature generation model, and send the (N + 1)-th fused feature generation model to each data provider respectively, so that each data provider conducts the (N + 2)-th feature generation model training;

[0063] Wherein, the (N + 1)-th feature generation model is obtained by each data provider using the image features extracted by the (N + 1)-th feature extraction model as real data, and the image features generated by the feature generation model after the N-th fusion as pseudo data, and training the feature generation model after the N-th fusion through an adversarial network; the (N + 1)-th classification model and the (N + 1)-th feature extraction model are obtained by each data provider training the (N + 1)-th first-updated feature extraction model and the classification model after the N-th fusion according to the pseudo images and real images generated by the (N + 1)-th data generation model; the (N + 1)-th data generation model and the (N + 1)-th first-updated feature extraction model are both obtained by each data provider training the N-th data generation model and the N-th feature extraction model according to the feature generation model after the N-th fusion and the classification model after the N-th fusion. The data generation model is used to generate pseudo images, and the feature extraction model is used to extract image features.

[0064] The present invention also provides an information processing device, which includes: a memory, a processor, and an information processing program stored on the memory and executable on the processor. When the information processing program is executed by the processor, it implements the steps of the information processing method described in any one of the foregoing items.

[0065] The present invention also provides a computer-readable storage medium, on which an information processing program is stored. When the information processing program is executed by a processor, it implements the steps of the information processing method described in any one of the foregoing items.

[0066] The present invention also provides a computer program product, including a computer program, which implements the method described in any one of the foregoing items when executed by a processor.

[0067] In the present invention, the data provider among multiple participants participating in federated learning can perform information processing, achieve local model training through federated learning, and be used to train a feature generation model and a classification model. After sequentially inputting the image to be predicted into the feature generation model and the classification model, the corresponding classification result is obtained. Among them, the feature generation model is used to generate images, and the classification model is used to determine the corresponding result according to the image features. First, obtain the Nth fused classification model and the Nth fused feature generation model sent by the server, and then train the Nth data generation model and the Nth feature extraction model according to the Nth fused feature generation model and the Nth fused classification model to obtain the (N + 1)th data generation model and the first updated feature extraction model in the (N + 1)th time. Here, the data generation model is used to generate pseudo-images, and the feature extraction model is used to extract image features. Then, according to the pseudo-images generated by the (N + 1)th data generation model and the real images, train the first updated feature extraction model in the (N + 1)th time and the Nth fused classification model to obtain the (N + 1)th classification model and the second updated feature extraction model in the (N + 1)th time. The second updated feature extraction model in the (N + 1)th time is used as the feature extraction model in the (N + 1)th time. Use the image features extracted by the feature extraction model in the (N + 1)th time as real data, and use the image features generated by the Nth fused feature generation model as pseudo data. Through the adversarial network, train the Nth fused feature generation model to make the output result of the feature generation model approximate the result output by the feature extraction model. Therefore, in order to ensure the accuracy of the federated learning model, the coordinator (such as the server) needs to aggregate the feature extraction model and the classification model and then distribute them to each data provider. However, it is easy to recover the original image features and then the original image data from the feature extraction model. Therefore, the feature generation model is used to replace the feature extraction model to protect user privacy. At the same time, in order to ensure the reliability of training (such as model performance or accuracy), adversarial network training is used to make the feature generation model gradually generate image features similar to the real image features (here the real features can be the image features output by the feature extraction model), so that the result of the classification model predicting this image feature is more accurate. Therefore, it is possible to improve the performance of the local model on the premise of effectively protecting privacy and make the output result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present invention;

[0069] Figure 2 It is a schematic flowchart of an information processing method provided by an embodiment of the present invention;

[0070] Figure 3 It is a schematic flowchart of model training provided by an embodiment of the present invention;

[0071] Figure 4 Schematic diagram of the process of model training provided by another embodiment of the present invention;

[0072] Figure 5 Schematic diagram of the process of model training provided by still another embodiment of the present invention;

[0073] Figure 6 Schematic diagram of the process of model training provided by yet another embodiment of the present invention;

[0074] Figure 7 Schematic diagram of the process of model training provided by another embodiment of the present invention;

[0075] Figure 8 Schematic diagram of the process of an information processing method provided by still another embodiment of the present invention;

[0076] Figure 9 Schematic diagram of the structure of an information processing device provided by an embodiment of the present invention;

[0077] Figure 10 Schematic diagram of the structure of an information processing device provided by another embodiment of the present invention;

[0078] Figure 11 Schematic diagram of the structure of an information processing device provided by an embodiment of the present invention.

[0079] The realization, functional features and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0080] Hereinafter, exemplary embodiments of the present invention will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present invention can be more thoroughly understood and the present invention can be completely conveyed to those skilled in the art.

[0081] Currently, in the process of local general knowledge fusion of a client (such as a data provider), since the training data (including features and labels) corresponding to each client belongs to non-independent and identically distributed data, therefore, in the premise of protecting privacy, the above method has the problem of non-independent and identically distributed data, so that each client can only be applicable to the local distributed data and cannot cover more data, thereby resulting in a low performance of the local model of the client and a low accuracy of the output result.

[0082] To solve this problem, the technical concept of the present invention is to use a conditional adversarial generation network to achieve high-level privacy protection and resist data recovery attacks. When performing general knowledge fusion locally on each client, by training a local data generation model, it can generate data with a distribution similar to that of all data providers. Each data provider uses this data generation model to generate data to train a classification model, thereby achieving the purpose of solving the problem of non-independent and identically distributed data (including features and labels). At the same time, to ensure the reliability of training (such as model performance or accuracy), adversarial network training is adopted, enabling the feature generation model to gradually generate image features similar to real image features (where the real features can be the image features output by the feature extraction model), and further making the prediction results of the classification model for these image features more accurate. And based on the pseudo-images output by the data generation model, the feature generation model is trained through the adversarial network. By using the feature generation model instead of the feature extraction model, when the server aggregates the feature generation model, it is impossible to reverse-infer the real image data from the feature generation model, thereby protecting user privacy at a relatively high level.

[0083] The following will describe in detail some embodiments of the present invention with reference to the accompanying drawings. Without conflict between the embodiments, the embodiments and the features in the embodiments can be combined with each other.

[0084] It should be noted that "a plurality of" in this application is used to represent two or more, and the specific number is not specifically limited here. Exemplarily, a plurality of participating parties can be two participating parties or more than two participating parties. Here, the plurality of participating parties include a coordinating party and data providers, and the data providers can be at least two.

[0085] The following terms appearing in the embodiments are explained:

[0086] Data generation model: Generates data from random noise;

[0087] Feature extraction model: Extracts features from raw data;

[0088] Classification model: Classifies features based on the input features;

[0089] Feature generation model: Generates features from random noise;

[0090] Feature discrimination model: Based on the input features, discriminates whether the features come from the feature extraction model or the feature generation model.

[0091] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present invention. As Figure 1As shown, the data providers participating in federated learning are regarded as the participants in information processing. Here, the data providers can be clients, and the coordinator can be a server. In the following, clients and servers are taken as examples. Each client can include a data generation model, a feature extraction model, a classification model, a feature generation model, and a feature discrimination model. Among them, the classification model and the feature generation model are globally trained by the server after local training, such as model aggregation.

[0092] Specifically, the data generation model generates data locally on the client side, which is used to generate data not covered by the client. The feature extraction model extracts features from the data, and these features are more conducive to training the task model (i.e., the classification model). The feature generation model generates features from random data. This feature generation model will be sent to the server for aggregating the common knowledge of all clients. Then, the aggregated feature generation model is distributed by the server to each client to help the client locally integrate the knowledge it does not have and improve the performance of the client's local task model. At the same time, the feature generation model is trained by random noise (or random data) and will not leak the information of the client data on the server. Feature discrimination model: used to train the feature generation model so that it can generate features from random noise data; make the generated features continuously approximate the features extracted by the feature extraction model, so that the feature discrimination model cannot distinguish the true or false of the results output by the feature generation model. Classification model: This is the task model used to implement specific machine learning tasks; for example, classification, regression; the following takes classification as an example for detailed description. During the local training process, the data generation model is trained with random noise data so that it can generate data similar to the data distributions of all clients. Each client uses the data generated by this data generation model to train the feature extraction model and the classification model, so as to achieve the purpose of solving the problem of non-independent and identically distributed data (including features and labels). Moreover, the data owned by each client will not leave its owner. The feature extraction model is retained locally and replaced by the feature generation model to perform federated learning. Then, based on the feature extraction model, using conditional adversarial generative networks, the training of the feature generation model is realized, so that the performance of the feature generation model is approximated to the performance of the feature extraction model. Since the feature generation model generates features from random noise, the server cannot reverse-engineer the original data from the feature generation model or the classification model, that is, a high level of privacy protection is achieved to resist data recovery attacks.

[0093] Among them, the random noise is random numbers following a Gaussian distribution. Through the data generation model, pseudo-images can be generated. That is to say, the output of the data generation model is of the same dimension as the original data (real images). The true label corresponding to the random noise is the conditional label, which means that when the random noise is input into the data generation model, the pathological label carried is intended to enable the data generation model to generate a pseudo-image with this pathological label from the random noise. It can also be regarded as a conditional generation model.

[0094] Specifically, for the first (or the first round) of local training of each client, since the global model (i.e., the common one) is not obtained from the server, local model fusion training will not be carried out. During the first local training process, each client first trains the feature extraction model and the classification model until the model performance of the feature extraction model and the classification model reaches the pre-set requirements; optionally, fix the classification model and the feature extraction model, and train the data generation model until the model performance of the data generation model reaches the pre-set requirements; then fix the feature extraction model and the classification model, and train the feature generation model and the feature discrimination model through the conditional adversarial model (or conditional adversarial generation network) until the model performance of the feature generation model and the feature discrimination model reaches the pre-set requirements. After each client completes one round of training, upload their respective classification models and feature generation models to the server. During the second and subsequent local training processes, the server obtains the classification model and the feature generation model, aggregates the classification model and the feature generation model respectively, and then distributes the aggregated models to each client. Each client will perform local model fusion training: first, fix the classification model and the feature generation model, and train the feature extraction model and the data generation model until the model performance of the feature extraction model and the data generation model reaches the pre-set requirements; then fix the data generation model, and train the classification model and the feature extraction model until the model performance of the classification model and the feature extraction model reaches the pre-set requirements; then fix the feature extraction model and the classification model, and train the feature generation model and the feature discrimination model through the conditional adversarial model (or conditional adversarial generation network) until the model performance of the feature generation model and the feature discrimination model reaches the pre-set requirements. After each client completes one round of training, upload their respective classification models and feature generation models to the server.

[0095] Among them, the process for the server to update the global model is as follows: The server weighted-averages the classification models and feature generation models of all clients to obtain the aggregated classification model and the aggregated feature generation model. Transmit the aggregated feature generation model and the aggregated classification model to each client. Each client continues to execute the steps of the second and subsequent local training as described above until the training end condition is reached, and all model training ends.

[0096] Combined with Figure 1As shown, each client (i.e., Client 1, Client 2, …, Client N) trains its local model and sends the feature generation model and classification model after the current round of training to the server. The server aggregates the feature generation models and classification models of each client respectively to obtain the aggregated feature generation model and aggregated classification model, and then distributes them to each client. Each client receives the aggregated feature generation model and aggregated classification model and performs local model fusion training, and so on, continuously iterating until the training end condition is reached, and all model training is ended. Among them, during the training process of the feature generation model and classification model, the training of the data generation model is added. By training the data generation model, data similar to the data distributions of all data providers can be learned. Each data provider uses this data generation model to generate data to train the feature generation model and classification model, so as to achieve the purpose of solving the problem of non-independent and identically distributed data (including features and labels). At the same time, based on the features generated by the feature generation model from the generated data, the server cannot reverse-infer the original data of the current client, effectively protecting the data privacy of each client.

[0097] Exemplarily, taking each client as a medical institution in different regions, each client is respectively Medical Institution A, Medical Institution B, and Medical Institution C, and the server is a third party; Medical Institution A, Medical Institution B, and Medical Institution C each have user data of different users. Here, the user data can be medical images. In order to provide better services to users while protecting user privacy, Medical Institution A, Medical Institution B, and Medical Institution C perform federated learning through the server. Each medical institution has its own local model parameters. By performing image recognition on medical images, it is then used to predict the category to which the medical image belongs, so as to more accurately assist in judging the health status of users, etc., and then provide better suggestions or services to users. In order to optimize the local model, make the prediction more accurate, and protect privacy, etc., first each client extracts features from the medical images of its own users, extracts features through random noise (such as random image data), and based on the extracted features and corresponding labels, trains the data generation model, feature extraction model, feature generation model, classification model, feature discrimination model, etc. locally, and uploads the trained feature generation model and classification model to the server respectively. The server aggregates the feature generation model and classification model respectively, and then distributes the aggregated feature generation model and classification model to each client respectively. Each client continues to train until the training end condition is reached and the training stops.

[0098] Specifically, through the training data generation model, data similar to the data distributions of all data providers can be learned. Each data provider uses this data generation model to generate data for training the feature generation model and the classification model, thereby achieving the purpose of solving the problem of non-independent and identically distributed data (including features and labels). At the same time, based on the features generated by the feature generation model from the generated data, the coordinator cannot reverse-infer the original data of each data provider, effectively protecting the data privacy of each data provider. Therefore, the problem of non-independent and identically distributed data is solved under the premise of effectively protecting privacy, thereby improving the accuracy of the output results of the local model.

[0099] Among them, the feature generation model is updated by continuously adversarial training through the adversarial network. Since it is based on the pseudo-images output by the data generation model and the feature generation model is trained through the adversarial network, the server cannot reverse-infer the real images from the feature generation model, thus achieving a high level of privacy protection. At the same time, since the data generation model randomly generates pseudo-images, the local model as the client can generate images similar to the data distributions of all data providers, that is, cover the images that do not exist locally (in the prior art, without the data generation model, the sample data for training the local model can only be the unique data of each client, so the trained model has limitations. When there is data of other distributions for model output, it may not be able to output relatively correct results, resulting in problems such as subsequent data being unusable). Each data provider uses this data generation model to generate pseudo-images for training the classification model, thereby achieving the purpose of solving the problem of non-independent and identically distributed image data (including features and labels), enabling the federated learning model to learn the knowledge of image data of other distributions, and thus improving the performance of the federated learning model.

[0100] Therefore, by using the conditional adversarial generative network to achieve a high level of privacy protection and resist data recovery attacks, and when general knowledge fusion is performed locally on each client, by training a local data generation model to enable it to generate data similar to the data distributions of all data providers, each data provider uses this data generation model to generate data for training the classification model, thereby achieving the purpose of solving the problem of non-independent and identically distributed data (including features and labels).

[0101] Figure 2 It is a schematic flowchart of an information processing method provided by an embodiment of the present invention. The method is applied to a data provider among multiple participants participating in federated learning, and is used to train a feature generation model and a classification model. After sequentially inputting the image to be predicted into the feature generation model and the classification model, the corresponding classification result is obtained; as Figure 2 shown, the method in this embodiment may include:

[0102] According to the obtained original samples (such as original images or real images), the true class labels corresponding to the original samples (such as the classes of real images), random noise (such as random image data), and the true class labels corresponding to the random noise (such as predefined classes for random image data), the following steps are repeatedly executed until the classification model and the feature generation model meet the predetermined training requirements. The classification model and the feature generation model are used to support multiple participating parties to achieve federated learning:

[0103] S201. Obtain the Nth fused classification model and the Nth fused feature generation model sent by the server; wherein, the feature generation model is used to generate image features.

[0104] Since N is greater than or equal to 1, this federated learning process is the second and subsequent training processes. Taking the second (or second round) local training as an example.

[0105] Specifically, the server transmits the first aggregated classification model and the first aggregated feature generation model to each client. Taking any one client as an example, this client receives the first aggregated classification model and the first aggregated feature generation model.

[0106] S202. According to the Nth fused feature generation model and the Nth fused classification model, train the Nth data generation model and the Nth feature extraction model to obtain the (N + 1)th data generation model and the first updated feature extraction model in the (N + 1)th time. The data generation model is used to generate pseudo-images, and the feature extraction model is used to extract image features.

[0107] Specifically, based on the random noise and the true class labels corresponding to the random noise, combined with the first aggregated classification model and the first aggregated feature generation model, train the data generation model. This training process can be: passing the random noise z1 and the true class label y1 corresponding to the random noise z1 through the data generation model to generate data x1, and then based on the generated data x1, determining the feature f1 corresponding to the data x1; passing the random noise z2 and the true class label y (where z1 and z2 have the same true label) through the first aggregated feature generation model to output the feature g1, and then according to the feature g1 and the feature f1, passing through the first aggregated classification model to obtain the distance loss value, and then updating the data generation model according to the distance loss value to obtain the second data generation model. Since the random noise is randomly generated, a model trained with data based on non-independent distributions can be learned and trained.

[0108] S203. Train the feature extraction model updated for the first time in the (N + 1)-th iteration and the classification model after fusion in the N-th iteration based on the pseudo-image and real image generated by the data generation model in the (N + 1)-th iteration, to obtain the classification model in the (N + 1)-th iteration and the feature extraction model updated for the second time in the (N + 1)-th iteration. The classification model is used to determine the corresponding result according to the image features, and the feature extraction model updated for the second time in the (N + 1)-th iteration is used as the feature extraction model in the (N + 1)-th iteration. Then send the classification model in the (N + 1)-th iteration to the server, so that the server aggregates the classification models in each (N + 1)-th iteration.

[0109] Specifically, the process of training the classification model does not train the data generation model. First, based on the random noise z1 and the corresponding real class label y1 of the random noise z1, obtain the data generation model through the second local training and generate the data x2. Select the original data s2 with the real class label y2 from the original samples, determine the feature f2 according to the data x2 and the original data s2, then based on the feature f2, obtain the predicted class label through the classification model after the first aggregation, and then calculate the loss values between the predicted class label and the real class labels y1 and y2. Update the classification model after the first aggregation through this loss value to obtain the classification model in the second iteration (obtained through local training). Then each client sends its own classification model in the second iteration to the server, and the server aggregates the classification models in the second iteration sent by each client to obtain the classification model after the second aggregation and distribute it to each client. If the classification model after the second aggregation does not meet the predetermined training end condition, each client continues the third local training, and so on in a loop until the training ends.

[0110] S204. Use the image features extracted by the feature extraction model in the (N + 1)-th iteration as the real data, and use the image features generated by the feature generation model after fusion in the N-th iteration as the pseudo data. Through the adversarial network, train the feature generation model after fusion in the N-th iteration to make the output result of the feature generation model approximate the output result of the feature extraction model. Then send the feature generation model in the (N + 1)-th iteration to the server, so that the server aggregates the feature generation models in each (N + 1)-th iteration.

[0111] Specifically, based on the original samples, the feature f3 (here referring to real data) is determined; based on the random noise and the corresponding true label y of the random noise, through the feature generation model after the first aggregation, the feature g2 (here referring to pseudo data) is obtained. Then, based on the feature f3 and the feature g2, the loss value of the conditional adversarial generation network (or conditional adversarial generation model) is calculated, and the feature generation model after the first aggregation is updated through this loss value to obtain the feature generation model of the second time (locally trained). Each client sends its own feature generation model of the second time to the server, and the server aggregates the feature generation models of the second time sent by each client to obtain the feature generation model after the second aggregation.

[0112] Among them, the difference between the feature extraction model and the feature generation model here is that the training data is different and the network layer structures are different, only the output results are approximated. Moreover, the feature generation model is generated by training with multi-party random data. Therefore, it is necessary to use the feature generation model to achieve fusion on the server side, so that other distributed data can be learned locally, and at the same time, the original data cannot be deduced inversely.

[0113] Optionally, use the classification model to update the feature generation model once first, and then based on the conditional adversarial generation network, update the feature generation model again through the feature discrimination model: Based on the original samples, the feature f3 is determined; based on the random noise and the corresponding true label y of the random noise, through the feature generation model after the first aggregation, the feature g2 is obtained. Based on the feature g2, through the classification model of the second time, the predicted class label is obtained, and the loss value between the predicted class label and the true class label corresponding to the feature g2 (i.e., the true label y) is calculated. The feature generation model after the first aggregation is updated through this loss value to obtain the feature generation model after the first update in the second local training. Then, based on the feature f3 and the feature g2, the loss value of the conditional adversarial generation network (or conditional adversarial generation model) is calculated, and the feature generation model after the first update in the second local training is updated through this loss value to obtain the feature generation model of the second time (locally trained). Each client sends its own feature generation model of the second time to the server, and the server aggregates the feature generation models of the second time sent by each client to obtain the feature generation model after the second aggregation.

[0114] To ensure the accuracy of the federated learning model, the coordinator (such as the server) needs to aggregate the feature extraction model and the classification model and then distribute them to each data provider. However, it is easy to recover the original image features from the feature extraction model and then recover the original image data. Therefore, a feature generation model is used to replace the feature extraction model to protect user privacy. At the same time, to ensure the reliability of training (such as model performance or accuracy), adversarial network training is adopted, so that the feature generation model gradually generates image features similar to the real image features (where the real features can be the image features output by the feature extraction model), and then the classification model makes more accurate predictions on the image features. Therefore, it is possible to improve the performance of the local model while effectively protecting privacy, making the output results more accurate.

[0115] Optionally, when N = 1, the training data generation model includes a training random data generation model or a training first data generation model. When N is greater than or equal to 2, the training data generation model includes a training Nth data generation model.

[0116] Specifically, for the first local training, the data generation model can be a random model (i.e., the model parameters are randomly generated as the initial data generation model), or in order to save resources and reduce the number of iterations, in the first local training, the selected data generation model can be obtained by training the data generation model during the processes of training the classification model, the feature generation model, etc.

[0117] The following details the process of the first local training. Among them, for the first local training of each client, since no common model is obtained from the server, local model fusion training will not be performed.

[0118] Optionally, before obtaining the Nth fused classification model and the Nth fused feature generation model sent by the server, the method can also be implemented through the following steps:

[0119] Step a1: Train the initial classification model and the initial feature extraction model according to the real image to obtain the first classification model and the first feature extraction model, and send the first classification model to the server so that the server fuses the first classification models of each client to obtain the first fused classification model.

[0120] Specifically, based on the original samples (i.e., real images), features are obtained through an initial feature extraction model. Based on these features, a predicted class label is obtained through an initial classification model. Then, the loss value between the predicted class label and the true class label corresponding to the original sample is calculated. Based on this loss value, the initial classification model and the initial feature extraction model are updated to obtain the first (locally trained) classification model and the first feature extraction model. Each client sends its first classification model to the server, and the server aggregates the first classification models sent by each client to obtain the first aggregated classification model.

[0121] Step a2: Using the image features extracted by the first feature extraction model as real data and the image features generated by the initial feature generation model as pseudo data, the initial feature generation model is trained through an adversarial network to obtain the first feature generation model, and the first feature generation model is sent to the server so that the server can fuse the first feature generation models of each client to obtain the first fused feature generation model.

[0122] Specifically, based on the original sample, feature f4 (here referring to real data) is obtained through the first feature extraction model; based on random noise and the corresponding true label y of the random noise, feature g3 (here referring to pseudo data) is obtained through the initial feature generation model. Then, based on feature f4 and feature g3, the loss value of the conditional adversarial generation network (or conditional adversarial generation model) is calculated. Based on this loss value, the initial feature generation model is updated to obtain the first (locally trained) feature generation model. Each client sends its first feature generation model to the server, and the server aggregates the first feature generation models sent by each client to obtain the first aggregated feature generation model.

[0123] Optionally, the feature generation model is first updated once using the classification model, and then based on the conditional adversarial generation network, the feature generation model is updated again through the feature discrimination model: based on feature g3, the predicted class label is obtained through the first classification model, and the loss value between the predicted class label and the true class label corresponding to feature g3 (i.e., true label y) is calculated. Based on this loss value, the initial feature generation model is updated to obtain the first updated feature generation model in the first local training. Then, based on feature f4 and feature g3, the loss value of the conditional adversarial generation network (or conditional adversarial generation model) is calculated. Based on this loss value, the first updated feature generation model in the first local training is updated to obtain the first (locally trained) feature generation model. Each client sends its first feature generation model to the server, and the server aggregates the first feature generation models sent by each client to obtain the first aggregated feature generation model.

[0124] Optionally, based on the above embodiment, this embodiment elaborates in detail on how to obtain the first classification model and feature extraction model. The step of training an initial classification model and an initial feature extraction model based on the original samples and the corresponding true class labels of the original samples to obtain the first classification model and the first feature extraction model can be achieved through the following steps:

[0125] Step b1: Input the real image into the initial feature extraction model to obtain the sixth image feature;

[0126] Step b2: Input the sixth image feature into the initial classification model to obtain the third predicted class label, and calculate the third loss value between the third predicted class label and the true label corresponding to the real image;

[0127] Step b3: Calculate the gradient value of the initial feature extraction model and the gradient value of the initial classification model respectively according to the third loss value;

[0128] Step b4: Update the initial feature extraction model according to the gradient value of the initial feature extraction model to obtain the first feature extraction model, and update the initial classification model according to the gradient value of the initial classification model to obtain the first classification model;

[0129] Among them, the first feature extraction model is used to support the operation of updating the feature generation model.

[0130] Specifically, as shown in Figure 3 shown, Figure 3 is a schematic flowchart of the model training provided by the embodiment of the present invention, that is, training a feature extraction model and a classification model until the model performance of the feature extraction model and the classification model reaches the preset requirements: input the original samples into the feature extraction model to obtain features; input the features into the classification model to obtain predicted class labels. Calculate the loss value between the predicted class label and the corresponding true class label. Calculate the gradient values of the feature extraction model and the classification model through the loss value, and then update the feature extraction model and the classification model through the gradient values.

[0131] Optionally, based on the above embodiment, this embodiment elaborates in detail on how to train the data generation model during the first local training process. After obtaining the first feature extraction model, the method may further include the following steps:

[0132] Step c1: Input the random number and the preset class label corresponding to the random number into the initial data generation model to obtain the third pseudo-image;

[0133] Step c2: Input the third pseudo-image into the first feature extraction model to obtain the seventh image feature;

[0134] Step c3: Input the seventh image feature into the classification model for the first time to obtain a fourth predicted class label, and calculate a fourth loss value between the fourth predicted class label and the preset class label.

[0135] Step c4: Update the initial data generation model according to the fourth loss value to obtain the data generation model for the first time.

[0136] Specifically, as shown in Figure 4 shown, Figure 4 is a schematic flowchart of model training provided by another embodiment of the present invention, that is, fixing the classification model and the feature extraction model, and training the data generation model until the model performance of the data generation model meets the pre-set requirements: randomly generate noise (i.e., random noise or random numbers) and corresponding true labels (i.e., true class labels). Then input the random noise and the corresponding true labels into the data generation model simultaneously to obtain generated data (here it is the third pseudo-image). Input the generated data into the feature extraction model to obtain feature f (here it is the seventh image feature). Input feature f into the classification model to obtain a predicted class label (here it is the fourth predicted class label). Calculate the loss value between the predicted class label and the true class label (here it is the fourth loss value). Calculate the gradient value of the data generation model through the loss value, and then update the data generation model through the gradient value.

[0137] Optionally, based on the above embodiment, this embodiment details how to obtain the feature generation model for the first time in the first local training. Taking the image feature extracted by the feature extraction model for the first time as real data and the image feature generated by the initial feature generation model as pseudo data, through an adversarial network, train the initial feature generation model to obtain the feature generation model for the first time, which can be achieved through the following steps:

[0138] Step d11: Input the real image into the feature extraction model for the first time to obtain an eighth image feature.

[0139] Step d12: Input the random number and the preset class label into the initial feature generation model to obtain a ninth image feature.

[0140] Step d13: Input the eighth image feature and the ninth image feature into the initial feature discrimination model respectively, calculate a second target loss value of the adversarial network, and update the initial feature discrimination model and the initial feature generation model respectively according to the second target loss value to obtain the feature discrimination model for the first time and the feature generation model for the first time.

[0141] Specifically, as shown in Figure 5 shown, Figure 5The flowchart of model training provided for another embodiment of the present invention, that is, fixing the feature extraction model, and training the feature generation model and the feature discrimination model through the general training method of the conditional adversarial generation network until the model performances of the feature generation model and the feature discrimination model reach the preset requirements: input the original sample into the feature extraction model to obtain the feature f (here it is the eighth image feature). Random noise (i.e., random noise or random numbers) and corresponding real labels are generated. Then, the random noise and the corresponding real labels are input into the feature generation model simultaneously to obtain the feature g (here it is the ninth image feature). The features f and g are respectively input into the feature discrimination model. Through the feature discrimination model, based on the features f and g, the loss value of the conditional adversarial generation network (here it is the second target loss value) is calculated. Based on this loss value, the gradient values of the feature discrimination model and the feature generation model are calculated, and then the sample discrimination model and the feature generation model are updated through the gradient values.

[0142] Optionally, use the classification model to update the feature generation model once first, and then, based on the conditional adversarial generation network, update the feature generation model again through the feature discrimination model:

[0143] Step d21: Input the real image into the first feature extraction model to obtain the eighth image feature;

[0144] Step d22: Input the random number and the preset class label into the initial feature generation model to obtain the ninth image feature, and input the ninth image feature into the first classification model to obtain the fifth predicted class label;

[0145] Step d23: Calculate the fifth loss value between the fifth predicted class label and the preset class label, and update the initial feature generation model according to the fifth loss value to obtain the first updated feature generation model in the first local training;

[0146] Step d24: Input the eighth image feature and the ninth image feature into the initial feature discrimination model respectively, calculate the second target loss value of the conditional adversarial generation network, and update the initial feature discrimination model and the first updated feature generation model in the first local training according to the second target loss value of the conditional adversarial generation network to obtain the first feature discrimination model and the second updated feature generation model in the first local training; the second updated feature generation model in the first local training is used to represent the first feature generation model.

[0147] Specifically, the original samples are input into the feature extraction model to obtain the feature f (here it is the eighth image feature). Random noise (i.e., random noise or random numbers) and corresponding true labels are randomly generated. Then, the random noise and the corresponding true labels are simultaneously input into the feature generation model to obtain the feature g (here it is the ninth image feature). The feature g is input into the classification model to obtain the predicted class label (here it is the fifth predicted class label). The loss value between the predicted class label and the true class label corresponding to g is calculated (here it is the fifth loss value). Based on this loss value, the gradient value of the feature generation model is calculated, and then the feature generation model is updated through the gradient value. The features f and g are respectively input into the feature discrimination model. Through the feature discrimination model, based on the features f and g, the loss value of the conditional adversarial generation network is calculated (here it is the second objective loss value). Based on this loss value, the gradient values of the feature discrimination model and the feature generation model are calculated, and then the sample discrimination model and the feature generation model are updated through the gradient values.

[0148] The following details the process of the second and subsequent rounds of local training. Among them, for the subsequent local training of each client except for the first local training, the classification model and the feature generation model are obtained from the server, and local model fusion training will be carried out.

[0149] Optionally, based on the above embodiment, this embodiment details how to train the data generation model in the second and subsequent rounds of local training. The training of the Nth data generation model and the Nth feature extraction model according to the Nth fused feature generation model and the Nth fused classification model to obtain the (N + 1)th data generation model and the first updated feature extraction model in the (N + 1)th time can be achieved through the following steps:

[0150] Step e1: Input the first target random number in the random numbers and the first preset class label corresponding to the first target random number into the Nth data generation model to obtain the first pseudo-image, and the true label corresponding to the first pseudo-image is the first preset class label;

[0151] Step e2: Input the first pseudo-image into the Nth feature extraction model to obtain the first image feature;

[0152] Step e3: Input the second target random number in the random numbers and the second preset class label corresponding to the second target random number into the Nth fused feature generation model to obtain the second image feature, and the true label corresponding to the second image feature is the second preset class label, and the first preset class label is the same as the second preset class label;

[0153] Step e4: Based on the first image feature and the second image feature, determine the (N + 1)-th data generation model and the first updated feature extraction model in the (N + 1)-th iteration through the classification model after the N-th fusion; wherein, the first updated feature extraction model in the (N + 1)-th iteration is used to support the update of the classification model.

[0154] Specifically, as shown in Figure 6 which is Figure 6 a schematic flow chart of model training provided by another embodiment of the present invention, that is, fixing the classification model and the feature generation model (not participating in training), training the feature extraction model and the data generation model until the model performance of the feature extraction model and the data generation model reaches the preset requirements: randomly generate noise (i.e., random noise or random numbers) z1 (here is the first target random number) and the corresponding true label y1 of z1. Then input z1 and y1 into the data generation model simultaneously to obtain the generated data x1 (here is the first pseudo-image). Input the generated data x1 into the feature extraction model to obtain the feature f1 (here is the first image feature). Randomly generate noise (i.e., random noise or random numbers) z2 (here is the second target random number). z1 and z2 have the same true label y1. Then input z2 and y1 into the feature generation model simultaneously to obtain the feature g1 (here is the second image feature). Based on the feature f1 and the feature g1, update the feature extraction model and the data generation model through the classification model.

[0155] Optionally, as shown in Figure 6 which is a detailed description of how to update the feature extraction model and the data generation model based on the above embodiment. The step of determining the (N + 1)-th data generation model and the first updated feature extraction model in the (N + 1)-th iteration through the classification model after the N-th fusion based on the first image feature and the second image feature can be implemented through the following steps:

[0156] Step f1: Input the first image feature and the second image feature into the classification model after the N-th fusion to obtain the first image class probability distribution and the second image class probability distribution;

[0157] Step f2: Calculate the distance loss value between the first image class probability distribution and the second image class probability distribution;

[0158] Step f3: Calculate the gradient value of the N-th data generation model and the first gradient value of the N-th feature extraction model respectively through the distance loss value;

[0159] Step f4: Update the data generation model of the N-th time according to the gradient value of the data generation model to obtain the data generation model of the N+1-th time, and update the feature extraction model of the N-th time according to the first gradient value to obtain the feature extraction model updated for the first time in the N+1-th local training.

[0160] Specifically, input the feature f1 (here it is the first image feature) and the feature g1 (here it is the second image feature) into the classification model to obtain the first category probability distribution p1 and the second category probability distribution p2. Calculate the distance loss value between p1 and p2. Calculate the gradient values of the feature extraction model and the data generation model (here it is the first gradient value) through the distance loss value. Then update the feature extraction model and the data generation model through the gradient values.

[0161] Optionally, based on the above embodiment, this embodiment details how to train the classification model in the second and subsequent rounds of local training. Training the feature extraction model updated for the first time in the N+1-th time and the classification model fused in the N-th time with the pseudo-image and the real image generated according to the data generation model of the N+1-th time to obtain the classification model of the N+1-th time and the feature extraction model updated for the second time in the N+1-th time can be achieved through the following steps:

[0162] Step g1: Input the first target random number and the first preset category label into the data generation model of the N+1-th time to obtain a second pseudo-image;

[0163] Step g2: Obtain the first original image containing the real label from the real images, and input the first original image and the second pseudo-image into the feature extraction model updated for the first time in the N+1-th time to obtain a third image feature;

[0164] Step g3: Input the third image feature into the classification model fused in the N-th time to obtain a first predicted category label, and calculate the first loss value of the first preset category label and the real label corresponding to the first original data and the corresponding first predicted category label;

[0165] Step g4: Calculate the second gradient value of the feature extraction model updated for the first time in the N+1-th time and the gradient value of the classification model fused in the N-th time according to the first loss value, and update the feature extraction model updated for the first time in the N+1-th time according to the second gradient value to obtain the feature extraction model updated for the second time in the N+1-th time, and update the classification model fused in the N-th time according to the gradient value of the classification model fused in the N-th time to obtain the classification model of the N+1-th time.

[0166] Specifically, as shown in Figure 7 shown, Figure 7The flowchart of model training provided by another embodiment of the present invention is as follows: a fixed data generation model (not participating in training) is used, and a classification model and a feature extraction model are trained until the model performances of the classification model and the feature extraction model meet the preset requirements. Random noise (i.e., random noise or random numbers) z1 (here is the first target random number) and the corresponding true label y1 are randomly generated. Then, z1 and y1 are input into the data generation model simultaneously to obtain generated data x2 (here is the second pseudo-image). The original data s2 (here is the first original image) with label y2 is sampled. The generated data x2 and the original data s2 are input into the feature extraction model to obtain feature f2 (here is the third image feature). The feature f2 is input into the classification model to obtain a predicted class label (here is the first predicted class label). The loss value between the predicted class label and the true class label is calculated (here is the first loss value). The gradient values of the feature extraction model and the classification model are calculated through the loss value (here is the second gradient value), and then the feature extraction model and the classification model are updated through the gradient values.

[0167] Optionally, based on the above embodiment, this embodiment details how to train the feature generation model in the second and subsequent rounds of local training.

[0168] Taking the image features extracted by the feature extraction model of the (N + 1)-th time as real data, and taking the image features generated by the feature generation model after the N-th fusion as pseudo data, the feature generation model after the N-th fusion can be trained through an adversarial network, which can be achieved through the following steps:

[0169] Step h11: Input the real image into the feature extraction model of the (N + 1)-th time to obtain the fourth image feature;

[0170] Step h12: Input the random number and the corresponding preset class label of the random number into the feature generation model after the N-th fusion to obtain the fifth image feature;

[0171] Step h13: According to the fourth image feature and the fifth image feature, through the feature discrimination model of the N-th time, calculate the first target loss value of the adversarial network, and update the feature discrimination model of the N-th time and the feature generation model after the N-th fusion respectively according to the first target loss value to obtain the feature discrimination model of the (N + 1)-th time and the feature generation model of the (N + 1)-th time.

[0172] Among them, the feature discrimination model is used to distinguish whether the image feature is real data or pseudo data.

[0173] Specifically, this training process is the same as Figure 5The training process shown is similar, that is, fix the feature extraction model, and train the feature generation model and the feature discrimination model through the general training method of the conditional adversarial generation network until the model performance of the feature generation model and the feature discrimination model reaches the preset requirements: train the feature generation model and the feature discrimination model through the general training method of the conditional adversarial generation network until the model performance of the feature generation model and the feature discrimination model reaches the preset requirements: input the original sample into the feature extraction model to obtain the feature f (here it is the fourth image feature). Randomly generate noise (i.e., random noise or random numbers) and the corresponding true label y. Then input the random noise and the corresponding true label into the feature generation model simultaneously to obtain the feature g (here it is the fifth image feature). Input the features f and g into the feature discrimination model respectively. Through the feature discrimination model, based on the features f and g, calculate the loss value of the conditional adversarial generation network (here it is the first target loss value). Calculate the gradient values of the feature discrimination model and the feature generation model based on this loss value, and then update the sample discrimination model and the feature generation model through the gradient values.

[0174] Optionally, use the classification model to update the feature generation model once first, and then based on the conditional adversarial generation network, update the feature generation model again through the feature discrimination model:

[0175] Step h21: Input the real image into the feature extraction model for the (N + 1)-th time to obtain the fourth image feature;

[0176] Step h22: Input the random number and the preset class label corresponding to the random number into the feature generation model aggregated for the N-th time to obtain the fifth image feature;

[0177] Step h23: Input the fifth image feature into the classification model for the (N + 1)-th time to obtain the second predicted class label, and update the feature generation model aggregated for the N-th time according to the second predicted class label and the true class label corresponding to the random noise (i.e., the true class label corresponding to the fifth image feature) to obtain the feature generation model updated for the first time in the (N + 1)-th local training;

[0178] Step h24: Calculate the first target loss value of the conditional adversarial generation network (i.e., the adversarial network) according to the fourth image feature and the fifth image feature through the feature discrimination model for the N-th time, and update the feature discrimination model for the N-th time and the feature generation model updated for the second time in the (N + 1)-th local training respectively according to the first target loss value of the conditional adversarial generation network to obtain the feature discrimination model for the (N + 1)-th time and the feature generation model updated for the second time in the (N + 1)-th local training; the feature generation model updated for the second time in the (N + 1)-th local training is used to represent the feature generation model for the (N + 1)-th time.

[0179] Specifically, the original sample is input into the feature extraction model to obtain the feature f (here it is the fourth image feature). Random noise (i.e., random noise or random numbers) and the corresponding true label y are randomly generated. Then, the random noise and the corresponding true label are input into the feature generation model simultaneously to obtain the feature g (here it is the fifth image feature). The feature g is input into the classification model to obtain the predicted class label (here it is the second predicted class label). The loss value between the predicted class label and the true class label corresponding to g is calculated. Based on this loss value, the gradient value of the feature generation model is calculated, and then the feature generation model is updated through the gradient value. The features f and g are respectively input into the feature discrimination model. Through the feature discrimination model, based on the features f and g, the loss value of the conditional adversarial generation network (here it is the first objective loss value) is calculated. Based on this loss value, the gradient values of the feature discrimination model and the feature generation model are calculated, and then the sample discrimination model and the feature generation model are updated through the gradient values.

[0180] After each client completes one round of training, their respective classification models and feature generation models are uploaded to the server.

[0181] The present invention can solve the problem of non-independent and identically distributed data among various clients, including the problem of non-independent and identically distributed features and the problem of non-independent and identically distributed labels. At the same time, the global models of each client are not directly trained from the original data of each client, which greatly reduces the possibility that the server can reverse-engineer the original data of the client through the global model, that is, effectively protecting the data privacy of the client.

[0182] Figure 8 It is a schematic flowchart of an information processing method provided for another embodiment of the present invention. The method is applied to a server among multiple participants participating in federated learning. As Figure 8 shown, the information processing method may include:

[0183] S801. Obtain the (N + 1)-th classification model and the (N + 1)-th feature generation model sent by each data provider among multiple participants; wherein, each data provider is used to train the feature generation model and the classification model, and after the image to be predicted is input into the feature generation model and the classification model in sequence, the corresponding classification result is obtained; the feature generation model is used to generate image features, and the classification model is used to output the corresponding classification result;

[0184] S802. Fuse each (N + 1)-th classification model to obtain the (N + 1)-th fused classification model, and send the (N + 1)-th fused classification model to each data provider respectively, so that each data provider conducts the (N + 2)-th training of the classification model;

[0185] Fuse the feature generation models for each (N + 1)-th time to obtain the feature generation model after the (N + 1)-th fusion, and send the feature generation model after the (N + 1)-th fusion to each data provider respectively, so that each data provider can perform the training of the feature generation model for the (N + 2)-th time.

[0186] Among them, the feature generation model for the (N + 1)-th time is obtained by training the feature generation model after the N-th fusion through an adversarial network, where the image features extracted by each data provider using the feature extraction model for the (N + 1)-th time are used as real data, and the image features generated by the feature generation model after the N-th fusion are used as pseudo data; the classification model and the feature extraction model for the (N + 1)-th time are obtained by training the first updated feature extraction model and the classification model after the N-th fusion using the pseudo images and real images generated by each data provider according to the data generation model for the (N + 1)-th time; the data generation model for the (N + 1)-th time and the first updated feature extraction model for the (N + 1)-th time are both obtained by each data provider training the data generation model and the feature extraction model for the N-th time according to the feature generation model after the N-th fusion and the classification model after the N-th fusion, where the data generation model is used to generate pseudo images and the feature extraction model is used to extract image features.

[0187] Specifically, the server weighted-averages the classification models and feature generation models of all clients to obtain the aggregated classification model and feature generation model. Transmit the aggregated feature generation model and classification model to each client. Continue the above local training process until the training end condition is reached, and end the training of all models.

[0188] Through the information processing method provided by the embodiments of the present invention, the data generation model can be trained by the client based on random noise and the real labels corresponding to the random noise to generate input data, and then combined with the feature generation model to generate features for training the classification model, etc. Then, the trained classification model is sent to the server, and the server performs global model update and distributes it to each client, and this process is iterated until the training ends. In this way, by training the data generation model, data with a data distribution similar to that of all data providers can be learned. Each data provider uses this data generation model to generate data to train the feature generation model and the classification model, so as to achieve the purpose of solving the problem of non-independent and identically distributed data (including features and labels). At the same time, based on the features generated by the feature generation model using the generated data, the server cannot reverse-infer the original data of each data provider, effectively protecting the data privacy of each data provider. Therefore, it is possible to improve the performance of the local model while effectively protecting privacy, making the output result more accurate.

[0189] Figure 9Schematic structural diagram of an information processing device provided by an embodiment of the present invention. The device is applied to a data provider among multiple participants participating in federated learning, and is used to train a local feature generation model and a common classification model. After sequentially inputting an image to be predicted into the feature generation model and the classification model, a corresponding classification result is obtained.

[0190] As Figure 9 shown, the information processing device may include:

[0191] An acquisition module 901, configured to acquire the Nth fused classification model and the Nth fused feature generation model sent by the server; wherein, the feature generation model is used to generate image features;

[0192] A first processing module 902, configured to train an Nth data generation model and an Nth feature extraction model according to the Nth fused feature generation model and the Nth fused classification model, to obtain an (N + 1)th data generation model and a first updated feature extraction model in the (N + 1)th time. The data generation model is used to generate pseudo-images, and the feature extraction model is used to extract image features;

[0193] A second processing module 903, configured to train the first updated feature extraction model in the (N + 1)th time and the Nth fused classification model according to the pseudo-images generated by the (N + 1)th data generation model and real images, to obtain an (N + 1)th classification model and a second updated feature extraction model in the (N + 1)th time. The classification model is used to determine a corresponding result according to the image features, and the second updated feature extraction model in the (N + 1)th time is used as the feature extraction model in the (N + 1)th time;

[0194] A third processing module 904, configured to use the image features extracted by the (N + 1)th feature extraction model as real data, and use the image features generated by the Nth fused feature generation model as pseudo data, and train the Nth fused feature generation model through an adversarial network, so that the output result of the feature generation model approximates the result output by the feature extraction model.

[0195] Optionally, the first processing module 902 includes a first processing unit, a second processing unit, a third processing unit, and a fourth processing unit; the first processing unit is configured to input a first target random number in the random numbers and a first preset class label corresponding to the first target random number into the data generation model of the Nth time, to obtain a first pseudo-image, and the true label corresponding to the first pseudo-image is the first preset class label; the second processing unit is configured to input the first pseudo-image into the feature extraction model of the Nth time, to obtain a first image feature; the third processing unit is configured to input a second target random number in the random numbers and a second preset class label corresponding to the second target random number into the feature generation model after fusion of the Nth time, to obtain a second image feature, and the true label corresponding to the second image feature is the second preset class label, and the first preset class label is the same as the second preset class label; the fourth processing unit is configured to determine the data generation model of the (N + 1)th time and the feature extraction model updated for the first time in the (N + 1)th time through the classification model after fusion of the Nth time according to the first image feature and the second image feature; wherein, the feature extraction model updated for the first time in the (N + 1)th time is used to support the update of the classification model.

[0196] Optionally, the fourth processing unit is specifically configured to:

[0197] Input the first image feature and the second image feature into the classification model after fusion of the Nth time, to obtain a first image class probability distribution and a second image class probability distribution;

[0198] Calculate a distance loss value between the first image class probability distribution and the second image class probability distribution;

[0199] Calculate a gradient value of the data generation model of the Nth time and a first gradient value of the feature extraction model of the Nth time respectively through the distance loss value;

[0200] Update the data generation model of the Nth time according to the gradient value of the data generation model of the Nth time, to obtain the data generation model of the (N + 1)th time, and update the feature extraction model of the Nth time according to the first gradient value, to obtain the feature extraction model updated for the first time in the local training of the (N + 1)th time.

[0201] Optionally, the second processing module is specifically configured to:

[0202] Input the first target random number and the first preset class label into the data generation model of the (N + 1)th time, to obtain a second pseudo-image;

[0203] Obtain a first original image containing real labels from the real image, and input the first original image and the second pseudo-image into the feature extraction model updated for the first time in the (N + 1)-th time to obtain a third image feature;

[0204] Input the third image feature into the classification model after fusion in the N-th time to obtain a first predicted class label, and calculate a first loss value of the first preset class label and the real label corresponding to the first original data and the corresponding first predicted class label;

[0205] According to the first loss value, calculate a second gradient value of the feature extraction model updated for the first time in the (N + 1)-th time and a gradient value of the classification model after fusion in the N-th time respectively, and update the feature extraction model updated for the first time in the (N + 1)-th time according to the second gradient value to obtain the feature extraction model updated for the second time in the (N + 1)-th time, and update the classification model after fusion in the N-th time according to the gradient value of the classification model after fusion in the N-th time to obtain the classification model in the (N + 1)-th time.

[0206] Optionally, the third processing module is specifically configured to:

[0207] Input the real image into the feature extraction model in the (N + 1)-th time to obtain a fourth image feature;

[0208] Input the random number and the preset class label corresponding to the random number into the feature generation model after fusion in the N-th time to obtain a fifth image feature;

[0209] According to the fourth image feature and the fifth image feature, calculate a first target loss value of the adversarial network through the feature discrimination model in the N-th time, and update the feature discrimination model in the N-th time and the feature generation model after fusion in the N-th time respectively according to the first target loss value to obtain the feature discrimination model in the (N + 1)-th time and the feature generation model in the (N + 1)-th time;

[0210] Wherein, the feature discrimination model is used to distinguish whether the image feature is real data or pseudo data.

[0211] Optionally, the device may further include: a fourth processing module; the fourth processing module includes a fifth processing unit and a sixth processing unit;

[0212] The fifth processing unit is configured to train an initial classification model and an initial feature extraction model according to the real image to obtain a classification model for the first time and a feature extraction model for the first time, and send the classification model for the first time to the server so that the server fuses the classification models for the first time to obtain a classification model after the first fusion;

[0213] The sixth processing unit is configured to use the image features extracted by the first feature extraction model as real data, use the image features generated by the initial feature generation model as pseudo data, and train the initial feature generation model through an adversarial network to obtain the first feature generation model, and send the first feature generation model to the server, so that the server fuses the first feature generation models to obtain the first fused feature generation model.

[0214] Optionally, the fifth processing unit is specifically configured to:

[0215] Input the real image into the initial feature extraction model to obtain the sixth image features;

[0216] Input the sixth image features into the initial classification model to obtain the third predicted class label, and calculate the third loss value between the third predicted class label and the real label corresponding to the real image;

[0217] According to the third loss value, calculate the gradient value of the initial feature extraction model and the gradient value of the initial classification model respectively;

[0218] Update the initial feature extraction model according to the gradient value of the initial feature extraction model to obtain the first feature extraction model, and update the initial classification model according to the gradient value of the initial classification model to obtain the first classification model;

[0219] Wherein, the first feature extraction model is used to support the operation of updating the feature generation model.

[0220] Optionally, the device further includes: a sixth processing module; the sixth processing module is configured to:

[0221] Input the random number and the preset class label corresponding to the random number into the initial data generation model to obtain the third pseudo image;

[0222] Input the third pseudo image into the first feature extraction model to obtain the seventh image features;

[0223] Input the seventh image features into the first classification model to obtain the fourth predicted class label, and calculate the fourth loss value between the fourth predicted class label and the preset class label;

[0224] Update the initial data generation model according to the fourth loss value to obtain the first data generation model.

[0225] Optionally, the sixth processing unit is specifically configured to:

[0226] Input the real image into the first feature extraction model to obtain the eighth image features;

[0227] Input the random number and the preset class label into the initial feature generation model to obtain the ninth image feature;

[0228] Input the eighth image feature and the ninth image feature into the initial feature discrimination model respectively, calculate the second objective loss value of the adversarial network, and update the initial feature discrimination model and the initial feature generation model respectively according to the second objective loss value to obtain the first feature discrimination model and the first feature generation model.

[0229] The information processing device provided in any of the foregoing embodiments is used to execute the technical solutions of any method embodiment described above with the data provider as the participating party of information processing. The implementation principles and technical effects are similar and will not be elaborated here.

[0230] Figure 10 It is a schematic structural diagram of an information processing device provided in another embodiment of the present invention. The device is applied to a server among multiple participating parties in federated learning.

[0231] As Figure 10 shown, the information processing device may include:

[0232] An acquisition module 1001, configured to acquire the (N + 1)-th classification model and the (N + 1)-th feature generation model sent by each data provider among multiple participating parties; wherein, each data provider is used to train a feature generation model and a classification model, and after sequentially inputting the image to be predicted into the feature generation model and the classification model, obtain the corresponding classification result; the feature generation model is used to generate image features, and the classification model is used to output the corresponding classification result;

[0233] A first processing module 1002, configured to fuse each (N + 1)-th classification model to obtain the (N + 1)-th fused classification model, and send the (N + 1)-th fused classification model to each data provider respectively, so that each data provider performs the (N + 2)-th classification model training;

[0234] A second processing module 1003, configured to fuse each (N + 1)-th feature generation model to obtain the (N + 1)-th fused feature generation model, and send the (N + 1)-th fused feature generation model to each data provider respectively, so that each data provider performs the (N + 2)-th feature generation model training;

[0235] Among them, the (N + 1)-th feature generation model is obtained by training the N-th fused feature generation model through an adversarial network, where each data provider uses the image features extracted by the (N + 1)-th feature extraction model as real data and the image features generated by the N-th fused feature generation model as fake data; the (N + 1)-th classification model and the (N + 1)-th feature extraction model are obtained by training the first updated feature extraction model in the (N + 1)-th and the N-th fused classification model with the fake images and real images generated by each data provider according to the (N + 1)-th data generation model; the (N + 1)-th data generation model and the first updated feature extraction model in the (N + 1)-th are both obtained by each data provider training the N-th data generation model and the N-th feature extraction model according to the N-th fused feature generation model and the N-th fused classification model, where the data generation model is used to generate fake images and the feature extraction model is used to extract image features.

[0236] The information processing device provided in any of the foregoing embodiments is used to execute the technical solutions of any method embodiment described with the coordinator as the participating party in information processing. Its implementation principle and technical effects are similar and will not be elaborated here.

[0237] Figure 11 This is a schematic structural diagram of an information processing device provided by an embodiment of the present invention. As Figure 11 shown, the device may include: a memory 1101, a processor 1102, and an information processing program stored on the memory 1101 and executable on the processor 1102. When the information processing program is executed by the processor 1102, it implements the steps of the information processing method described in any of the foregoing embodiments.

[0238] Optionally, the memory 1101 can be either independent or integrated with the processor 1102.

[0239] The implementation principle and technical effects of the device provided in this embodiment can be referred to the foregoing embodiments and will not be elaborated here.

[0240] An embodiment of the present invention further provides a computer-readable storage medium, on which an information processing program is stored. When the information processing program is executed by a processor, it implements the steps of the information processing method described in any of the foregoing embodiments.

[0241] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the method described in any of the foregoing embodiments when executed by a processor.

[0242] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0243] The integrated modules implemented in the form of software function modules can be stored in a computer-readable storage medium. The above software function modules are stored in a storage medium and include several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the methods described in various embodiments of the present invention.

[0244] It should be understood that the above processor may be a central processing unit (CPU for short), or may also be other general-purpose processors, digital signal processors (DSP for short), application specific integrated circuits (ASIC for short), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly implemented by the execution of the hardware processor, or can be implemented by the combination of the hardware and software modules in the processor.

[0245] The memory may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disc, etc.

[0246] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disc. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0247] An exemplary storage medium is coupled to a processor, enabling the processor to read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a master device.

[0248] It should be noted that in this document, the terms "comprising", "including" or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising such element.

[0249] The serial numbers of the above-described embodiments of the present invention are for description only and do not represent the superiority or inferiority of the embodiments.

[0250] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such understanding, the technical solution of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0251] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. An information processing method, characterized in that, The method is applied to the data provider among multiple participants participating in federated learning, and is used to train a feature generation model and a classification model. After sequentially inputting the image to be predicted into the feature generation model and the classification model, the corresponding classification result is obtained. The method includes: Obtain the Nth fused classification model and the Nth fused feature generation model sent by the server; wherein, the feature generation model is used to generate image features; According to the Nth fused feature generation model and the Nth fused classification model, train the Nth data generation model and the Nth feature extraction model to obtain the (N + 1)th data generation model and the first updated feature extraction model in the (N + 1)th time. The data generation model is used to generate pseudo-images, and the feature extraction model is used to extract image features; According to the pseudo-images generated by the (N + 1)th data generation model and the real images, train the first updated feature extraction model in the (N + 1)th time and the Nth fused classification model to obtain the (N + 1)th classification model and the second updated feature extraction model in the (N + 1)th time. The classification model is used to determine the corresponding result according to the image features, and the second updated feature extraction model in the (N + 1)th time is used as the feature extraction model in the (N + 1)th time; Use the image features extracted by the (N + 1)th feature extraction model as real data, and use the image features generated by the Nth fused feature generation model as pseudo data. Through the adversarial network, train the Nth fused feature generation model to make the output result of the feature generation model approximate the result output by the feature extraction model.

2. The method according to claim 1, wherein The training of the Nth data generation model and the Nth feature extraction model according to the Nth fused feature generation model and the Nth fused classification model to obtain the (N + 1)th data generation model and the first updated feature extraction model in the (N + 1)th time includes: Input the first target random number in the random numbers and the first preset class label corresponding to the first target random number into the Nth data generation model to obtain a first pseudo-image, and the real label corresponding to the first pseudo-image is the first preset class label; Input the first pseudo-image into the Nth feature extraction model to obtain a first image feature; Input the second target random number in the random numbers and the second preset class label corresponding to the second target random number into the Nth fused feature generation model to obtain a second image feature, and the real label corresponding to the second image feature is the second preset class label. The first preset class label is the same as the second preset class label; According to the first image feature and the second image feature, determine the (N + 1)th data generation model and the first updated feature extraction model in the (N + 1)th time through the Nth fused classification model; wherein, the first updated feature extraction model in the (N + 1)th time is used to support the update of the classification model.

3. The method according to claim 2, wherein Determining the (N + 1)-th data generation model and the first updated feature extraction model in the (N + 1)-th time according to the first image feature and the second image feature through the classification model after the N-th fusion, includes: Inputting the first image feature and the second image feature into the classification model after the N-th fusion to obtain the first image class probability distribution and the second image class probability distribution; Calculating the distance loss value between the first image class probability distribution and the second image class probability distribution; Respectively calculating the gradient value of the N-th data generation model and the first gradient value of the N-th feature extraction model through the distance loss value; Updating the N-th data generation model according to the gradient value of the N-th data generation model to obtain the (N + 1)-th data generation model, and updating the N-th feature extraction model according to the first gradient value to obtain the first updated feature extraction model in the (N + 1)-th local training.

4. The method according to claim 2 or 3, characterized in that, Training the first updated feature extraction model in the (N + 1)-th time and the classification model after the N-th fusion according to the pseudo-image generated by the (N + 1)-th data generation model and the real image to obtain the (N + 1)-th classification model and the second updated feature extraction model in the (N + 1)-th time, includes: Inputting the first target random number and the first preset class label into the (N + 1)-th data generation model to obtain a second pseudo-image; Obtaining a first original image containing a real label from the real images, and inputting the first original image and the second pseudo-image into the first updated feature extraction model in the (N + 1)-th time to obtain a third image feature; Inputting the third image feature into the classification model after the N-th fusion to obtain a first predicted class label, and calculating the first loss value between the first preset class label and the real label corresponding to the first original data and the corresponding first predicted class label; Respectively calculating the second gradient value of the first updated feature extraction model in the (N + 1)-th time and the gradient value of the classification model after the N-th fusion according to the first loss value, and updating the first updated feature extraction model in the (N + 1)-th time according to the second gradient value to obtain the second updated feature extraction model in the (N + 1)-th time, and updating the classification model after the N-th fusion according to the gradient value of the classification model after the N-th fusion to obtain the (N + 1)-th classification model.

5. The method according to claim 4, characterized in that, Using the image features extracted by the (N + 1)-th feature extraction model as real data and the image features generated by the feature generation model after the N-th fusion as pseudo data to train the feature generation model after the N-th fusion through an adversarial network, includes: Inputting the real image into the (N + 1)-th feature extraction model to obtain a fourth image feature; Inputting the random number and the preset class label corresponding to the random number into the feature generation model after the N-th fusion to obtain a fifth image feature; According to the fourth image feature and the fifth image feature, calculate the first target loss value of the adversarial network through the Nth feature discrimination model, and respectively update the Nth feature discrimination model and the Nth fused feature generation model according to the first target loss value to obtain the (N + 1)th feature discrimination model and the (N + 1)th feature generation model; Among them, the feature discrimination model is used to distinguish whether the image feature is real data or pseudo data.

6. The method according to claim 2 or 3, characterized in that, Before obtaining the Nth fused classification model and the Nth fused feature generation model sent by the server, the method further includes: According to the real image, train the initial classification model and the initial feature extraction model to obtain the first classification model and the first feature extraction model, and send the first classification model to the server so that the server fuses each of the first classification models to obtain the first fused classification model; Using the image features extracted by the first feature extraction model as real data and the image features generated by the initial feature generation model as pseudo data, train the initial feature generation model through the adversarial network to obtain the first feature generation model, and send the first feature generation model to the server so that the server fuses each of the first feature generation models to obtain the first fused feature generation model.

7. The method according to claim 6, wherein The training of the initial classification model and the initial feature extraction model according to the real image to obtain the first classification model and the first feature extraction model includes: Input the real image into the initial feature extraction model to obtain the sixth image feature; Input the sixth image feature into the initial classification model to obtain the third predicted class label, and calculate the third loss value between the third predicted class label and the real label corresponding to the real image; According to the third loss value, calculate the gradient values of the initial feature extraction model and the initial classification model respectively; Update the initial feature extraction model according to the gradient value of the initial feature extraction model to obtain the first feature extraction model, and update the initial classification model according to the gradient value of the initial classification model to obtain the first classification model; Among them, the first feature extraction model is used to support the operation of updating the feature generation model.

8. The method according to claim 7, wherein After obtaining the first classification model and the first feature extraction model, the method further includes: Input the random number and the preset class label corresponding to the random number into the initial data generation model to obtain the third pseudo image; Input the third pseudo image into the first feature extraction model to obtain the seventh image feature; Input the seventh image feature into the first classification model to obtain the fourth predicted class label, and calculate the fourth loss value between the fourth predicted class label and the preset class label; Update the initial data generation model according to the fourth loss value to obtain the first data generation model.

9. The method according to claim 7, wherein Using the image features extracted by the first feature extraction model as real data and the image features generated by the initial feature generation model as pseudo data, training the initial feature generation model through an adversarial network to obtain the first feature generation model, including: Inputting the real image into the first feature extraction model to obtain the eighth image features; Inputting the random number and the preset class label into the initial feature generation model to obtain the ninth image features; Inputting the eighth image features and the ninth image features into the initial feature discrimination model respectively, calculating the second objective loss value of the adversarial network, and updating the initial feature discrimination model and the initial feature generation model respectively according to the second objective loss value to obtain the first feature discrimination model and the first feature generation model.

10. An information processing method, characterized in that, The method is applied to a server among multiple parties participating in federated learning, and the method includes: Obtaining the (N + 1)-th classification model and the (N + 1)-th feature generation model sent by each data provider among multiple parties; wherein, each data provider is used to train a feature generation model and a classification model, and after sequentially inputting the image to be predicted into the feature generation model and the classification model, obtain the corresponding classification result; the feature generation model is used to generate image features, and the classification model is used to output the corresponding classification result; Fusing each (N + 1)-th classification model to obtain the (N + 1)-th fused classification model, and sending the (N + 1)-th fused classification model to each data provider respectively, so that each data provider performs the (N + 2)-th classification model training; Fusing each (N + 1)-th feature generation model to obtain the (N + 1)-th fused feature generation model, and sending the (N + 1)-th fused feature generation model to each data provider respectively, so that each data provider performs the (N + 2)-th feature generation model training; Among them, the (N + 1)-th feature generation model is obtained by each data provider using the image features extracted by the (N + 1)-th feature extraction model as real data and the image features generated by the (N)-th fused feature generation model as pseudo data, and training the (N)-th fused feature generation model through an adversarial network; the (N + 1)-th classification model and the (N + 1)-th feature extraction model are obtained by each data provider training the (N + 1)-th first-updated feature extraction model and the (N)-th fused classification model according to the pseudo images and real images generated by the (N + 1)-th data generation model; the (N + 1)-th data generation model and the (N + 1)-th first-updated feature extraction model are both obtained by each data provider training the (N)-th data generation model and the (N)-th feature extraction model according to the (N)-th fused feature generation model and the (N)-th fused classification model, the data generation model is used to generate pseudo images, and the feature extraction model is used to extract image features.

11. An information processing apparatus, characterized in that, The device is applied to a data provider among multiple participants participating in federated learning, and is used to train a local feature generation model and a common classification model. After sequentially inputting an image to be predicted into the feature generation model and the classification model, a corresponding classification result is obtained. The device includes: An acquisition module, configured to acquire the Nth fused classification model and the Nth fused feature generation model sent by the server; wherein, the feature generation model is used to generate image features; A first processing module, configured to train the Nth data generation model and the Nth feature extraction model according to the Nth fused feature generation model and the Nth fused classification model, to obtain the (N + 1)th data generation model and the first updated feature extraction model in the (N + 1)th time. The data generation model is used to generate pseudo-images, and the feature extraction model is used to extract image features; A second processing module, configured to train the first updated feature extraction model in the (N + 1)th time and the Nth fused classification model according to the pseudo-images generated by the (N + 1)th data generation model and real images, to obtain the (N + 1)th classification model and the second updated feature extraction model in the (N + 1)th time. The classification model is used to determine a corresponding result according to image features, and the second updated feature extraction model in the (N + 1)th time is used as the feature extraction model in the (N + 1)th time; A third processing module, configured to use the image features extracted by the (N + 1)th feature extraction model as real data, and use the image features generated by the Nth fused feature generation model as pseudo data, and train the Nth fused feature generation model through an adversarial network, so that the output result of the feature generation model approximates the result output by the feature extraction model.

12. An information processing apparatus, characterized in that, The device is applied to a server among multiple participants participating in federated learning. The device includes: An acquisition module, configured to acquire the (N + 1)th classification model and the (N + 1)th feature generation model sent by each data provider among multiple participants; wherein, each data provider is used to train a feature generation model and a classification model, and after sequentially inputting an image to be predicted into the feature generation model and the classification model, a corresponding classification result is obtained; the feature generation model is used to generate image features, and the classification model is used to output a corresponding classification result; A first processing module, configured to fuse each (N + 1)th classification model to obtain the (N + 1)th fused classification model, and send the (N + 1)th fused classification model to each data provider respectively, so that each data provider conducts the (N + 2)th classification model training; A second processing module, configured to fuse each (N + 1)th feature generation model to obtain the (N + 1)th fused feature generation model, and send the (N + 1)th fused feature generation model to each data provider respectively, so that each data provider conducts the (N + 2)th feature generation model training; Among them, the (N + 1)-th feature generation model is obtained by each data provider using the image features extracted by the (N + 1)-th feature extraction model as real data, and the image features generated by the feature generation model after the N-th fusion as pseudo data, and training the feature generation model after the N-th fusion through an adversarial network; the (N + 1)-th classification model and the (N + 1)-th feature extraction model are obtained by each data provider using the pseudo images and real images generated by the (N + 1)-th data generation model to train the first updated feature extraction model in the (N + 1)-th time and the classification model after the N-th fusion; the (N + 1)-th data generation model and the first updated feature extraction model in the (N + 1)-th time are both obtained by each data provider training the N-th data generation model and the N-th feature extraction model according to the feature generation model after the N-th fusion and the classification model after the N-th fusion. The data generation model is used to generate pseudo images, and the feature extraction model is used to extract image features.

13. An information processing apparatus, characterized in that, The information processing device includes: a memory, a processor, and an information processing program stored on the memory and executable on the processor. When the information processing program is executed by the processor, it implements the steps of the information processing method according to any one of claims 1-10.

14. A computer-readable storage medium, characterized in that, An information processing program is stored on the computer-readable storage medium. When the information processing program is executed by the processor, it implements the steps of the information processing method according to any one of claims 1-10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the information processing method according to any one of claims 1-10.

Images