Federated learning method and system based on communication-aided perception

Abstract

The application provides a federated learning method and system based on communication-assisted perception. By learning the training data in the mentor model, soft labels are obtained, and then the soft labels are migrated to the student model, realizing the compression of the model. In addition, the tensor decomposition technology is introduced, further compressing the parameters of the student model, which can reduce the size of the model while maintaining the accuracy and precision of the global model, reduce the communication overhead in the model transmission process, and improve the training efficiency of the global model. In addition, model weighted aggregation can evaluate the contribution of different clients, further improving the accuracy and training efficiency of the global model. Homomorphic encryption technology can calculate the data without exposing the content of the data to ensure the security of the data, improve the security of the model, and protect the privacy of the user.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a federated learning method and system based on communication-assisted perception. Background Art

[0002] With the advancement of deep learning, models have become increasingly large, with some networks containing millions or even hundreds of millions of learnable parameters. Uploading and downloading these massive models incurs significant overhead, making them particularly unsuitable for clients with relatively limited communication bandwidth and throughput. Direct model compression can severely degrade model accuracy. Therefore, despite their impressive performance, large models are rarely deployed in traditional federated learning systems due to communication overhead. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a federated learning method and system based on communication-assisted perception, which is used to ensure the accuracy of the model while reducing the communication overhead in the process of transmitting the model.

[0004] Based on the above objectives, a first aspect of the present application provides a federated learning method based on communication-assisted perception, which is applied to each client of the federated learning; the method includes:

[0005] Obtaining training data and inputting the training data into the tutor model to obtain soft labels for output images;

[0006] jointly training a student model based on the soft labels and the true labels of the training data;

[0007] Performing tensor decomposition on the model parameters of the student model to obtain compressed model parameters;

[0008] Performing homomorphic encryption on the compression model parameters to obtain encrypted encryption model parameters;

[0009] Uploading the encrypted model parameters to a central server, so that the central server can train a global model based on the encrypted model parameters;

[0010] Receive the global model parameters fed back by the central server, update the student model according to the global model parameters, obtain new encrypted model parameters, and send the new encrypted model parameters to the central server until the global model sent by the central server is received, and determine the image recognition result according to the global model.

[0011] A second aspect of the present application provides a client, including:

[0012] The label determination module is configured to: obtain training data, and input the training data into a tutor model to obtain a soft label of an output image;

[0013] The model training module is configured to: jointly train a student model according to the soft label and a real label of the training data;

[0014] The tensor decomposition module is configured to: perform tensor decomposition on model parameters of the student model to obtain compressed model parameters after compression;

[0015] The homomorphic encryption module is configured to: perform homomorphic encryption on the compressed model parameters to obtain encrypted model parameters after encryption;

[0016] The parameter uploading module is configured to: upload the encrypted model parameters to a central server, so that the central server trains a global model according to the encrypted model parameters;

[0017] The image recognition module is configured to: receive global model parameters fed back by the central server, update the student model according to the global model parameters to obtain new encrypted model parameters, and send the new encrypted model parameters to the central server until a global model sent by the central server is received, and determine an image recognition result according to the global model.

[0018] The third aspect of the application provides a federated learning method based on communication-assisted perception, applied to a central server of the federated learning; the method comprises:

[0019] Receiving encrypted model parameters uploaded by a plurality of clients;

[0020] According to the encrypted model parameters uploaded by the plurality of clients, performing model aggregation through a weighted average mechanism to obtain a global model;

[0021] Sending global model parameters of the global model to each client, so that each client updates a student model according to the received global model parameters;

[0022] Judging whether the global model converges, if the global model converges, deploying the global model to the client, so that the client determines an image recognition result according to the global model; if the global model does not converge, performing iterative training of the federated learning on the global model until the global model converges.

[0023] The fourth aspect of the application provides a federated learning system based on communication-assisted perception, comprising a central server and a plurality of clients in communication connection with the central server; the system comprises:

[0024] The client obtains training data and inputs the training data into a tutor model to obtain a soft label of an output image;

[0025] The client jointly trains a student model according to the soft label and a true label of the training data;

[0026] The client performs tensor decomposition on model parameters of the student model to obtain compressed compressed model parameters;

[0027] The client homomorphically encrypts the compressed model parameters to obtain encrypted encrypted model parameters;

[0028] The client uploads the encrypted model parameters to the central server;

[0029] The central server aggregates the global model according to the encrypted model parameters uploaded by multiple clients through a weighted average mechanism to obtain a global model;

[0030] The central server sends global model parameters of the global model to each client, so that each client updates the student model according to the received global model parameters;

[0031] The central server determines whether the global model converges, and if the global model converges, deploys the global model to a preset target client to determine an image recognition result according to the global model; if the global model does not converge, iteratively trains the global model through the federated learning until the global model converges.

[0032] From the above, it can be seen that the federated learning method and system based on communication aided perception provided by the application realize the compression of the model by learning the training data in the tutor model to obtain the soft label, and then migrating the soft label to the student model. In addition, the tensor decomposition technology is introduced to further compress the parameters of the student model, which can reduce the size of the model while maintaining the accuracy and precision of the global model, reduce the communication overhead in the model transmission process, and improve the training efficiency of the global model. In addition, model weighted aggregation can evaluate the contribution of different clients, further improving the accuracy and training efficiency of the global model. Considering privacy protection, the homomorphic encryption mechanism is also introduced. The homomorphic encryption mechanism can encrypt and calculate data to ensure data security. The homomorphic encryption technology can calculate the data without exposing the content of the data. Therefore, even if the data is stolen during transmission, the attacker cannot obtain useful information, achieving the effect of improving the security of the model and protecting the privacy of the user. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description only constitute the embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0034] Figure 1 a schematic diagram of a federated learning process for an embodiment of the application;

[0035] Figure 2 a schematic diagram of a federated learning system based on communication aided perception provided by an embodiment of the application;

[0036] Figure 3 a flowchart of a federated learning method based on communication aided perception executed in a client by an embodiment of the application;

[0037] Figure 4 a flowchart of a federated learning method based on communication aided perception executed in a central server by an embodiment of the application;

[0038] Figure 5 a flowchart of obtaining training data by an embodiment of the application;

[0039] Figure 6 a flowchart of training a student model by an embodiment of the application;

[0040] Figure 7 a flowchart of determining compressed model parameters by an embodiment of the application;

[0041] Figure 8 a flowchart of training a global model by an embodiment of the application;

[0042] Figure 9 a structural schematic diagram of a client by an embodiment of the application;

[0043] Figure 10 a structural schematic diagram of an electronic device by an embodiment of the application. DETAILED DESCRIPTION

[0044] In order to make the purposes, technical solutions and advantages of the application clearer, the following will further describe the application in detail with specific embodiments and with reference to the drawings.

[0045] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the application should be understood as the usual meanings understood by those skilled in the art in the field to which the application belongs. The words such as "include" or "contain" mean that the elements or objects appearing before the words cover the elements or objects listed after the words and their equivalents, without excluding other elements or objects.

[0046] In this document, it should be understood that the number of any elements in the figures of the drawing is given by way of example only and not limitation, and that any naming is merely for the purpose of distinguishing between like elements and does not have any limiting meaning.

[0047] Based on the description of the above background art, there are also the following cases in the related art:

[0048] With the continuous development of communication technology, the performance of communication and perception is approaching their respective theoretical limits. By integrating communication and sensing modules on the same hardware to obtain possible hardware gain or information gain, integrated perception and communication has gradually become a development trend in these two fields.

[0049] Federal learning is an important technology for model aggregation training of distributed nodes, which can learn a global model from scattered private data. Federal learning provides a privacy-aware learning paradigm that does not require raw data sharing but allows participating clients to collaborate to build a better global model by sharing only their local model parameters. In federal learning, private data is not uploaded to a centralized database but stored locally on different clients, so when training a machine learning model on scattered data, user privacy can be protected to some extent.

[0050] Among them, as shown in Figure 1 Federal learning is a machine learning paradigm, and the central server of federal learning and multiple participating clients ensure that data can complete the training of the federal learning model without leaving the local through the transmission of model parameters, that is, the data does not move and the model moves, so as to ensure the privacy and security of the client data. For any participating client participating in federal learning, the central server sends the initial global model to the client, the client updates the initial global model using its own data set, and then returns the updated model or model parameters to the central server. The central server aggregates the updated models returned by each client to complete the model training of one round of communication rounds, and after multiple rounds of communication training, the global model obtained in the last round of communication is output after the global model converges.

[0051] In the scenario of communication-aided perception, the classic case is that multiple clients perform the same sensing task from different perspectives, and then obtain the result of perception through data exchange and decision fusion. In the federated learning system, the communication efficiency between the client and the central server directly affects the performance of sensing. The central server and the client need to transmit model parameters or models intensively during model training. Therefore, if the size of the model is large and the number of parameters is too much, the communication cost between the central server and the client is huge. However, with the development of deep learning, the size of the model becomes larger and larger, and some networks even contain millions or even hundreds of millions of learnable parameters. When uploading and downloading these huge models, it will cause huge overhead, which is particularly not suitable for clients with relatively limited communication bandwidth and throughput. Therefore, although the performance of large models is very excellent, due to the limitation of communication overhead, they are rarely deployed in traditional federated learning systems.

[0052] In the related art, in the current research of large deep learning models, the main method used to improve the communication efficiency of models between nodes is to compress the model. The current model compression methods mainly include model pruning and quantization. The model compression method based on pruning reduces the number of parameters of the model by deleting unnecessary neurons or connections, thereby realizing the compression of the model. Specifically, this method can use structured pruning or unstructured pruning techniques to prune the model according to the importance of neurons or connections, thereby reducing the size of the model. Structured pruning can delete entire convolutional layers, while unstructured pruning can delete individual neurons or connections. Although these methods can effectively reduce the size of the model, it is difficult to ensure that the compressed model still has high precision and robustness.

[0053] Gradient compression can also be used in the related art to directly reduce the size of the model. However, when the required compression ratio is very large, the model usually suffers from severe performance loss, and when the required compression ratio is small, the effect of reducing communication overhead cannot be achieved.

[0054] For large-scale integrated machine learning of distributed nodes, frequent communication is needed between nodes for model updating and synchronization, which brings a lot of communication overhead, especially in the case of a large number of nodes, the communication cost can become the bottleneck of the system; at the same time, due to the time delay, network fluctuation and other factors between nodes, the synchronization failure between nodes may increase the error and affect the model training effect of the whole system; the algorithm of the model usually needs to process massive data and high-dimensional features, and complex algorithms and optimization methods are needed to ensure the efficiency and accuracy of the model, so the pruned or quantized model still has a large scale, and the communication overhead is still the bottleneck of the system performance. And the traditional pruning method only deletes part of the neurons or connections, which cannot reduce the communication overhead and may also cause the accuracy of the model to decrease. And if the deleted neuron nodes or connection nodes have sensitive data, or there are some malicious nodes in the whole model, there will be privacy security problems after pruning.

[0055] Based on the above description, the principles and spirits of the present application will be explained in detail below with reference to several representative embodiments of the present application.

[0056] Reference Figure 2 , which is a federated learning system based on communication aided perception provided by the embodiments of the present application. The system is based on the scene of integrating perception and communication, and adopts the network architecture of federated learning. There is a central server S and N clients C={c1, c2,…,c N} in the whole system. Each client in the system is represented as c i , c i ∈C, index i∈{1,2,…,N}, each client collects a local data set D i using perception means, the size of D i is represented by S i , and D i contains K i classes of data. Each local data set D i will not leave the client that stores it. Each client c i has trained a large local mentor model Mentor i with parameter set using the local data set D i , and has constructed a small local student model Mentee i without training with parameter set . The student model Mentee iBefore uploading, tensor decomposition is performed to further compress model parameters. A central server, S, randomly selects M clients to establish communication and coordinates these M clients to collaboratively learn the model. The goal is to learn a model that is communication-efficient and has good discriminative performance in a privacy-preserving manner.

[0057] The embodiment provided in this application achieves model compression by learning the training data in a tutor model to obtain soft labels, and then migrating the soft labels to the student model. In addition, tensor decomposition technology is introduced to further compress the parameters of the student model, which can reduce the scale of the model while maintaining the accuracy and precision of the global model, reduce the communication overhead during the model transmission process, and improve the training efficiency of the global model. Furthermore, model weighted aggregation can evaluate the contributions of different clients, further improving the accuracy and training efficiency of the global model. Based on privacy protection considerations, a homomorphic encryption mechanism is also introduced. The homomorphic encryption mechanism can encrypt and calculate data to ensure the security of the data. Homomorphic encryption technology can calculate the data without exposing its content. Therefore, even if the data is stolen during transmission, the attacker cannot obtain useful information, thereby improving the security of the model and protecting the privacy of the user.

[0058] The client and the central server may be connected via a wired or wireless communication network. Clients include but are not limited to desktop computers, mobile phones, mobile computers, tablet computers, media players, smart wearable devices, personal digital assistants (PDAs), or other electronic devices capable of performing the above functions. The central server may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0059] The following combination Figure 2 The application scenario is used to describe the federated learning method based on communication-assisted perception according to the exemplary embodiment of the present application. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0060] In some embodiments, as Figure 3 As shown, a federated learning method based on communication-assisted perception is applied to each client of the federated learning; the method includes:

[0061] Step 301: Obtain training data, and input the training data into the mentor model to obtain a soft label of an output image.

[0062] In specific implementation, remote sensing data in a SAR scene is collected by a synthetic aperture radar, radar data is processed by two-dimensional matched filtering to realize high-resolution imaging of radar waveform data. The obtained image is preprocessed to obtain a data set with uniform size. In order to perform a classification task, each picture in the SAR image data set is preprocessed, and all pictures are divided into a training set (training data) and a test set (test data) after classification.

[0063] In the training phase, each participating client (any one of the M clients randomly selected by the central server S in each training process is a participating client) inputs the training data into the Mentor model to generate a prediction result of the Mentor model, and the prediction result is referred to as a soft label of the data.

[0064] Optionally, the Mentor model can use a Resnext50_32x4d network, which can realize high-precision image classification tasks and does not have overfitting problems.

[0065] Step 302: Train the student model according to the soft label and the true label of the training data.

[0066] In specific implementation, the soft label contains the correlation between data learned by the Mentor model, fully considers the distance between classes and the variance information within the class, and therefore contains more data information. Training the student model in combination with the soft label and the true label of the training data can realize knowledge transfer, transfer the knowledge learned in the large Mentor model to the small student model, and realize model compression.

[0067] The knowledge distillation can be used to transfer knowledge between the large Mentor model and the small Mee student model. Since the network structure of the Mentor model is usually much more complex than that of the Mee student model, more classification knowledge (such as soft labels) can be learned from the common data set. The classification knowledge can be used to guide the training of the Mee student model, thereby realizing the knowledge transfer from the Mentor model to the Mee student model, and enabling the simple structure of the Mee student model to still have classification accuracy close to that of the Mentor model.

[0068] Optionally, the Mentee student model uses a classifier of a convolutional neural network (CNN) composed of 3 convolutional layers and 3 linear layers. The model is trained once, and the batch size is set to 64. The model is trained using the Adam optimizer.

[0069] Step 303: Tensor decomposition is performed on the model parameters of the student model to obtain compressed model parameters.

[0070] In specific implementation, on the basis of knowledge distillation, the Mentee student model is further divided into smaller models through tensor decomposition, and then the compressed model parameters of the smaller models after decomposition are uploaded, thereby realizing compression of the Mentee student model while maintaining the accuracy of the model. Optionally, the decomposed Mentee student model can be shared among different clients through the central server and used for collaborative learning, thereby further effectively reducing the communication cost.

[0071] Step 304: Homomorphic encryption is performed on the compressed model parameters to obtain encrypted model parameters.

[0072] In specific implementation, the BGN homomorphic encryption scheme is optionally used for encryption. Since the BGN algorithm realizes public key encryption based on bilinear mapping, it can support infinite homomorphic addition operations and one homomorphic multiplication operation. During transmission of the encrypted model parameters, the encrypted compressed model parameter data can be protected, and the homomorphic encryption technology can be calculated without exposing the content of the compressed model parameters. Therefore, even if the compressed model parameter data is stolen during transmission, the attacker cannot obtain useful information.

[0073] Step 305: The encrypted model parameters are uploaded to the central server, so that the central server trains a global model according to the encrypted model parameters.

[0074] In specific implementation, after encryption, the model parameters are uploaded to the central server located in the cloud. Due to the complexity of the real network environment, correct transmission of information will face many difficulties, and even some malicious nodes will tamper with the local student model or the global model during global aggregation, and different clients often have different sizes and different distributions of data sets. Therefore, a weighted average mechanism based on model correlation and model data volume is introduced in the central server. The Pearson correlation coefficient is used to represent the correlation degree of the Mentee student model of each client to be aggregated with the Mentee student models of the remaining clients to be aggregated, and the weight f of the model parameters of each client is determined according to the correlation degree. i ​According to the weight, model aggregation is performed to obtain a global model.

[0075] Step 306: receiving the global model parameters fed back by the central server, updating the student model according to the global model parameters, obtaining new encrypted model parameters, and sending the model parameters to the central server until the global model sent by the central server is received, and determining the image recognition result according to the global model.

[0076] In specific implementation, after the global model is updated, the central service area sends the latest global model parameters to each participating client participating in this round of model aggregation. After the client receives the global model parameters, the local Mentee student model parameters are updated to obtain new encrypted model parameters, and the model parameters are sent to the central server until the global model converges, the global model sent by the central server is received, and the image recognition result is determined according to the global model.

[0077] In some embodiments, as shown in Figure 4 , a federated learning method based on communication auxiliary perception is applied to a central server of federated learning; the method comprises:

[0078] Step 401: receiving encrypted model parameters uploaded by a plurality of clients.

[0079] In specific implementation, the encrypted model parameters uploaded by the client in the receiving step 305 are received, and the homomorphic encrypted encrypted model parameters are decrypted to prepare for subsequent model aggregation.

[0080] Step 402: according to the encrypted model parameters uploaded by the plurality of clients, model aggregation is performed through a weighted average mechanism to obtain a global model.

[0081] In specific implementation, the encrypted model parameters uploaded by the plurality of clients are decrypted to obtain compressed model parameters of the plurality of clients. Due to the complexity of the real network environment, the correct propagation of information will face many difficulties, and even some malicious nodes will tamper with the local model or the global model in the global aggregation process, and different clients often have different sizes and different distributions of data sets. Therefore, a weighted average mechanism based on model correlation and model data volume is introduced in the central server. The Pearson correlation coefficient is used to represent the correlation degree of the Mentee student model of each client to be aggregated with the Mentee student model of the remaining clients to be aggregated, and the weight f of the compressed model parameters of each client is determined according to the correlation degree. i According to the weight, model aggregation is performed to obtain a global model.

[0082] Step 403: sending the global model parameters of the global model to each client for each client to update the student model according to the received global model parameters.

[0083] In implementation, after obtaining the global model, the global model parameters of the global model are sent to each client for each client to perform steps 301 to 305 again according to the received global model parameters, so as to update the Mentee student model locally of the client, and then the central server performs the next round of model aggregation.

[0084] Step 404: determining whether the global model converges, if the global model converges, deploying the global model to the client for the client to determine the image recognition result according to the global model; if the global model does not converge, performing iterative training of the global model by federated learning until the global model converges.

[0085] In implementation, the central server and the client perform a plurality of rounds of iterative process of federated learning until the central server determines that the global model converges, and then the global model is deployed to the client for the client to determine the image recognition result according to the global model.

[0086] Overall, as shown in the application scenario of the federated learning system based on communication-aided perception, the system includes a central server and a plurality of clients in communication connection with the central server; the execution process of the federated learning method based on communication-aided perception in the system is as follows: Figure 2

[0087] The client obtains the training data and inputs the training data into the mentor model to obtain the soft label of the output image;

[0088] The client trains the student model according to the soft label and the real label of the training data;

[0089] The client performs tensor decomposition on the model parameters of the student model to obtain the compressed model parameters after compression;

[0090] The client performs homomorphic encryption on the compressed model parameters to obtain the encrypted model parameters after encryption;

[0091] The client uploads the encrypted model parameters to the central server;

[0092] The central server aggregates the model according to the encrypted model parameters uploaded by the plurality of clients through a weighted average mechanism to obtain a global model;

[0093] The central server sends the global model parameters of the global model to each client for each client to update the student model according to the received global model parameters;

[0094] ​The central server judges whether the global model converges, and if the global model converges, deploys the global model to a preset target client, so that the target client determines an image recognition result according to the global model; if the global model does not converge, iteratively trains the global model through federated learning until the global model converges.

[0095] The steps that the federated learning system based on communication-assisted perception needs to go through in global model training are summarized as follows:

[0096] (1) The training data is input into the Mentor model, and the soft label of the image is output, and the Mentor model is trained together with the real label and the soft label.

[0097] (2) Each client performs tensor decomposition on the parameters of the local Mentee model to obtain compressed model parameters.

[0098] (3) Homomorphic encryption is performed on the compressed model parameters to obtain encrypted model parameters.

[0099] (4) Each client uploads the encrypted model parameters to the central server, and then the central server aggregates the global model according to the weighted average mechanism.

[0100] (5) The center service area sends the latest global model parameters to each client participating in this round of model aggregation, and the client updates the local Mentee model parameters after receiving the global model parameters.

[0101] (6) Return to step 1 to continue the iterative training of the global model until the global model converges.

[0102] In summary, the federated learning system based on communication-assisted perception provided by the embodiments of the present application focuses on solving effective and efficient model transmission. In this system, based on knowledge distillation and tensor decomposition technology, communication means is used to assist perception in the integrated communication and perception scenario.

[0103] The federated learning system based on communication-assisted perception provided by the embodiments of the present application is a novel federated learning uplink optimization method. The system model combines knowledge distillation and tensor decomposition technology to minimize the communication cost in the federated process. In addition, homomorphic encryption and weighted aggregation methods are also used in the system to protect the privacy and security of the system.

[0104] Further, the system can also learn personalized models on non-identically distributed data, thereby alleviating the challenges brought by data heterogeneity and model heterogeneity in real-world scenarios. Meanwhile, the system also utilizes homomorphic encryption technology, so that the data can be protected during transmission. Homomorphic encryption technology can perform calculations on data without exposing the content of the data. Therefore, even if the data is stolen during transmission, the attacker cannot obtain useful information. In addition, the weighted aggregation technology can also evaluate the contribution of each participant, thereby further improving the accuracy and communication efficiency of the model. The federated learning method based on communication-assisted perception provided by the embodiments of the application is a general method that can provide a new direction for compressing large deep learning models, while maintaining high precision and improving communication efficiency.

[0105] The federated learning method based on communication-assisted perception provided by the embodiments of the application achieves model compression by learning in the mentor model, obtaining soft labels, and then migrating the soft labels to the student model. In addition, the tensor decomposition technology is introduced, which further compresses the parameters of the student model, reduces the size of the model, reduces the communication overhead during model transmission, and improves the training efficiency of the global model while maintaining the accuracy and precision of the global model. Further, model weighted aggregation can evaluate the contribution of different clients, further improving the accuracy and efficiency of the global model. Considering privacy protection, a homomorphic encryption mechanism is also introduced. The homomorphic encryption mechanism can encrypt and calculate data to ensure data security. Homomorphic encryption technology can perform calculations on data without exposing the content of the data. Therefore, even if the data is stolen during transmission, the attacker cannot obtain useful information, achieving the effect of improving the security of the model and protecting the privacy of the user.

[0106] In some embodiments, as shown in Figure 5 The training data is obtained, including:

[0107] Step 501: Collect remote sensing data using a synthetic aperture radar (SAR).

[0108] In specific implementation, the remote sensing data of the preset sensing target can be collected using a synthetic aperture radar (SAR) in a preset sensing area. For example, the radar waveform data of a bus, a truck or other vehicles in a certain area is collected using a synthetic aperture radar, and the radar waveform data is determined as the remote sensing data.

[0109] Step 502: Perform imaging processing on the remote sensing data using two-dimensional matched filtering to obtain a plurality of radar sensing images.

[0110] In specific implementation, the collected remote sensing data is processed by two-dimensional matched filter imaging to realize high-resolution imaging of the radar waveform data, and a plurality of radar perception images are obtained.

[0111] For a radar communication dual-functional system (Dual-Functional Radar and Communication, DFRC) with a synthetic aperture radar, remote sensing data thereof can be processed by two-dimensional matched filtering to obtain high-resolution images, and the generated images are not affected by light and weather conditions, can present a target hidden by vegetation, and have good target recognition performance, so that the radar target detection and classification task can be converted into an image classification and recognition task.

[0112] Step 503: classifying and preprocessing the plurality of radar perception images to obtain an image dataset; wherein the preprocessing is to uniformly process the perception images of different radars into the same pixel size.

[0113] In specific implementation, the obtained radar perception images are classified and labeled, and the plurality of radar perception images are uniformly processed into 64x64 pixel size, so as to provide traversal for subsequent image recognition and improve image training efficiency.

[0114] Step 504: dividing the image dataset into training data and test data according to a preset ratio.

[0115] In specific implementation, the processed image dataset is divided into a training set and a test set according to a ratio of 4:1, and the training set is used to train the Mentee student model of the client.

[0116] In some embodiments, as shown in Figure 6 the student model is trained according to the soft label and the true label of the training data, including:

[0117] Step 601: determining a supervised learning loss of the student model according to the true label.

[0118] In specific implementation, the true label of the training data and the soft label output by the Mentor teacher model are used to guide the training of the respective local Mentee student model, so that two loss functions are used in the training of the Mentee student model, including a supervised learning loss of the Mentee student model under the true label:

[0119]

[0120]

[0121] wherein, represents a cross-entropy loss of an individual sample of the i-th client; CE represents a cross-entropy loss function. represents the average cross-entropy loss of all samples of the i-th client, S i represents the number of samples of all samples in the i-th client.

[0122] Step 602: Extract the knowledge distillation loss from the soft label by knowledge distillation.

[0123] In specific implementation, the loss function used in the training of the Mentee student model also includes the knowledge distillation loss extracted from the soft label by knowledge distillation:

[0124]

[0125]

[0126] wherein, represents the knowledge distillation loss of a single sample of the i-th client; KL represents the divergence loss function; represents the average knowledge distillation loss of all samples of the i-th client.

[0127] Step 603: Determine the weighted value of the supervised learning loss and the knowledge distillation loss as the total loss, and train the student model according to the total loss.

[0128] In specific implementation, the weighted value of the knowledge distillation loss and the supervised learning loss is used as the total loss value in the training process:

[0129]

[0130] wherein, L i represents the total weighted loss of all samples of the i-th client, and λ represents the proportion of the average knowledge distillation loss in the total loss.

[0131] In some embodiments, as Figure 7 shown, the model parameters of the student model are decomposed by tensor decomposition to obtain compressed compressed model parameters, including:

[0132] Step 701: Decompose the linear layer of the student model into two layers of cascaded linear layers by singular value decomposition, and introduce a preset intermediate layer to obtain compressed compressed linear layer parameters.

[0133] In specific implementation, for the linear layer in the Mentee student model, singular value decomposition (Singular Value Decomposition, SVD) is used to decompose it into a linear layer cascade form. Generally, the decrease of singular value size is very rapid, and the energy of the first d singular values can reach more than 99% of the total energy, so the largest first d singular values and the corresponding left and right singular vectors can be used to approximate the original parameter matrix:

[0134]

[0135] Since the matrix dot product satisfies the commutative law, its output can be expressed as:

[0136] y = A'x + b = (U m×d W d×n )x + b = (U m×d x)W d×n +b

[0137] The above mapping relationship can be expressed as two linear layers in cascade, the first layer is expressed as: y1 = U m×d x, and the second layer is expressed as: y = y1W d×n +b.

[0138] wherein A' m×n represents the parameter matrix of the linear layer, U m×d represents the left matrix after singular value decomposition, represents the right matrix after singular value decomposition; W d×n represents the product of the singular value matrix and the right matrix, that is, the parameter matrix of the second linear layer after decomposition; d represents the number of effective singular values, and b represents the bias in the linear layer parameters.

[0139] The SVD decomposition introduces an intermediate layer Middle containing d neurons, reduces the number of full connections and the size of the weights, and reduces the number of parameters of the Mentee student model after introducing the intermediate layer, to obtain the compressed linear layer parameters after compression.

[0140] The process of reducing the linear layer parameters is as follows: the dimension of the model parameter A' m×n before singular value decomposition is m x n, and after singular value decomposition, the original model parameter matrix is converted into the product of two decomposition matrices, which can be equivalent to two linear layers in series, the parameter of the first linear layer is U m×d , the dimension is m x d, the parameter of the second linear layer is W d×n , the dimension is n x d, and the total number of parameters is (m+n) x d. Therefore, the number of model parameters before decomposition is m x n, and the number of model parameters after decomposition is (m+n) x d. The value of d is generally much smaller than m and n, so as to achieve the purpose of reducing the parameters.

[0141] Step 702: decompose the convolution layer of the student model into three layers of cascade convolution layers through high-order singular value decomposition, to obtain the compressed convolution layer parameters after compression.

[0142] In a specific implementation, for the convolution layer in the Mentee student model, the weight matrix is a four-dimensional tensor K, and therefore we introduce Tucker decomposition (high-order singular value decomposition) to realize model compression of the convolution layer. For the convolution layer, the relationship between the input tensor and the output tensor can be expressed as follows:

[0143]

[0144] where y e R h′×w′×t represents the output tensor, K e R m×n×s×t , x e R h×w×s represents the input tensor, and by using Tucker decomposition, the tensor K can be decomposed as follows:

[0145]

[0146] where G e R represents the core tensor, and G1, G2, G3, and G4 represent the corresponding factor matrices (the upper right corner mark indicates the corresponding mode number). Because mode 1 and mode 2 are associated with the spatial dimension, they are already quite small, and therefore only need to be decomposed along mode 3 and mode 4, and the tensor K can be decomposed as follows:

[0147]

[0148] Substituting this into the output expression, we can obtain:

[0149]

[0150] This mapping relationship can be expressed as a three-layer cascaded convolution layer after decomposition, and the compressed convolution layer parameters after compression are obtained.

[0151] where G1 e R represents the factor matrix of mode 1; represents the factor matrix of mode 2; represents the factor matrix of mode 3; represents the factor matrix of mode 4, which is also the parameter matrix of the third layer of the convolution layer after decomposition; represents the simplified core tensor; represents the parameter matrix of the first layer of the convolution layer after decomposition represents the parameter matrix of the second layer of the convolution layer after decomposition; y h′，w′，t represents the parameter matrix of the convolution layer.

[0152] The process of realizing parameter reduction of the convolution layer is the same as that of the linear layer, and therefore is not described in detail here.

[0153] Step 703: integrate the compressed linear layer parameters and the compressed convolution layer parameters to obtain the compressed model parameters.

[0154] In specific implementation, the decomposed compressed linear layer parameters and the compressed convolution layer parameters are integrated to obtain the compressed model parameters, completing the compression of the model, reducing the communication overhead in the model transmission process, and improving the training efficiency of the global model.

[0155] In some embodiments, the compressed model parameters are homomorphic encrypted to obtain encrypted model parameters, including:

[0156] The compressed model parameters are subjected to one homomorphic multiplication operation and at least one homomorphic addition operation to obtain the encrypted model parameters.

[0157] In specific implementation, the BGN homomorphic encryption scheme is used for encryption. Since the BGN algorithm realizes public key encryption based on bilinear mapping, it can support infinite homomorphic addition operations and one homomorphic multiplication operation. This makes the encrypted model parameters protected during transmission, and the homomorphic encryption technology can calculate the encrypted model parameters without exposing the content of the compressed model parameters. Therefore, even if the compressed model parameters are stolen during transmission, attackers cannot obtain useful information, improving the security during transmission.

[0158] In some embodiments, as shown in Figure 8 , the global model is obtained by model aggregation through a weighted average mechanism according to the encrypted model parameters uploaded by multiple clients, including:

[0159] Step 801: decrypt the encrypted model parameters to obtain the compressed model parameters.

[0160] In specific implementation, the encrypted model parameters uploaded by the client c i are decrypted to obtain the compressed model parameters of the client c i . The encrypted model parameters uploaded by the client c j are decrypted to obtain the compressed model parameters of the client c j .

[0161] Step 802: determine the weight of the compressed model parameters of each client according to the compressed model parameters.

[0162] In some embodiments, the weight of the compressed model parameters of each client is determined according to the compressed model parameters, including:

[0163] Step 8021: determine the correlation coefficient between the model parameters according to the compressed model parameters.

[0164] In specific implementation, the correlation coefficient (such as Pearson correlation coefficient) r i It can be expressed as:

[0165]

[0166] Where M represents the number of clients participating in a federated learning; Represents the compression model parameters of the i-th client The mean of the pth parameter; ω ip Represents the compression model parameters of the i-th client The pth parameter of jp Represents the compression model parameters of the jth client The pth parameter of ; E represents the compression model parameter of the i-th client The total number of parameters.

[0167] Step 8022: Determine the weight of the compression model parameters of each client according to the correlation coefficient.

[0168] In specific implementation, for client c i For example, client c i The weight f i The definition is as follows:

[0169]

[0170] Among them, S i represents the number of samples of all samples in the i-th client, f i Represents the weight of the compression model parameters of the i-th client.

[0171] Step 803: Perform model aggregation based on the compression model parameters and weights to obtain a global model.

[0172] In specific implementation, the global model parameters It can be expressed as:

[0173]

[0174] After the global model update is completed, the central server will send the latest global model parameters to each node participating in this round of model aggregation. After receiving the global model parameters, the client will update the local model parameters until the central server determines that the global model has converged and the model training is completed.

[0175] It should be noted that the method of the embodiments of the present application can be executed by a single device, for example, a computer or a server, etc. The method of the embodiments can also be applied to a distributed scenario, and be completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiments of the present application, and the multiple devices can interact with each other to complete the method.

[0176] It should be noted that some embodiments of the present application have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order in order to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

[0177] Based on the same inventive concept, the present application also provides a client corresponding to the method of any of the above embodiments.

[0178] Reference Figure 9 The client comprises:

[0179] The label determination module 10 is configured to: obtain training data, and input the training data into a tutor model to obtain a soft label of an output image;

[0180] The model training module 20 is configured to: jointly train a student model according to the soft label and a real label of the training data;

[0181] The tensor decomposition module 30 is configured to: perform tensor decomposition on a model parameter of the student model to obtain compressed compressed model parameters;

[0182] The homomorphic encryption module 40 is configured to: perform homomorphic encryption on the compressed model parameters to obtain encrypted encrypted model parameters;

[0183] The parameter uploading module 50 is configured to: upload the encrypted model parameters to a central server, so that the central server trains a global model according to the encrypted model parameters;

[0184] The image recognition module 60 is configured to: receive global model parameters fed back by the central server, and update the student model according to the global model parameters to obtain new encrypted model parameters, and send the model parameters to the central server until a global model sent by the central server is received, and determine an image recognition result according to the global model.

[0185] For the convenience of description, the above apparatus is described in various modules in terms of functions. Of course, the functions of the modules can be implemented in one or more software and / or hardware in the implementation of the present application.

[0186] The apparatus of the above embodiments is used to implement the corresponding communication-assisted perception-based federated learning method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0187] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the communication-assisted perception-based federated learning method according to any of the above embodiments.

[0188] Figure 10 A more specific hardware structure of an electronic device according to the present embodiment is shown, which can include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for communication within the device.

[0189] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the present embodiment.

[0190] The memory 1020 can be implemented by a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs, and when the technical solutions provided by the present embodiment are implemented by software or firmware, the related program codes are stored in the memory 1020 and executed by the processor 1010.

[0191] The input / output interface 1030 is configured to connect an input / output module to realize information input and output. The input / output module can be configured in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0192] The communication interface 1040 is configured to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0193] The bus 1050 includes a channel to transmit information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0194] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only contain components necessary for implementing the embodiments of the present specification, and does not have to contain all the components shown in the figure.

[0195] The electronic device of the above embodiments is used to implement the corresponding communication-assisted perception-based federated learning method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0196] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the communication-assisted perception-based federated learning method according to any of the above embodiments.

[0197] The computer readable media of the embodiments can include permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible to a computing device.

[0198] The storage medium of the above embodiments stores computer instructions for causing the computer to execute the communication-assisted perception-based federated learning method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0199] Those skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope (including claims) of the present application is limited to these examples; the above embodiments or technical features between different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of the different aspects of the embodiments of the present application as described above. In order to be brief, they are not provided in detail.

[0200] In addition, in order to simplify the description and discussion, and so as not to make the embodiments of the present application difficult to understand, the well-known power / ground connections of integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. In addition, the devices can be shown in the form of block diagrams in order to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented the embodiments of the present application (i.e. these details should be fully within the understanding of those skilled in the art). Where specific details (e.g. circuits) are set forth in order to describe the exemplary embodiments of the present application, it will be apparent to those skilled in the art that the embodiments of the present application can be implemented without these specific details or with variations of these specific details. Therefore, these descriptions should be considered illustrative rather than limiting.

[0201] While the present application has been described in connection with certain embodiments thereof, many modifications, substitutions, changes, and of forms will be apparent to those of ordinary skill in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0202] Embodiments of the present application are intended to cover all such alterations, modifications, and variations as they can come within the scope of the appended claims. Accordingly, although specific embodiments have been furthered in connection with the present application, any omission, substitution, change, improvement, etc. made by one of ordinary skill in the art to the disclosed embodiments should be considered to be within the scope of the present application.

Claims

1. A federated learning method based on communication-assisted perception, characterized in that: Applied to each client of the federated learning; the method includes: Obtaining training data and inputting the training data into the tutor model to obtain soft labels for output images; jointly training a student model based on the soft labels and the true labels of the training data; Performing tensor decomposition on the model parameters of the student model to obtain compressed model parameters; Performing homomorphic encryption on the compression model parameters to obtain encrypted encryption model parameters; Uploading the encrypted model parameters to a central server, so that the central server can train a global model based on the encrypted model parameters; Receive the global model parameters fed back by the central server, update the student model according to the global model parameters, obtain new encrypted model parameters, and send the new encrypted model parameters to the central server until the global model sent by the central server is received, and determine the image recognition result according to the global model.

2. The method for federated learning based on communication-assisted perception according to claim 1, characterized in that: The obtaining of training data includes: Collect remote sensing data using synthetic aperture radar (SAR); Performing imaging processing on the remote sensing data using two-dimensional matched filtering to obtain a plurality of radar perception images; Classifying and preprocessing the plurality of radar perception images to obtain an image data set; wherein the preprocessing is to uniformly process the perception images of different radars into the same pixel size; The image data set is divided into training data and test data according to a preset ratio.

3. The method for federated learning based on communication-assisted perception according to claim 1, characterized in that: The method of jointly training a student model according to the soft labels and the true labels of the training data comprises: Determining a supervised learning loss for the student model based on the true label; extracting a knowledge distillation loss from the soft labels through knowledge distillation; A weighted value of the supervised learning loss and the knowledge distillation loss is determined as a total loss, and the student model is trained according to the total loss.

4. The method for federated learning based on communication-assisted perception according to claim 1, characterized in that: The tensor decomposition of the model parameters of the student model to obtain compressed model parameters includes: Decomposing the linear layer of the student model into two cascaded linear layers by singular value decomposition, and introducing a preset intermediate layer to obtain compressed linear layer parameters; Decomposing the convolutional layer of the student model into three cascaded convolutional layers by high-order singular value decomposition to obtain compressed convolutional layer parameters; The compression model parameters are obtained by integrating the compression linear layer parameters and the compression convolutional layer parameters.

5. The method for federated learning based on communication-assisted perception according to claim 1, characterized in that: The homomorphically encrypting the compression model parameters to obtain encrypted encryption model parameters includes: Performing one homomorphic multiplication operation and at least one homomorphic addition operation on the compression model parameters to obtain the encryption model parameters.

6. The method for federated learning based on communication-assisted perception according to claim 1, characterized in that: The central server of the federated learning is in communication with a plurality of clients; the central server is configured to: Receive encrypted model parameters uploaded by multiple clients; Based on the encrypted model parameters uploaded by multiple clients, model aggregation is performed through a weighted average mechanism to obtain a global model; Sending global model parameters of the global model to each client, so that each client updates the student model according to the received global model parameters; Determine whether the global model converges. If the global model converges, deploy the global model to the client so that the client can determine the image recognition result based on the global model. If the global model does not converge, perform iterative training of the federated learning on the global model until the global model converges.

7. The method for federated learning based on communication-assisted perception according to claim 6, characterized in that: The method of performing model aggregation based on the encrypted model parameters uploaded by multiple clients through a weighted average mechanism to obtain a global model includes: Decrypting the encrypted model parameters to obtain compressed model parameters; Determining a weight of the compression model parameter of each client according to the compression model parameter; Model aggregation is performed according to the compression model parameters and the weights to obtain the global model.

8. The method for federated learning based on communication-assisted perception according to claim 7, characterized in that: Determining the weight of the compression model parameter of each client according to the compression model parameter includes: determining correlation coefficients between model parameters based on the compression model parameters; The weight of the compression model parameter of each client is determined according to the correlation coefficient.

9. A client, characterized in that: include: The label determination module is configured to: obtain training data, and input the training data into the tutor model to obtain a soft label for the output image; A model training module is configured to: jointly train a student model based on the soft labels and the true labels of the training data; A tensor decomposition module is configured to: perform tensor decomposition on the model parameters of the student model to obtain compressed model parameters; The homomorphic encryption module is configured to: perform homomorphic encryption on the compression model parameters to obtain encrypted encryption model parameters; a parameter uploading module configured to: upload the encryption model parameters to a central server, so that the central server can train a global model according to the encryption model parameters; The image recognition module is configured to: receive the global model parameters fed back by the central server, update the student model according to the global model parameters, obtain new encrypted model parameters, and send the new encrypted model parameters to the central server until the global model sent by the central server is received, and determine the image recognition result according to the global model.

10. A federated learning system based on communication-assisted perception, characterized in that: The system includes a central server and a plurality of clients in communication with the central server; The client obtains training data and inputs the training data into the tutor model to obtain soft labels for output images; The client jointly trains a student model based on the soft labels and the true labels of the training data; The client performs tensor decomposition on the model parameters of the student model to obtain compressed model parameters; The client performs homomorphic encryption on the compression model parameters to obtain encrypted encryption model parameters; The client uploads the encryption model parameters to the central server; The central server aggregates the models using a weighted average mechanism based on the encrypted model parameters uploaded by multiple clients to obtain a global model; The central server sends the global model parameters of the global model to each client, so that each client updates the student model according to the received global model parameters; The central server determines whether the global model has converged. If the global model has converged, the global model is deployed to a preset target client so that the target client can determine the image recognition result based on the global model. If the global model has not converged, the global model is iteratively trained by the federated learning until the global model converges.

Images