Gradient Decomposition Method for Federated Domain Adaptation Object Detection and Electronic Device

Abstract

The present invention discloses a gradient decomposition method and an electronic device for federated domain adaptation object detection, which relates to the technical field of federated learning and solves the problem of gradient entanglement existing in the existing technology for the federated domain generalization object detection task. The method includes: respectively inputting different images to be detected into different clients, collecting multiple object detection frames of the images to be detected according to the local models in the clients, and respectively calculating the detection loss values of the multiple object detection frames; calculating the parameter gradients in the local models according to the detection loss values corresponding to the respective local models, using principal component analysis to decompose the gradients from each client, and only transmitting the principal gradients to the server for further aggregation, and obtaining the target model according to the aggregation result; realizing that on the basis of decomposing the gradients, a special adversarial attack is introduced to synthesize adversarial samples, further improving the generalization ability.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and particularly to a gradient decomposition method and an electronic device for federated domain adaptation object detection. Background Art

[0002] As an extension of domain generalization, methods for domain generalization object detection can be roughly divided into single-domain data optimization and multi-domain data integration. For single-domain data optimization, domain-invariant representations are extracted from input visual features without domain labels through a cyclic decomposition module. For multi-domain data integration, multiple data domain features are integrated through data augmentation, enhancing the domain diversity of small datasets; a linear transformation matrix is introduced during the training phase to transform multiple domain information to help the model learn different domain information and overcome the domain gap of a single dataset.

[0003] Currently, the most widely used methods in federated learning are FedAvg, FedProx, FedBN, and FedProto. However, these methods perform poorly in domain generalization tasks. Therefore, some researchers have proposed federated domain generalization tasks. In federated domain generalization tasks, mainstream methods improve the generalization performance by designing different aggregation schemes. On the one hand, the generalization performance of the model on unknown domains is improved by adding a reference distribution to align the data distributions from different clients; furthermore, a continuous frequency domain interpolation method is introduced into federated domain generalization, or a domain classifier is introduced on the basis of traditional federated learning methods to improve the performance of the domain-invariant feature extractor in the model by enhancing the model's ability to distinguish domain information.

[0004] In the prior art, there is a gradient entanglement problem in federated domain generalization object detection. On the one hand, this entanglement is necessary for the server to learn from the client, which can better generalize to unknown domains; on the other hand, these heterogeneous gradients usually contain different or even contradictory components, which can lead to a deviation from the expected optimization direction, resulting in poor generalization performance. These two aspects of gradient entanglement constitute the dilemma of generalization. Summary of the Invention

[0005] The present invention provides a gradient decomposition method for federated domain adaptation object detection, which solves the gradient entanglement problem existing in the prior art for federated domain generalization object detection tasks, and realizes that on the basis of decomposing the gradient, a special adversarial attack is introduced to synthesize adversarial samples, further improving the generalization ability.

[0006] In a first aspect, the present invention provides a gradient decomposition method for federated domain adaptation object detection, the method comprising:

[0007] S1. Use the local model in the client to collect multiple target detection boxes of the image to be detected, and calculate the detection loss values of the multiple target detection boxes respectively; wherein, the images to be detected of different clients are different;

[0008] S2. Calculate the parameter gradients of the local model according to the detection loss values corresponding to the detection target boxes, and perform principal component analysis on the parameter gradients to obtain the main gradients corresponding to each local model;

[0009] S3. Generate adversarial samples corresponding to each local model by the fast gradient sign method according to the main gradients corresponding to each local model and the images to be detected corresponding to each local model;

[0010] S4. Calculate new gradient parameters according to the adversarial samples corresponding to each local model, and perform principal component analysis on the new gradient parameters to obtain the main gradients of each local model in the current round;

[0011] S5. Input the main gradients of each local model in the current round into the central server for aggregation to obtain an aggregation result, update the global model according to the aggregation result, and send the updated global model to each client for updating the local models in each client;

[0012] S6. Determine whether the main gradients of each local model in the current round are greater than a first threshold. If so, execute S7; if not, execute S3;

[0013] S7. Upload the main gradients of each local model in the current round to the central server for aggregation to obtain the aggregation result of the current round, and obtain the target model according to the aggregation result of the current round.

[0014] In combination with the first aspect, in a possible implementation manner, the calculation formula of the detection loss value is:

[0015]

[0016] wherein, represents the classification loss; represents the loss of the region proposal network module; represents the bounding box regression loss.

[0017] In combination with the first aspect, in a possible implementation manner, the calculating the parameter gradients of the local model according to the detection loss values corresponding to the detection target boxes includes:

[0018] Use the local model to obtain the gradient set corresponding to the 3×3 convolutional kernel of the detection loss value;

[0019] Reshape the gradient set in terms of dimensions to obtain the parameter gradients.

[0020] In combination with the first aspect, in a possible implementation, the principal component analysis of the parameter gradient to obtain the principal gradients corresponding to each local model includes:

[0021] Retain the maximum variance value corresponding to the convolution kernel of each element in the parameter gradient, and set the variance values less than the maximum variance value to zero to obtain the principal gradients corresponding to each local model.

[0022] In combination with the first aspect, in a possible implementation, generating the adversarial samples corresponding to each local model according to the principal gradients corresponding to each local model and the images to be detected corresponding to each local model by the fast gradient sign method includes:

[0023] Generate the training sample loss value corresponding to the image to be detected according to the image to be detected, the parameters corresponding to the local model, and the principal gradients corresponding to the local model;

[0024] Generate gradient perturbations according to the training sample loss value;

[0025] Use the Sign function to convert the gradient perturbations into adversarial noises, and add the adversarial noises to the image to be detected, thereby generating the adversarial samples corresponding to each local model.

[0026] In combination with the first aspect, in a possible implementation, the generation formula of the adversarial sample is expressed as:

[0027] I adv = x + εSign(g);

[0028] where I adv represents the adversarial sample; x represents the image to be detected; ε represents the step size parameter; Sign(·) represents the Sign function; g represents the gradient perturbation.

[0029] In combination with the first aspect, in a possible implementation, different first thresholds correspond to different clients.

[0030] In a second aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the gradient decomposition method for federated domain adaptive object detection are implemented.

[0031] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the gradient decomposition method for federated domain adaptive object detection are implemented.

[0032] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0033] (1) By adopting the federated gradient decomposition method, the present invention decomposes the gradients from the clients and extracts only the principal components to the server, which helps to aggregate the most discriminative gradient knowledge from different domains;

[0034] (2) Through the adversarial attack based on the principal gradient, the present invention further improves the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flowchart of the steps of the gradient decomposition method for federated domain adaptive object detection provided by an embodiment of the present invention;

[0036] Figure 2 is an overall flowchart of the federated gradient decomposition proposed by the present invention provided by an embodiment of the present invention;

[0037] Figure 3 is a visualization schematic diagram of a specific embodiment of a rainy dusk and night scene provided by an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of the generation of adversarial samples provided by an embodiment of the present invention;

[0039] Figure 5 is a detection result diagram of a foggy day scene in a specific embodiment provided by an embodiment of the present invention;

[0040] Figure 6 is a schematic diagram of the gradient entanglement problem in federated domain generalization object detection provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0042] The present invention provides a gradient decomposition method for federated domain adaptive object detection, as Figure 1 shown, the method includes:

[0043] S1, using the local model in the client to collect multiple object detection frames of the image to be detected, and respectively calculating the detection loss values of the multiple object detection frames; wherein, the images to be detected of different clients are different.

[0044] Specifically, the calculation formula of the detection loss value is:

[0045]

[0046] Among them, represents the classification loss; represents the loss of the region candidate network module; represents the bounding box regression loss.

[0047] S2. Calculate the parameter gradients of the local model according to the detection loss values corresponding to the detected target boxes, and perform principal component analysis on the parameter gradients to obtain the main gradients corresponding to each local model;

[0048] Specifically, calculating the parameter gradients in the local model according to the detection loss values corresponding to each local model includes:

[0049] (1) Use the local model to obtain the gradient set corresponding to the 3×3 convolution kernel of the detection loss value;

[0050] (2) Reshape the gradient set in terms of dimensions to obtain the parameter gradients.

[0051] Here, performing principal component analysis on the parameter gradients to obtain the main gradients corresponding to each local model includes: retaining the maximum variance value corresponding to the convolution kernel of each element in the parameter gradients, and setting the variance values less than the maximum variance value to zero to obtain the main gradients corresponding to each local model.

[0052] Exemplarily, use to calculate the gradients of the parameters in the local model. Usually, since the images faced by the underlying convolutional layers are much larger than the size of the convolution kernels, the features obtained through 7×7 convolutions are mostly the original / basic features in the images, while the images faced by the top convolutional layers are already close to the size of the convolution kernels, and the features obtained through 3×3 convolutions are more abstract semantic and concept-related features.

[0053] To effectively utilize the gradients related to semantic features in the local model for subsequent operations, collect the gradients corresponding to all common 3×3 convolution kernels to obtain a gradient set where g represents the corresponding gradient, L is the number of layers with 3×3 convolution kernels, and d1, d2, d L represent the number of channels of the corresponding 3×3 convolution kernels. To ensure the effective execution of subsequent data processing, G is reshaped into and perform PCA on the dimension corresponding to the convolution kernel in each element of G, that is, the dimension reshaped into 9, that is, g i = UΣV T where, is the corresponding variance matrix. Retain the maximum variance value and set the other variance values to zero.

[0054] Through this operation, a new gradient can be obtained, which is expressed as follows:

[0055]

[0056] where \(i = 1, 2, \cdots, L\), and \(\sigma_1\) is the maximum variance value.

[0057] The advantage of this operation is that the principal component corresponding to the maximum variance value usually retains rich information of the original gradient. At the same time, this operation reduces the influence of other components on the current gradient, helps to reduce the negative impact of gradient entanglement, and promotes the optimization of the model towards the generalization direction.

[0058] S3. According to the principal gradients corresponding to each local model and the images to be detected corresponding to each local model, generate adversarial samples corresponding to each local model through the Fast Gradient Sign Method.

[0059] Specifically, generating adversarial samples corresponding to each local model according to the principal gradients corresponding to each local model and the images to be detected corresponding to each local model through the Fast Gradient Sign Method includes:

[0060] (1) Generate the loss value of the training sample corresponding to the image to be detected according to the image to be detected, the parameters corresponding to the local model, and the principal gradient corresponding to the local model;

[0061] (2) Generate gradient perturbation according to the loss value of the training sample;

[0062] (3) Use the Sign function to convert the gradient perturbation into adversarial noise, and add the adversarial noise to the image to be detected, thereby generating adversarial samples corresponding to each local model.

[0063] Exemplarily, the Adversarial Attack (AA) method based on PCA is adopted. Adversarial attack is an attack method against machine learning models. By adding carefully designed tiny perturbations to the input data, the local model generates changed outputs.

[0064] After obtaining the result of gradient decomposition, use the new gradient to generate adversarial samples.

[0065] Then, use these adversarial samples for adversarial training to further improve the generalization ability of the model. Specifically, on the basis of the gradient decomposition result \(\{g_1, g_2, \cdots, g\}\) L}, the Fast Gradient Sign Method (FGSM) is used. FGSM is a method for generating adversarial samples by calculating the gradient of the input data on the loss function and adding perturbations along the gradient direction.

[0066] The training using FGSM to generate adversarial samples generally consists of three steps:

[0067] (1) Generate the training sample loss value corresponding to the image to be detected according to the image to be detected, the parameters corresponding to the local model, and the main gradient corresponding to the local model;

[0068] (2) Generate perturbations using the obtained loss and the gradients generated by the current model parameters combined with the model training of this time;

[0069] (3) Use the Sign function to convert the generated perturbations into adversarial noise and directly add it to the image to be detected, thereby generating adversarial samples.

[0070] Therefore, the formula for generating adversarial samples is as follows:

[0071] I adv = x + εSign(g) (3)

[0072]

[0073] Among them, x represents the original input image, Sign(g) is the adversarial noise generated based on the current input x, where g is the gradient perturbation, θ is the model parameter, y is the original input label, J is the loss value of this training, is the gradient of the model for this training, ε is the step size parameter, and I adv represents the corresponding adversarial sample. Since the adversarial noise does not modify the visual content of the visualization, the adversarial sample and the image to be detected are jointly input into the model to perform adversarial training. Through this operation, the generalization ability of the model can be further improved.

[0074] S4. Calculate new gradient parameters according to the adversarial samples corresponding to each local model, and perform principal component analysis on the new gradient parameters to obtain the main gradient of each local model in the current round.

[0075] S5. Input the main gradients corresponding to each local model to the central server for aggregation to obtain an aggregation result. Update the global model according to the aggregation result, and send the updated global model to each client for updating the local models in each client.

[0076] Specifically, finally, the gradients calculated based on PCA are uploaded to the server for federated averaging operation.

[0077] S6. Determine whether the main gradient of the current round in each local model is greater than the first threshold. If so, execute S7; if not, execute S3;

[0078] Here, different clients can correspond to different first thresholds.

[0079] Exemplarily, as the training of the local model in the client progresses, the performance of each client on its respective validation set is verified simultaneously. If the performance of a certain client model reaches the expected state on the validation set, the trainer of this client will trigger an instruction to stop training and send a message to the central server to stop the gradient upload of this client. Subsequently, the local client stops model training, and at the same time, the central server pauses the weight download of the trainer of this client and no longer uses any gradients uploaded by this client subsequently for gradient aggregation. Throughout the training process, the communication between each client and the central server will remain connected.

[0080] S7. Upload the main gradients of each local model in the current round to the central server for aggregation to obtain the aggregation result of the current round, and obtain the target model based on the aggregation result of the current round.

[0081] In a specific embodiment provided by the present invention, the problem setting of federated domain generalization object detection is first introduced, and then the gradient entanglement problem in the federated training process is introduced.

[0082] Problem setting of federated domain generalization object detection. In federated domain generalization object detection, (x, y) is represented as the joint space of an image and an annotation, where y includes the label annotation y c and the bounding box position annotation y b . S = {S 1 , S 2 ,..., S K} is defined as a set of K distributed source domains, and each domain contains a data and annotation pair The goal of federated domain generalization object detection is to learn a detector f θ : x → y, so that it can effectively generalize to a completely unseen test domain and have excellent performance.

[0083] Standard federated training involves communication between a central server and K clients. Specifically, first, the detection model in each client is trained based on the data in the current domain, and then the training gradients from all clients are uploaded to the server to perform an aggregation operation. Finally, the aggregated gradients are sent back to each client to update the current client model.

[0084] Since each client only has one training domain, federated domain generalization object detection faces two important challenges: First, multi-source data is distributedly stored, which causes the models in each client to be unable to effectively learn generalized representations. Second, each local model has a different convergence speed, which affects the generalization performance of the aggregated model. Therefore, improving the generalization ability of detectors based on distributed datasets is crucial for federated domain generalization object detection.

[0085] Gradient entanglement during the aggregation process. During the gradient aggregation process, Federated Averaging (FedAvg) is the most popular algorithm. It aggregates the gradients of local parameters proportional to the size of each dataset to update each client model, that is:

[0086]

[0087] where, represents the aggregated gradient.

[0088] As Figure 6 shown, due to the large domain gap between client datasets, the gradients of clients calculated based on different datasets have different optimization directions. The FedAvg operation may cause the aggregated gradient to deviate from the expected generalization direction. At this time, it is considered that the aggregated gradient is entangled with the gradients of all clients in different directions, weakening the generalization ability of the model. Therefore, reducing the negative impact of gradient entanglement is a proper solution to improve the generalization ability of the model.

[0089] As Figure 2 shown, there is a central server and K clients in the federated framework. The training dataset of each client does not repeat with other clients. The clients are used to train local object detection models and provide parameters and gradients containing the features of the local datasets. The central server collects the parameters and gradients from the clients and sends the aggregated gradients back to each client to update the current client model.

[0090] As Figure 2 shown, the present invention first determines corresponding training samples for different clients. When calculating gradients for the first time, the training samples are evenly divided into multiple rounds, and the number of samples in each round is the same. The training samples of one round are input into the local model of the client. The local model determines the object detection boxes for the training samples and calculates the detection loss values of multiple object detection boxes respectively. Then, according to the detection loss values corresponding to each local model, the parameter gradients in the local model are calculated, and the principal component analysis is performed on the parameter gradients using the principal component analysis method to obtain the principal gradients corresponding to each local model;

[0091] At the local client, the principal gradient and the fast gradient sign method are used to add gradient perturbations (noise) along the gradient direction to generate adversarial samples; the local model is trained according to the adversarial samples to obtain a new round of principal gradients;

[0092] On the central server side, the global model is updated using the uploaded main gradient, and the global model is sent to the client to update the local model of the client; until the main gradient of the local model is greater than the first threshold, a training stop instruction for the local model is triggered, and a message for the client to stop gradient upload is sent to the central server. Subsequently, the local client stops model training, and at the same time, the central server suspends the download of the model weights of this client and no longer uses any gradients uploaded by this client subsequently for gradient fusion. Throughout the training process, the communication from each client to the central server remains connected.

[0093] As Figure 3 shown, the first and fourth columns are the object detection results of the method of the present invention, the second and fifth columns are the feature visualization results of the ordinary method, and the third and sixth columns are the feature visualization results of the method of the present invention.

[0094] As Figure 4 Example diagram of adversarial attack. The first column is the image to be detected, the second column is the adversarial noise generated based on the main gradient, and the third column is the adversarial sample generated after adding the adversarial noise: that is, the adversarial sample generated by superimposing the adversarial noise on the image to be detected in formula (3), corresponding to Figure 2 the image + noise step in

[0095] As Figure 5 shown, the detection result diagram of the foggy scene during the day. The first and second rows respectively show the detection results of Faster R-CNN and the method of the present invention. It can be seen that compared with Faster R-CNN, the method of the present invention can detect objects more accurately.

[0096] The federated domain generalization object detection method proposed by the present invention can effectively improve the object detection performance of the model in the unknown domain, and proposes a federated gradient decomposition method to decompose the gradients from the clients and only extract the main components to the server, which helps to summarize the most discriminative gradient knowledge from different domains; it also uses adversarial attacks based on the main gradient to further improve the generalization ability of the model.

[0097] The present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the gradient decomposition method for federated domain adaptive object detection.

[0098] The present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps of the gradient decomposition method for federated domain adaptive object detection.

[0099] The various embodiments in this specification are described in a progressive manner. For the same or similar parts among the various embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. All or part of the present invention can be used in many general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on.

[0100] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting the present invention; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A gradient decomposition method for federated domain adaptive object detection, characterized in that: include: S1, using a local model in the client to collect multiple target detection frames of the image to be detected, and respectively calculating the detection loss values ​​of the multiple target detection frames; wherein the images to be detected of different clients are different; S2, calculating the parameter gradient of the local model according to the detection loss value corresponding to the detection target box, and performing principal component analysis on the parameter gradient to obtain the main gradient corresponding to each local model, specifically performing principal component analysis on the parameter gradient to obtain the main gradient corresponding to each local model, including: retaining the maximum variance value corresponding to the convolution kernel of each element in the parameter gradient, and setting the variance value less than the maximum variance value to zero, to obtain the main gradient corresponding to each local model; S3, according to the main gradient corresponding to each local model and the image to be detected corresponding to each local model, generate the adversarial sample corresponding to each local model by the fast gradient sign method; specifically, according to the main gradient corresponding to each local model and the image to be detected corresponding to each local model, generate the adversarial sample corresponding to each local model by the fast gradient sign method, including: generating the training sample loss value corresponding to the image to be detected according to the image to be detected, the parameters corresponding to the local model and the main gradient corresponding to the local model; generating the gradient perturbation according to the training sample loss value; converting the gradient perturbation into adversarial noise by using the Sign function, and adding the adversarial noise to the image to be detected, so as to generate the adversarial sample corresponding to each local model; wherein the generation formula of the adversarial sample is expressed as: I adv =x+εSign(g); Among them, I adv represents adversarial samples; x represents the image to be detected; ε represents the step size parameter; Sign(·) represents the Sign function; g represents the gradient perturbation; S4, calculate new gradient parameters according to the adversarial samples corresponding to each local model, and perform principal component analysis on the new gradient parameters to obtain the main gradient of each local model in the current round; S5: Input the main gradient of the current round of each local model to the central server for aggregation to obtain the aggregation result, update the global model according to the aggregation result, and send the updated global model to each client to update the local model in each client; S6, determining whether the main gradient of the current round in each local model is greater than the first threshold, if so, executing S7, if not, returning to S3; S7, upload the main gradient of the current round of each local model to the central server for aggregation, obtain the aggregation result of the current round, and obtain the target model according to the aggregation result of the current round.

2. The gradient decomposition method for federated domain adaptive object detection according to claim 1, characterized in that: The calculation formula of the detection loss value is: in, represents the classification loss; represents the loss of the region proposal network module; represents the bounding box regression loss; Represents the detection loss value.

3. The gradient decomposition method for federated domain adaptive object detection according to claim 1, characterized in that: The calculating the parameter gradient of the local model according to the detection loss value corresponding to the detection target frame includes: Using the local model to obtain a gradient set corresponding to the 3*3 convolution kernel of the detection loss value; The gradient set is reshaped in dimension to obtain the parameter gradient.

4. The gradient decomposition method for federated domain adaptive object detection according to claim 1, characterized in that: Different clients correspond to different first thresholds.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images