Gradient decomposition method for federated domain adaptive target detection and electronic equipment

Abstract

The invention discloses a gradient decomposition method for federated domain adaptive target detection and electronic equipment, relates to the technical field of federated learning, and solves the problem of gradient entanglement in a federated domain generalization target detection task in the prior art. The method comprises the following steps: respectively inputting different images to be detected into different clients, acquiring a plurality of target detection frames of the images to be detected according to local models in the clients, and respectively calculating detection loss values of the plurality of target detection frames; calculating a parameter gradient in the local model according to the detection loss value corresponding to each local model, decomposing the gradient from each client by using principal component analysis, only transmitting the main gradient to a server for further aggregation, and obtaining a target model according to an aggregation result; on the basis of the decomposition gradient, a special antagonistic attack is introduced to synthesize an antagonistic sample, and the generalization ability is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and in particular to a gradient decomposition method and electronic device for federated domain adaptive target detection. Background Art

[0002] As an extension of domain generalization, methods for domain generalization object detection can be roughly divided into single-domain data optimization and multi-domain data integration. For single-domain data optimization, domain-invariant representations are extracted from input visual features without domain labels through a loop decomposition module. For multi-domain data integration, multiple data domain features are integrated through data augmentation to enhance the domain diversity of small datasets; linear transformation matrices are introduced in the training stage to transform multiple domain information to help the model learn different domain information and overcome the domain gap of a single dataset.

[0003] Federated Learning Currently, the most widely used methods are FedAvg, FedProx, FedBN, and FedProto. However, these methods perform poorly in domain generalization tasks. Therefore, some researchers have proposed a federated domain generalization task. In the federated domain generalization task, mainstream methods improve the generalization performance by designing different aggregation schemes. On the one hand, by adding a reference distribution to align the data distribution from different clients, the generalization performance of the model on unknown domains is improved; on the other hand, the continuous frequency domain interpolation method is introduced into the federated domain generalization, or a domain classifier is introduced on the basis of the traditional federated learning method, which improves the performance of the domain invariant feature extractor in the model by improving the model's ability to distinguish domain information.

[0004] In the prior art, generalized object detection in federated domains suffers from gradient entanglement. On the one hand, this entanglement is necessary for the server to learn from the client, so that it can better generalize to unknown domains. On the other hand, these heterogeneous gradients usually contain different or even contradictory components, which can lead to deviations from the expected optimization direction, resulting in poor generalization performance. These two aspects of gradient entanglement constitute the dilemma of generalization. Summary of the invention

[0005] The present invention solves the gradient entanglement problem existing in the prior art for generalized target detection tasks in federal domains by providing a gradient decomposition method for adaptive target detection in federal domains. It introduces a special adversarial attack to synthesize adversarial samples on the basis of decomposing the gradient, thereby further improving the generalization ability.

[0006] In a first aspect, the present invention provides a gradient decomposition method for federated domain adaptive object detection, the method comprising:

[0007] S1, using the local model in the client to collect multiple target detection frames of the image to be detected, and respectively calculating the detection loss values ​​of the multiple target detection frames; wherein the images to be detected of different clients are different;

[0008] S2, calculating the parameter gradient of the local model according to the detection loss value corresponding to the detection target box, and performing principal component analysis on the parameter gradient to obtain the main gradient corresponding to each local model;

[0009] S3, generating adversarial samples corresponding to each local model by a fast gradient sign method according to the main gradient corresponding to each local model and the image to be detected corresponding to each local model;

[0010] S4, calculate new gradient parameters according to the adversarial samples corresponding to each local model, and perform principal component analysis on the new gradient parameters to obtain the main gradient of each local model in the current round;

[0011] S5: Input the main gradient of the current round of each local model to the central server for aggregation to obtain the aggregation result, update the global model according to the aggregation result, and send the updated global model to each client to update the local model in each client;

[0012] S6, determining whether the main gradient of the current round in each local model is greater than the first threshold, if so, executing S7, if not, executing S3;

[0013] S7, upload the main gradient of the current round of each local model to the central server for aggregation, obtain the aggregation result of the current round, and obtain the target model according to the aggregation result of the current round.

[0014] In combination with the first aspect, in a possible implementation manner, the calculation formula of the detection loss value is:

[0015]

[0016] in, represents the classification loss; represents the loss of the region proposal network module; represents the bounding box regression loss.

[0017] In combination with the first aspect, in a possible implementation manner, calculating the parameter gradient of the local model according to the detection loss value corresponding to the detected target box includes:

[0018] Using the local model to obtain the gradient set corresponding to the 3*3 convolution kernel of the detection loss value;

[0019] The gradient set is reshaped in dimension to obtain the parameter gradient.

[0020] In combination with the first aspect, in a possible implementation manner, performing principal component analysis on the parameter gradient to obtain the principal gradient corresponding to each local model includes:

[0021] The maximum variance value corresponding to the convolution kernel of each element in the parameter gradient is retained, and the variance values ​​smaller than the maximum variance value are set to zero, so as to obtain the main gradient corresponding to each local model.

[0022] In combination with the first aspect, in a possible implementation manner, generating adversarial samples corresponding to each local model by a fast gradient sign method according to the main gradient corresponding to each local model and the image to be detected corresponding to each local model, includes:

[0023] Generate a training sample loss value corresponding to the image to be detected according to the image to be detected, the parameters corresponding to the local model and the main gradient corresponding to the local model;

[0024] Generate a gradient perturbation according to the loss value of the training sample;

[0025] The gradient perturbation is converted into adversarial noise using the Sign function, and the adversarial noise is added to the image to be detected, thereby generating adversarial samples corresponding to each local model.

[0026] In combination with the first aspect, in a possible implementation, the generation formula of the adversarial sample is expressed as:

[0027] I adv =x+εSign(g);

[0028] Among them, I adv represents adversarial samples; x represents the image to be detected; ε represents the step size parameter; Sign(·) represents the Sign function; g represents the gradient perturbation.

[0029] In combination with the first aspect, in a possible implementation manner, different clients correspond to different first thresholds.

[0030] In a second aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a gradient decomposition method for federated domain adaptive target detection are implemented.

[0031] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a gradient decomposition method for federated domain adaptive target detection.

[0032] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0033] (1) The present invention adopts a federated gradient decomposition method to decompose the gradient from the client and extract only the principal component to the server, which helps to aggregate the most discriminative gradient knowledge from different domains;

[0034] (2) The present invention further improves the generalization ability of the model through adversarial attacks based on principal gradients. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of the steps of a gradient decomposition method for federated domain adaptive object detection provided by an embodiment of the present invention;

[0036] Figure 2 The overall flow chart of the federated gradient decomposition proposed by the present invention provided in the embodiment of the present invention;

[0037] Figure 3 A schematic diagram of visualization of a rainy dusk and nighttime scene in a specific embodiment provided in an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of generating adversarial samples provided by an embodiment of the present invention;

[0039] Figure 5 A detection result diagram of a foggy scene during the day in a specific embodiment provided in an embodiment of the present invention;

[0040] Figure 6 Schematic diagram of the gradient entanglement problem in generalized object detection in the federated domain provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The present invention provides a gradient decomposition method for federated domain adaptive target detection, such as Figure 1 As shown, the method includes:

[0043] S1, using the local model in the client to collect multiple target detection frames of the image to be detected, and respectively calculate the detection loss values ​​of the multiple target detection frames; wherein the images to be detected of different clients are different.

[0044] Specifically, the calculation formula for the detection loss value is:

[0045]

[0046] in, represents the classification loss; represents the loss of the region proposal network module; represents the bounding box regression loss.

[0047] S2, calculate the parameter gradient of the local model according to the detection loss value corresponding to the detection target box, and perform principal component analysis on the parameter gradient to obtain the main gradient corresponding to each local model;

[0048] Specifically, the parameter gradients in the local models are calculated according to the detection loss values ​​corresponding to the local models, including:

[0049] (1) Use the local model to obtain the gradient set corresponding to the 3*3 convolution kernel of the detection loss value;

[0050] (2) Reshape the gradient set dimensionally to obtain the parameter gradient.

[0051] Here, principal component analysis is performed on the parameter gradient to obtain the main gradient corresponding to each local model, including: retaining the maximum variance value corresponding to the convolution kernel of each element in the parameter gradient, and setting the variance value smaller than the maximum variance value to zero, to obtain the main gradient corresponding to each local model.

[0052] For example, use To calculate the gradient of the parameters in the local model. Usually, because the image faced by the bottom convolution layer is much larger than the size of the convolution kernel, the features obtained by 7×7 convolution are mostly the original / basic features in the image, while the image faced by the top convolution layer is close to the size of the convolution kernel, and the features obtained by 3×3 convolution are more abstract semantic and concept-related features.

[0053] In order to effectively use the gradients related to the semantic features in the local model for subsequent operations, the gradients corresponding to all commonly used 3×3 convolution kernels are collected to obtain a gradient set Among them, g represents the corresponding gradient, L is the number of layers with 3×3 convolution kernels, d1, d2, d L Indicates the number of channels of the corresponding 3×3 convolution kernel. In order to ensure the effective execution of subsequent data processing, G is reshaped into And perform PCA on the dimension corresponding to the convolution kernel in each element of G, which is reshaped into a dimension of 9, that is, g i =UΣV T ,in, is the corresponding variance matrix. The largest variance value is retained and the others are set to zero.

[0054] Through this operation, a new gradient can be obtained, which is expressed as follows:

[0055]

[0056] Among them, i=1,2,...,L, σ1 is the maximum variance value.

[0057] The benefit of this operation is that the principal component corresponding to the maximum variance value usually retains rich information of the original gradient. At the same time, this operation reduces the influence of other components on the current gradient, helps reduce the negative impact of gradient entanglement, and promotes the optimization of the model towards generalization.

[0058] S3, according to the main gradient corresponding to each local model and the image to be detected corresponding to each local model, generate the adversarial sample corresponding to each local model through the fast gradient sign method.

[0059] Specifically, according to the main gradient corresponding to each local model and the image to be detected corresponding to each local model, the adversarial sample corresponding to each local model is generated by the fast gradient sign method, including:

[0060] (1) Generate a training sample loss value corresponding to the image to be detected based on the image to be detected, the parameters corresponding to the local model, and the main gradient corresponding to the local model;

[0061] (2) Generate gradient perturbation based on the loss value of the training sample;

[0062] (3) The Sign function is used to convert the gradient perturbation into adversarial noise, and the adversarial noise is added to the image to be detected, thereby generating adversarial samples corresponding to each local model.

[0063] Exemplarily, a PCA-based adversarial attack (AA) method was used. Adversarial attack is an attack method against machine learning models, which causes the local model to produce changed outputs by adding carefully designed tiny perturbations to the input data.

[0064] After obtaining the results of gradient decomposition, the new gradient is used to generate adversarial samples.

[0065] Then, we use these adversarial examples for adversarial training to further improve the generalization ability of the model. Specifically, in the gradient decomposition results {g1, g2, ..., g L}, the Fast Gradient Sign Method (FGSM) is used, where FGSM is a method for generating adversarial samples by calculating the gradient of the input data on the loss function and adding perturbations along the gradient direction.

[0066] Generating adversarial samples using FGSM for training is generally divided into three steps:

[0067] (1) Generate a training sample loss value corresponding to the image to be detected based on the image to be detected, the parameters corresponding to the local model, and the main gradient corresponding to the local model;

[0068] (2) Generate perturbations using the obtained loss and current model parameters combined with the gradients generated by this model training;

[0069] (3) The generated perturbation is converted into adversarial noise using the Sign function and directly added to the image to be detected, thereby generating adversarial samples.

[0070] Therefore, the formula for generating adversarial samples is as follows:

[0071] I adv =x+εSign(g) (3)

[0072]

[0073] Where x represents the original input image, Sign(g) is the adversarial noise generated based on the current input x, g is the gradient perturbation, θ is the model parameter, y is the original input label, and J is the loss value of this training. is the gradient of the model for this training, ε is the step size parameter, I adv Denotes the corresponding adversarial sample. Since adversarial noise does not modify the visual content of the visualization, the adversarial sample and the image to be detected are jointly input into the model to perform adversarial training. Through this operation, the generalization ability of the model can be further improved.

[0074] S4: Calculate new gradient parameters according to the adversarial samples corresponding to each local model, and perform principal component analysis on the new gradient parameters to obtain the main gradient of each local model in the current round.

[0075] S5, input the main gradient corresponding to each local model to the central server for aggregation, obtain the aggregation result, update the global model according to the aggregation result, and send the updated global model to each client to update the local model in each client.

[0076] Specifically, the gradient calculated based on PCA is finally uploaded to the server for federated averaging.

[0077] S6, determining whether the main gradient of the current round in each local model is greater than the first threshold, if so, executing S7, if not, executing S3;

[0078] Here, different clients may correspond to different first thresholds.

[0079] For example, as the training of the local model in the client proceeds, the performance of each client on its own validation set is verified. If the performance of a client model reaches the expected state on the validation set, the trainer of the client will trigger the instruction to stop training and send a message to the central server that the client stops uploading gradients. Then the local client stops model training, and the central server suspends the download of the trainer weights of the client, and no longer uses any gradients subsequently uploaded by the client for gradient fusion. During the entire training process, the communication between each client and the central server will remain connected.

[0080] S7, upload the main gradient of the current round of each local model to the central server for aggregation, obtain the aggregation result of the current round, and obtain the target model according to the aggregation result of the current round.

[0081] In a specific embodiment provided by the present invention, the problem setting of generalized target detection in the federated domain is first introduced, and then the gradient entanglement problem in the federated training process is introduced.

[0082] Problem setting for generalized object detection in federated domains. In generalized object detection in federated domains, (x, y) is represented as the joint space of images and annotations, where y includes the label annotation y c and bounding box position annotation y b . S={S 1 ,S 2 ,...,S K} is defined as a set of K distributed source domains, each containing data and annotation pairs The goal of federated domain generalization object detection is to learn a detector f using K distributed source domains. θ :x→y, so that it can effectively generalize to a completely unseen test domain with superior performance.

[0083] Standard federated training involves communication between a central server and K clients. Specifically, the detection model in each client is first trained based on the data in the current domain, and then the training gradients from all clients are uploaded to the server to perform an aggregation operation. Finally, the aggregated gradients are sent back to each client to update the current client model.

[0084] Since each client has only one training domain, generalized object detection in the federated domain faces two important challenges: First, multi-source data is stored in a distributed manner, which makes it impossible for the model in each client to effectively learn generalized representations. Second, each local model has a different convergence speed, which affects the generalization performance of the aggregated model. Therefore, improving the generalization ability of detectors based on distributed datasets is crucial for generalized object detection in the federated domain.

[0085] Gradient entanglement during aggregation. During the gradient aggregation process, Federated Averaging (FedAvg) is the most popular algorithm, which aggregates the gradients of local parameters proportional to the size of each dataset to update each client model, namely:

[0086]

[0087] in, represents the aggregation gradient.

[0088] like Figure 6 As shown in the figure, due to the large domain gap between client datasets, the gradients of clients calculated based on different datasets have different optimization directions, and the FedAvg operation may cause the aggregated gradient to deviate from the expected generalization direction. At this time, it is considered that the aggregated gradient is entangled with all the client gradients in different directions, which weakens the generalization ability of the model. Therefore, reducing the negative impact of gradient entanglement is a proper solution to improve the generalization ability of the model.

[0089] like Figure 2 As shown in Figure 1, there is a central server and K clients in the joint framework. The training dataset of each client will not be repeated with other clients. The client is used to train a local object detection model and provide parameters and gradients containing the characteristics of the local dataset. The central server collects parameters and gradients from the clients and sends the aggregated gradients back to each client to update the current client model.

[0090] like Figure 2 As shown, the present invention first determines corresponding training samples for different clients. When performing gradient calculation for the first time, the training samples are evenly divided into multiple rounds, and the number of samples in each round is the same. The training samples of one round are input into the local model of the client. The local model determines the target detection frame of the training samples and calculates the detection loss values ​​of multiple target detection frames respectively. Then, the parameter gradient in the local model is calculated according to the detection loss value corresponding to each local model, and the principal component analysis method is used to perform principal component analysis on the parameter gradient to obtain the main gradient corresponding to each local model.

[0091] On the local client, the main gradient and fast gradient sign method are used to add gradient perturbations (noise) along the gradient direction to generate adversarial samples. The local model is trained based on the adversarial samples to obtain a new round of main gradients.

[0092] On the central server side, the global model is updated using the uploaded master gradient, and the global model is sent to the client to update the client's local model; until the master gradient of the local model is greater than the first threshold, the training stop instruction of the local model is triggered, and a message is sent to the central server that the client stops uploading gradients. Then the local client stops model training, and the central server suspends the download of the client's model weights, and no longer uses any subsequent gradients uploaded by the client for gradient fusion. During the entire training process, the communication between each client and the central server will remain connected.

[0093] like Figure 3 As shown, the first and fourth columns are target detection results of the method of the present invention, the second and fifth columns are feature visualization results of the common method, and the third and sixth columns are feature visualization results of the method of the present invention.

[0094] like Figure 4 Adversarial attack example diagram. The first column is the image to be tested, the second column is the adversarial noise generated based on the main gradient, and the third column is the adversarial sample generated after adding the adversarial noise: that is, the adversarial sample generated after adding the adversarial noise to the image to be tested in formula (3) corresponds to Figure 2 Medium image + noise step.

[0095] like Figure 5 As shown in FIG. 1 , the detection result diagram of a foggy scene during the day. The first and second rows show the detection results of Faster R-CNN and the method of the present invention, respectively. It can be seen that compared with Faster R-CNN, the method of the present invention can detect objects more accurately.

[0096] The federated domain generalized target detection method proposed in the present invention can effectively improve the target detection performance of the model in unknown domains, and proposes a federated gradient decomposition method to decompose the gradient from the client and extract only the principal component to the server, which helps to summarize the most discriminative gradient knowledge from different domains; it also uses adversarial attacks based on principal gradients to further improve the generalization ability of the model.

[0097] The present invention provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, steps of a gradient decomposition method for federated domain adaptive target detection are implemented.

[0098] The present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of a gradient decomposition method for federated domain adaptive target detection are implemented.

[0099] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of the present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0100] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the present invention.

Claims

1. A gradient decomposition method for federated domain adaptive object detection, characterized in that: include: S1, using the local model in the client to collect multiple target detection frames of the image to be detected, and respectively calculating the detection loss values ​​of the multiple target detection frames; wherein the images to be detected of different clients are different; S2, calculating the parameter gradient of the local model according to the detection loss value corresponding to the detection target box, and performing principal component analysis on the parameter gradient to obtain the main gradient corresponding to each local model; S3, generating adversarial samples corresponding to each local model by a fast gradient sign method according to the main gradient corresponding to each local model and the image to be detected corresponding to each local model; S4, calculate new gradient parameters according to the adversarial samples corresponding to each local model, and perform principal component analysis on the new gradient parameters to obtain the main gradient of each local model in the current round; S5: Input the main gradient of the current round of each local model to the central server for aggregation to obtain the aggregation result, update the global model according to the aggregation result, and send the updated global model to each client to update the local model in each client; S6, determining whether the main gradient of the current round in each local model is greater than the first threshold, if so, executing S7, if not, returning to S3; S7, upload the main gradient of the current round of each local model to the central server for aggregation, obtain the aggregation result of the current round, and obtain the target model according to the aggregation result of the current round.

2. The gradient decomposition method for federated domain adaptive object detection according to claim 1, characterized in that: The calculation formula of the detection loss value is: in, represents the classification loss; represents the loss of the region proposal network module; represents the bounding box regression loss.

3. The gradient decomposition method for federated domain adaptive object detection according to claim 1, characterized in that: The calculating the parameter gradient of the local model according to the detection loss value corresponding to the detection target frame includes: Using the local model to obtain a gradient set corresponding to the 3*3 convolution kernel of the detection loss value; The gradient set is reshaped in dimension to obtain the parameter gradient.

4. The gradient decomposition method for federated domain adaptive object detection according to claim 1, characterized in that: The performing principal component analysis on the parameter gradient to obtain the principal gradient corresponding to each local model includes: The maximum variance value corresponding to the convolution kernel of each element in the parameter gradient is retained, and the variance values ​​smaller than the maximum variance value are set to zero, so as to obtain the main gradient corresponding to each local model.

5. The gradient decomposition method for federated domain adaptive object detection according to claim 1, characterized in that: The method of generating adversarial samples corresponding to each local model by a fast gradient sign method according to the main gradient corresponding to each local model and the image to be detected corresponding to each local model includes: Generate a training sample loss value corresponding to the image to be detected according to the image to be detected, the parameters corresponding to the local model and the main gradient corresponding to the local model; Generate a gradient perturbation according to the loss value of the training sample; The gradient perturbation is converted into adversarial noise using the Sign function, and the adversarial noise is added to the image to be detected, thereby generating adversarial samples corresponding to each local model.

6. The gradient decomposition method for federated domain adaptive object detection according to claim 5, characterized in that: The generation formula of the adversarial sample is expressed as: I adv =x+εSign(g); Among them, I adv represents the adversarial sample; x represents the image to be detected; ε represents the step size parameter; Sign(·) represents the Sign function; g represents the gradient perturbation.

7. The gradient decomposition method for federated domain adaptive object detection according to claim 1, characterized in that: Different clients correspond to different first thresholds.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images