A domain generalization method for UAV target detection based on frequency domain decoupling

Abstract

The present invention discloses a domain-generalized drone target detection method based on frequency domain decoupling, comprising: 1. constructing a drone target detection dataset, including data acquisition and preprocessing; 2. constructing a domain-generalized drone target detection network based on frequency domain decoupling; 3. using the network to process and calculate input images to obtain a total loss function; and 4. iteratively optimizing the network based on the total loss function. This invention fully considers the characteristics of drone target detection scenarios, namely, decoupling the global dependencies of target-related features, and combines the advantages of the frequency domain obeying global modeling. The proposed frequency domain decoupling network decouples target-related features in drone images from the perspective of global information, effectively improving the generalization of the drone target detection network and significantly enhancing the detection accuracy of the drone target detection network in complex and changing real-world scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of UAV target detection, and in particular relates to domain generalized UAV target detection. A domain generalized UAV target detection method based on frequency domain decoupling is proposed. Background Art

[0002] Drone object detection aims to detect objects in drone-captured images. However, the maneuverability and flexibility of drones lead to unpredictable operating environments. If a drone object detection network trained on a given dataset (the source domain) is applied to an unseen environment (the target domain), its detection performance inevitably degrades due to the effects of domain shift. Therefore, improving the generalization of drone object detection networks is crucial.

[0003] There are currently two main methods that can solve the above dilemma: domain adaptation and domain generalization. (1) Domain adaptation aims to solve the domain migration problem by learning domain-invariant or domain-aligned features between the source and target domains. However, for drone target detection networks with highly unpredictable deployment environments, we cannot guarantee that all target domain data that may be encountered will be collected. Therefore, domain adaptation methods cannot be well applied to drone target detection networks. (2) Domain generalization aims to use data from a single or multiple related and different source domains to learn a model with good generalization performance in the case of domain migration. Most existing domain generalization methods focus on decoupling target-related features through spatial raw convolution to enhance the generalization of the detection network. However, unlike traditional target detection scenarios based on surveillance or other ground-based cameras, the high-speed movement of drones will cause the global appearance of the pictures they take to change rapidly. For drone target detection scenarios where the global appearance will change significantly, decoupling target-related features requires more reliance on global information. The spatial raw convolution decoupling target-related features method used by existing domain generalization methods only emphasizes the attention of local pixels and cannot explore global dependencies well. Therefore, the existing domain generalization methods cannot effectively improve the detection accuracy of drone target detection networks in complex and changeable real-world scenarios. Summary of the Invention

[0004] In order to address the shortcomings of the above-mentioned existing technologies, the present invention proposes a domain generalization UAV target detection method based on frequency domain decoupling, in order to effectively improve the generalization of the UAV target detection network, thereby improving the detection accuracy of the UAV target detection network in complex and changeable real-world scenarios.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] The domain generalization UAV target detection method based on frequency domain decoupling of the present invention is characterized in that it is performed according to the following steps:

[0007] Step 1: Obtain the drone target detection dataset along with the category and bounding box labels.

[0008] Step 1.1: Select the drone image data with category and bounding box labels and perform size normalization and preprocessing to obtain the preprocessed drone image dataset with dimensions H×W×C, denoted as X={x1,x2,...,x i ,...,x n}, where x i Represents the i-th drone image, and the label set of the drone image dataset is in, Represents the i-th drone image x i The category label of the j-th target in, Represents the i-th drone image x i The bounding box label of the j-th object in m i Represents the i-th drone image x i The total number of objects included, i = 1, 2, ..., n, where n is the total number of drone images with category and bounding box labels; H represents the height of the image after preprocessing, W represents the width of the image after preprocessing, and C represents the number of channels;

[0009] Step 2: Construct a domain generalized UAV target detection network based on frequency domain decoupling, which consists of a frequency domain decoupling network and a target detection network F;

[0010] Step 2.1: Build the target detection network F based on the Yolov5 model, which consists of the Backbone module, the Neck module, and the Prediction module.

[0011] The target detection network F is pre-trained using the pre-processed drone image dataset X and its label set T to obtain the pre-trained target detection network

[0012] Step 2.2: Construct a frequency domain decoupling network consisting of two domain-invariant frequency domain filters to be learned and domain-specific frequency domain filters The size of each frequency domain filter is H×W, and the value range of each element in the frequency domain filter is [0,1];

[0013] Step 3: Training of domain generalized UAV target detection network based on frequency domain decoupling;

[0014] Step 4: Use the trained domain-invariant frequency domain filter Filter the amplitude spectrum of the drone image to be tested to obtain the domain-invariant drone image; input the domain-invariant drone image into the trained target detection network Get the final test results.

[0015] The domain generalization UAV target detection method based on frequency domain decoupling described in the present invention is also characterized in that step 3 is performed according to the following steps:

[0016] Step 3.1: For the i-th drone image x i Perform discrete Fourier transform to obtain the frequency domain signal

[0017] Step 3.2: Frequency domain signal Further decomposed into amplitude spectrum and phase spectrum

[0018] Step 3.3: The amplitude spectrum Domain-invariant frequency-domain filters in frequency-domain decoupling networks and domain-specific frequency domain filters Multiply each element by element to get the domain invariant amplitude spectrum and domain-specific amplitude spectra

[0019] Step 3.4: Convert the domain-invariant amplitude spectrum Phase spectrum Synthesize into domain-invariant frequency domain signal Domain-Specific Amplitude Spectrum Phase spectrum Synthesize into domain-specific frequency domain signals

[0020] Step 3.5: Convert the domain-invariant frequency domain signal Perform inverse discrete Fourier transform to obtain domain-invariant image Domain-specific frequency domain signals Perform inverse discrete Fourier transform to obtain domain-specific images

[0021] Step 3.6: Domain-invariant image and domain-specific images Input pre-trained object detection network The Backbone module is used to extract features and output domain-invariant features f i a and domain-specific features f i b ;

[0022] Step 3.7: Based on the i-th drone image x i Bounding box of Domain-invariant images With domain-invariant features f i a Size ratio between domain-specific images With domain-specific features f i b The size ratio between them is used to crop the domain invariant features f i a m in i target and domain-specific features f i b m in i targets, and then use the RoI-Alignment operation to align the target size to obtain the domain-invariant target feature set and domain-specific target feature sets in, Represents the domain-invariant feature f i a The kth domain-invariant target feature in , Represents domain-specific features f i b The kth domain-specific target feature in , k = 1, 2, ..., m i ;

[0023] Use formula (1) to calculate the i-th drone image x i Contrastive loss

[0024]

[0025] In formula (1), sim(·) represents the similarity function, and represent and The number of elements in , τ represents the temperature coefficient of contrast loss; ^ represents the logical AND; is a collection The oth element in is a collection The oth element in Indicates the containing element A collection of Indicates the containing element A collection of is a collection With collection The pth element in the difference set of is a collection With collection The pth element in the difference set of is a collection With collection The union of The qth element in the difference set of is a collection With collection The union of The qth element in the difference set of ;

[0026] Step 3.8: Domain-invariant feature f i a Input pre-trained object detection network In the Neck module and Prediction module, the predicted target category p is obtained class , predict the target bounding box p box And the predicted target confidence p confidence ;

[0027] Use formula (2) to calculate the i-th drone image x i Detection loss

[0028]

[0029] In formula (2), Represents the i-th drone image x i BCE loss for target categories in Represents the i-th drone image x i CIOU loss of the target bounding box in Represents the i-th drone image x i BCE loss for target confidence;

[0030] Step 3.9: Use Equation (3) to establish the total loss function of the domain generalization UAV target detection network

[0031]

[0032] In formula (3), λ is a hyperparameter that balances detection loss and contrast loss in the total loss function;

[0033] Step 3.10: Use the stochastic gradient descent method to learn the two domain-invariant frequency domain filters in the frequency domain decoupling network. and domain-specific frequency domain filters and target detection network Perform optimization and calculate the total loss function Then perform gradient backpropagation until the total loss function is reached Until convergence, the trained domain invariant frequency domain filter is obtained Domain-specific frequency domain filters and target detection network

[0034] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the domain generalization drone target detection method, and the processor is configured to execute the program stored in the memory.

[0035] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the domain generalization UAV target detection method when the computer program is executed by a processor.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The present invention fully considers the characteristics of UAV target detection scenarios, namely, decoupling the global dependencies of target-related features. By adopting the frequency domain decoupling method, the target-related features in UAV images are decoupled from a global perspective, overcoming the problem that existing methods cannot well explore global dependencies, and improving the detection accuracy of UAVs in complex and changeable real-world scenarios.

[0038] 2. The present invention fully considers the characteristics of the UAV target detection task, which includes two stages: target positioning and target classification, and proposes a new contrast loss. By cropping target features according to the target position label and bringing domain-invariant target features and domain-specific target features of the same category closer, it can fully combine the position information and category information of the target in the UAV image, achieve a better decoupling effect of target-related features in the UAV image, and further improve the detection accuracy of UAVs in complex and changeable real-world scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the domain generalization UAV target detection algorithm based on frequency domain decoupling of the present invention;

[0040] Figure 2 This is a visual comparison of the decoupling effects of the present invention and other methods at the feature level. DETAILED DESCRIPTION

[0041] In this embodiment, a domain generalization UAV target detection method based on frequency domain decoupling is proposed. It takes into account the characteristics of UAV target detection scenarios, that is, decoupling the global dependency of target-related features, and combines the advantages of frequency domain obeying global modeling. The generalization of the UAV target detection network is improved through frequency domain decoupling, and a new contrast loss is designed to help the training of the frequency domain decoupling network. Specifically, Figure 1 As shown, the following steps are included:

[0042] Step 1: Obtain the drone target detection dataset along with the category and bounding box labels.

[0043] Step 1.1: Select the drone image data with category and bounding box labels and perform size normalization and preprocessing to obtain the preprocessed drone image dataset with dimensions H×W×C, denoted as X={x1,x2,...,x i ,...,x n}, where x i Represents the i-th drone image, and the label set of the drone image dataset is in, Represents the i-th drone image x i The category label of the j-th target in, Represents the i-th drone image x i The bounding box label of the j-th object in m i Represents the i-th drone image x i The total number of objects included, i = 1, 2, ..., n, where n is the total number of drone images with category and bounding box labels; H represents the height of the image after preprocessing, W represents the width of the image after preprocessing, and C represents the number of channels; during preprocessing, the input drone image data is subjected to corresponding preprocessing processes, including cropping, flipping, rotation, and other operations.

[0044] Step 2: Construct a domain generalized UAV target detection network based on frequency domain decoupling, which consists of a frequency domain decoupling network and a target detection network F;

[0045] Step 2.1: Build the target detection network F based on the Yolov5 model, which consists of the Backbone module, the Neck module, and the Prediction module.

[0046] The target detection network F is pre-trained using the pre-processed drone image dataset X and its label set T to obtain the pre-trained target detection network

[0047] Step 2.2: Construct a frequency domain decoupling network consisting of two domain-invariant frequency domain filters to be learned and domain-specific frequency domain filters The size of each frequency domain filter is H×W, and the value range of each element in the frequency domain filter is [0,1];

[0048] Step 3: Training of domain generalized UAV target detection network based on frequency domain decoupling;

[0049] Step 3.1: For the i-th drone image x i Perform discrete Fourier transform to obtain the frequency domain signal

[0050] Step 3.2: Frequency domain signal Further decomposed into amplitude spectrum and phase spectrum

[0051] Step 3.3: The amplitude spectrum Domain-invariant frequency-domain filters in frequency-domain decoupling networks and domain-specific frequency domain filters Multiply each element by element to get the domain invariant amplitude spectrum and domain-specific amplitude spectra

[0052] Step 3.4: Convert the domain-invariant amplitude spectrum Phase spectrum Synthesize into domain-invariant frequency domain signal Domain-Specific Amplitude Spectrum Phase spectrum Synthesize into domain-specific frequency domain signals

[0053] Step 3.5: Convert the domain-invariant frequency domain signal Perform inverse discrete Fourier transform to obtain domain-invariant image Domain-specific frequency domain signals Perform inverse discrete Fourier transform to obtain domain-specific images

[0054] Step 3.6: Domain-invariant image and domain-specific images Input pre-trained object detection network Feature extraction is performed in the Backbone module, and domain-invariant features are output and domain-specific features

[0055] Step 3.7: Based on the i-th drone image x i Bounding box of Domain-invariant images Domain-invariant features Size ratio between domain-specific images With domain-specific features f i b The size ratio between them is used to crop the domain invariant features f i a m in itarget and domain-specific features f i b m in i targets, and then use the RoI-Alignment operation to align the target size to obtain the domain-invariant target feature set and domain-specific target feature sets in, Represents the domain-invariant feature f i a The kth domain-invariant target feature in , Represents domain-specific features f i b The kth domain-specific target feature in , k = 1, 2, ..., m i ;

[0056] Use formula (1) to calculate the i-th drone image x i Contrastive loss

[0057]

[0058] In formula (1), sim(·) represents the similarity function, and represent and The number of elements in , τ represents the temperature coefficient of contrast loss; ^ represents the logical AND; is a collection The oth element in is a collection The oth element in Indicates the containing element A collection of Indicates the containing element A collection of is a collection With collection The pth element in the difference set of is a collection With collection The pth element in the difference set of is a collection With collection The union of The qth element in the difference set of is a collection With collection The union of The qth element in the difference set of ; In this embodiment, τ is fixed to 0.7; In contrastive loss, for domain-invariant target features, domain-invariant target features from the same category are regarded as positive samples, and all remaining target features are regarded as negative samples; for domain-specific target features, domain-specific target features from the same category are regarded as positive samples, and all remaining target features are regarded as negative samples; By optimizing contrastive loss, positive sample pairs are pulled closer and negative sample pairs are pulled farther away, ensuring a good frequency domain decoupling effect;

[0059] Step 3.8: Domain-invariant feature f i a Input pre-trained object detection network In the Neck module and Prediction module, the predicted target category p is obtained class , predict the target bounding box p box And the predicted target confidence p confidence ;

[0060] Use formula (2) to calculate the i-th drone image x i Detection loss

[0061]

[0062] In formula (2), Represents the i-th drone image x i BCE loss for target categories in Represents the i-th drone image x i CIOU loss of the target bounding box in Represents the i-th drone image x i BCE loss for target confidence;

[0063] Step 3.9: Use Equation (3) to establish the total loss function of the domain generalization UAV target detection network

[0064]

[0065] In formula (3), λ is a hyperparameter that balances detection loss and contrast loss in the total loss function. In this embodiment, λ is fixed to 0.15.

[0066] Step 3.10: Use the stochastic gradient descent method to learn the two domain-invariant frequency domain filters in the frequency domain decoupling network. and domain-specific frequency domain filters and target detection network Perform optimization and calculate the total loss function Then perform gradient backpropagation until the total loss function is reached Until convergence, the trained domain invariant frequency domain filter is obtained Domain-specific frequency domain filters and target detection network

[0067] Step 4: Use the trained domain-invariant frequency domain filter Filter the amplitude spectrum of the drone image to be tested to obtain the domain-invariant drone image; input the domain-invariant drone image into the trained target detection network Get the final test results.

[0068] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0069] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0070] In order to quantitatively evaluate the effect of the present invention, the proposed method is compared with a variety of domain generalization algorithms with the best performance on the UAVDT dataset and the VisDrone2019 dataset, and the AP 50 , AP 75 The three performance indicators, Avg, and AP, are used as evaluation metrics. The UAVDT dataset can be divided into three parts based on weather labels: 23,741 daytime images, 11,489 nighttime images, and 2,492 foggy images. This paper trains on 20,891 of the 23,741 daytime images and tests on the remaining 2,850 daytime images (representing domain migration of scene changes), 11,489 nighttime images (representing domain migration from day to night), and 2,492 foggy images (representing domain migration from day to foggy). The experimental results are shown in Table 1.

[0071] Table 1 is a comparison table of the generalization performance of the present invention and other methods on the UAVDT dataset;

[0072]

[0073] In addition, the present invention was trained on 16,238 daytime images in the VisDrone2019 dataset and tested on 2,850 daytime images (representing the domain migration of scene changes), 11,489 night images (representing the domain migration from day to night), and 2,492 foggy images (representing the domain migration from day to foggy days) in the above-mentioned UAVDT dataset. The experimental results are shown in Table 2.

[0074] Table 2 is a comparison table of the generalization performance of the present invention and other methods on the VisDrone2019 dataset;

[0075]

[0076]

[0077] From the above generalization results, it can be clearly found that the present invention has achieved the best generalization results compared with the current optimal methods JiGen, RSC, StableNet and Single-DGOD. Figure 2 A visual comparison of the decoupling effects of the present invention and the Single-DGOD method at the feature level is shown. The comparison of the decoupling effects shows that the present invention can more effectively decouple domain-invariant features and domain-specific features. For example, Figure 2 The billboard in the first row and the area illuminated by car lights in the second row.

Claims

1. A domain generalization UAV target detection method based on frequency domain decoupling, characterized by: Follow these steps: Step 1: Obtain the drone target detection dataset along with the category and bounding box labels. Step 1.1: Select the drone image data with category and bounding box labels and perform size normalization and preprocessing to obtain the dimension The preprocessed UAV image dataset is denoted as ,in, Indicates the drone images, let the label set of the drone image dataset be ,in, Indicates the drone images Middle The category labels of the targets, Indicates the drone images Middle Bounding box labels for objects, Indicates the drone images The total number of targets included, , is the total number of drone images with category and bounding box labels; Indicates the height of the image after preprocessing, Indicates the width of the image after preprocessing, Indicates the number of channels; Step 2: Construct a domain generalized UAV target detection network based on frequency domain decoupling, which is composed of frequency domain decoupling network and target detection network. composition; Step 2.1: Build a target detection network based on the Yolov5 model , consisting of Backbone module, Neck module and Prediction module; Using the preprocessed drone image dataset and its tag collection The target detection network Perform pre-training to obtain the pre-trained target detection network ; Step 2.2: Construct a frequency domain decoupling network consisting of two domain-invariant frequency domain filters to be learned and domain-specific frequency domain filters The size of each frequency domain filter is , the value range of each element in the frequency domain filter is ; Step 3: Training of domain generalization UAV target detection network based on frequency domain decoupling: The two domain-invariant frequency domain filters to be learned in the frequency domain decoupling network are trained by stochastic gradient descent. and domain-specific frequency domain filters and target detection network Perform optimization and calculate the total loss function Then perform gradient backpropagation until the total loss function is reached Until convergence, the trained domain invariant frequency domain filter is obtained , domain-specific frequency domain filters and target detection network ; Step 4: Use the trained domain-invariant frequency domain filter Filter the amplitude spectrum of the drone image to be tested to obtain the domain-invariant drone image; input the domain-invariant drone image into the trained target detection network , and obtain the final test results.

2. The domain generalization UAV target detection method based on frequency domain decoupling according to claim 1 is characterized in that: The step 3 is performed according to the following steps: Step 3.1, the drone images Perform discrete Fourier transform to obtain the frequency domain signal ; Step 3.2: Frequency domain signal Further decomposed into amplitude spectrum and phase spectrum ; Step 3.3: The amplitude spectrum Domain-invariant frequency-domain filters in frequency-domain decoupling networks and domain-specific frequency domain filters Multiply each element by element to get the domain invariant amplitude spectrum and domain-specific amplitude spectra ; Step 3.4: Convert the domain-invariant amplitude spectrum Phase spectrum Synthesize into domain-invariant frequency domain signal ; The domain-specific amplitude spectrum Phase spectrum Synthesize into domain-specific frequency domain signals ; Step 3.5: Convert the domain-invariant frequency domain signal Perform inverse discrete Fourier transform to obtain domain-invariant image ; Domain-specific frequency domain signals Perform inverse discrete Fourier transform to obtain domain-specific images ; Step 3.6: Domain-invariant image and domain-specific images Input pre-trained object detection network Feature extraction is performed in the Backbone module, and domain-invariant features are output and domain-specific features ; Step 3.7, according to drone images Bounding box of , domain-invariant images Domain-invariant features Size ratio between domain-specific images Domain-specific features The size ratio between them is used to crop the domain invariant features. in target and domain-specific features in targets, and then use the RoI-Alignment operation to align the target size to obtain the domain-invariant target feature set and domain-specific target feature sets ;in, Representing domain-invariant features The domain-invariant target features, Representing domain-specific features The Domain-specific target features, ; Use formula (1) to calculate the drone images Contrastive loss ; (1) In formula (1), represents the similarity function, and represent and The number of elements in represents the temperature coefficient of contrast loss; Representation and Logic; is a collection The elements, is a collection The elements, Indicates the containing element A collection of Indicates the containing element A collection of is a collection With collection The first elements, is a collection With collection The first elements, is a collection With collection The union of The first elements, is a collection With collection The union of The first elements; Step 3.8: Domain-invariant features Input pre-trained object detection network In the Neck module and Prediction module, the predicted target category is obtained , predict the target bounding box and the predicted target confidence ; Use formula (2) to calculate the drone images Detection loss ; (2) In formula (2), Indicates the drone images BCE loss for target categories in Indicates the drone images CIOU loss of the target bounding box in Indicates the drone images BCE loss for target confidence; Step 3.9: Use Equation (3) to establish the total loss function of the domain generalization UAV target detection network : (3) In formula (3), It is a hyperparameter that balances detection loss and contrast loss in the total loss function.

3. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the domain generalization drone target detection method according to claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the domain generalized drone target detection method according to claim 1 or 2 are performed.

Images