Cross-domain infrared target detection method based on frequency domain style enhancement

Abstract

The invention relates to a cross-domain infrared target detection method based on frequency domain style enhancement, and belongs to the technical field of infrared image target detection in computer vision. A Fourier transform domain migration module converts a source domain image and a target domain image into a frequency domain space based on Fourier transform; a similar target domain image is generated by replacing the low-frequency amplitude component of the source domain image with the low-frequency amplitude component of the target domain image; the difference between the class target domain and the target domain on the image level and the class level is reduced through the coarse-to-fine domain alignment module, and it is guaranteed that the final target detection model keeps efficient detection performance in the target domain. According to the method, the similar target domain image is constructed, the migration problem of the two sub-domains is solved in sequence, effective migration from the source domain to the target domain is achieved, different characteristics of frequency domain information and a coarse-to-fine domain alignment strategy are utilized, the influence of domain migration is reduced to the maximum extent, and therefore the accuracy and robustness of target detection are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of infrared image target detection in computer vision, and in particular relates to a cross-domain infrared target detection method based on frequency domain style enhancement. Background Art

[0002] Object detection, as a key research direction in computer vision, has attracted a lot of attention in the past decade. The strategy of using convolutional neural networks (CNNs) for object detection has significantly improved the accuracy of the task. However, these advances usually rely on training with large-scale RGB datasets such as ImageNet, which often suffer from performance degradation in poor lighting conditions due to poor image quality. Unlike visible light cameras, infrared thermal imagers can still maintain excellent performance under these conditions and can operate effectively at any time of the day; moreover, they are small, low-cost, and unobtrusive, and have been widely used in both commercial and military fields.

[0003] Unfortunately, due to the lack of large-scale annotated datasets, the performance of object detection in infrared images is limited to comparable performance to that of visible light images. In addition, each object category in the image needs to be precisely outlined with a bounding box, which makes the annotation work very time-consuming, further exacerbating this problem.

[0004] Currently, a common approach to solve this problem is domain adaptation object detection (DAOD). This method originated from unsupervised domain adaptation (UDA), which has shown effectiveness in transferring knowledge from a labeled domain (source domain) to a related unlabeled domain (target domain). In contrast, DAOD is more complex because it not only requires identifying all target instances in an image, but also requires accurate classification of them. DAOD provides a key solution to improve the robustness of target detection under a variety of lighting conditions and is expected to overcome the limitations caused by the lack of labeled datasets.

[0005] Therefore, it is not only of great significance in theoretical research, but also shows great potential in practical applications, and can push target detection technology to a new level. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a cross-domain infrared target detection method based on frequency domain style enhancement. By constructing a target-like domain image and solving the two sub-domain migration problems in sequence, effective migration from the source domain to the target domain is achieved. The different characteristics of frequency domain information and the coarse-to-fine domain alignment strategy are utilized to minimize the impact of domain offset, thereby improving the accuracy and robustness of target detection.

[0007] The present invention solves the technical problem by the following technical solutions: A cross-domain infrared target detection method based on frequency domain style enhancement, the detection method comprising a Fourier transform domain migration module and a coarse-to-fine domain alignment module; The Fourier transform domain migration module converts the source domain image and the target domain image into the frequency domain space based on Fourier transform, extracts the amplitude and phase information of the source domain image and the target domain image respectively, and generates a target domain-like image by replacing the low-frequency amplitude component of the source domain image with the low-frequency amplitude component of the target domain image; After generating the class target domain image, the distribution difference between the class target domain and the target domain is aligned through the coarse-to-fine domain alignment module. The coarse-to-fine domain alignment module gradually reduces the difference between the class target domain and the target domain at the image level and the category level through coarse-grained alignment and fine-grained alignment, thereby ensuring that the final target detection model maintains efficient detection performance in the target domain.

[0008] Moreover, the Fourier transform domain migration module is specifically: For source domain images X s and the target domain image X t Perform Fourier transform respectively to get its frequency domain representation: ; in: u, v Represent the frequencies in the horizontal and vertical directions respectively; A s ( u, v )and A t ( u, v ) represent the amplitude components of the source domain and target domain images respectively; φ s ( u, v )and φ t ( u, v ) represent the phase components of the source domain and target domain images respectively; Preserve the phase information of the source domain image φ s ( u, v ), and the low-frequency amplitude component of the target domain image Replace the low-frequency amplitude components of the source domain image , generate frequency domain representation of target domain image F ( X ps ): ; Where: σ is a predefined low-frequency threshold used to distinguish low-frequency and high-frequency components in the frequency domain; After the low-frequency amplitude is replaced, the frequency domain representation of the target domain image is F ( X ps )( u, v ) is generated and converted back to the spatial domain by inverse Fourier transform to obtain the target domain image X ps : .

[0009] Moreover, the coarse-to-fine domain alignment module is specifically: The goal of coarse-grained alignment is to reduce the overall distribution difference between the class target domain and the target domain by aligning the global feature distribution. The maximum mean difference is used to measure the global feature distribution difference between the class target domain and the target domain. The maximum mean difference is defined as: ; in: D ps and D t Represent the sample distribution of the class domain and the target domain respectively; n ps and n t are the number of samples in the class domain and the target domain, respectively; Φ ( x ) means that the sample x Mapping to feature space High-dimensional feature representation in ; Minimizing the maximum mean difference achieves coarse-grained alignment between the class target domain and the target domain. The optimization goal is: Min MMD D ps ，D t ); Fine-grained alignment aims to further align the feature distributions of different categories. We introduce the adversarial loss, an alignment loss function at the category level, and construct a domain discriminator DDD. The goal of this discriminator is to distinguish whether the sample comes from the class target domain or the target domain. The adversarial loss is optimized to deceive the domain discriminator so that the discriminator cannot effectively distinguish the two domains. That is, the feature distributions of the class target domain and the target domain at the category level become similar. The adversarial loss is defined as: ; Minimizing the adversarial loss achieves feature alignment between the class-target domain and the target domain at the category level; The optimization goal of the coarse-to-fine domain alignment module is: ; in: λ is a hyperparameter used to balance the weight between coarse-grained and fine-grained alignment.

[0010] The advantages and beneficial effects of the present invention are: 1. The present invention uses a Fourier transform-based domain transfer module (FDT) and a coarse-to-fine domain alignment module (CFDA) to respectively complete two domain adaptation sub-problems: first, use the FDT module that does not require training to convert the image in the source domain to a target-like domain while keeping the image semantic information unchanged. The FDT module converts the source domain image and the target domain image to the frequency domain space through FFT, and then uses the low-frequency part of the target domain image to replace the low-frequency part of the source domain image. Finally, an image similar to the target domain is generated through IFFT.

[0011] 2. The present invention proposes a coarse-to-fine domain alignment module CFDA to solve the domain deviation between the target domain and the class target domain. The module progressively realizes the image-level and category-level domain alignment between the class target domain and the target domain from two levels of coarse granularity and fine granularity.

[0012] 3. The present invention realizes effective migration from source domain to target domain by constructing target-like domain images and solving two subdomain migration problems in sequence. It utilizes the different characteristics of frequency domain information and the coarse-to-fine domain alignment strategy to minimize the impact of domain offset, thereby improving the accuracy and robustness of target detection.

[0013] 4. The Fourier transform domain migration module of the present invention can change the style information of the image while maintaining the high-level semantic information of the image (such as spatial structure and shape), thereby generating a target domain-like image that is closer to the target domain. Compared with the traditional end-to-end domain migration method, the Fourier transform domain migration module does not require a large amount of labeled data, nor does it rely on complex adversarial training, and has high computational efficiency and practical application value; the target domain-like image generated by the Fourier transform domain migration module not only reduces the domain offset between the source domain and the target domain, but also provides better initial conditions for subsequent domain alignment, which helps to improve the cross-domain detection performance of the target detection model.

[0014] 5. The coarse-to-fine domain alignment module of the present invention can effectively reduce the domain offset between the class target domain and the target domain, so that the aligned class target domain image can be better used in the target detection model, thereby improving the detection performance of the model in the target domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the present invention. DETAILED DESCRIPTION

[0016] The present invention is further described in detail below through specific examples. The following examples are only illustrative and not restrictive, and the protection scope of the present invention cannot be limited thereto.

[0017] A cross-domain infrared target detection method based on frequency domain style enhancement, the innovation of which lies in: by constructing a target-like domain image and solving the two sub-domain migration problems in sequence, the effective migration from the source domain to the target domain is realized, and the different characteristics of frequency domain information and the coarse-to-fine domain alignment strategy are used to minimize the impact of domain offset, thereby improving the accuracy and robustness of target detection. Specifically, the present invention includes two modules. The specific network structure diagram can be seen in Figure 1 : 1. Fourier Domain Transfer (FDT): Based on Fourier transform, this module converts the source domain image and the target domain image into the frequency domain space and extracts their amplitude and phase information respectively. By replacing the low-frequency amplitude component of the source domain image (containing the style information of the image) with the low-frequency amplitude component of the target domain image, a target domain image is generated. This step effectively retains the semantic information of the source domain image and introduces style information that is closer to the target domain.

[0018] 2. Coarse-to-Fine Domain Alignment (CFDA): After generating the target domain image, the CFDA module is used to further align the distribution difference between the target domain and the target domain. This module gradually reduces the difference between the target domain and the target domain at the image level and category level through two steps: coarse-grained alignment and fine-grained alignment, ensuring that the final object detection model can maintain efficient detection performance in the target domain.

[0019] The Fourier transform domain migration module converts the source domain image and the target domain image into the frequency domain space based on Fourier transform, extracts the amplitude and phase information of the source domain image and the target domain image respectively, and generates a target domain-like image by replacing the low-frequency amplitude component of the source domain image with the low-frequency amplitude component of the target domain image; After generating the class target domain image, the distribution difference between the class target domain and the target domain is aligned through the coarse-to-fine domain alignment module. The coarse-to-fine domain alignment module gradually reduces the difference between the class target domain and the target domain at the image level and the category level through coarse-grained alignment and fine-grained alignment, thereby ensuring that the final target detection model maintains efficient detection performance in the target domain.

[0020] The Fourier transform domain migration module is specifically: For source domain images X s and the target domain image Xt Perform Fourier transform respectively to get its frequency domain representation: ; in: A s ( u, v )and A t ( u, v ) represent the amplitude components of the source domain and target domain images respectively; φ s ( u, v )and φ t ( u, v ) represent the phase components of the source domain and target domain images respectively; Preserve the phase information of the source domain image φ s ( u, v ), and the low-frequency amplitude component of the target domain image Replace the low-frequency amplitude components of the source domain image , generate frequency domain representation of target domain image F ( X ps ): ; Where: σ is a predefined low-frequency threshold used to distinguish low-frequency and high-frequency components in the frequency domain; After the low-frequency amplitude is replaced, the frequency domain representation of the target domain image is F ( X ps )( u, v ) is generated and converted back to the spatial domain by inverse Fourier transform to obtain the target domain image X ps : .

[0021] The advantage of the Fourier transform domain transfer module is that it can change the style information of the image while maintaining the high-level semantic information of the image (such as spatial structure and shape), thereby generating a target-domain image that is closer to the target domain. Compared with the traditional end-to-end domain transfer method, the FDT module does not require a large amount of labeled data, nor does it rely on complex adversarial training, and has high computational efficiency and practical application value.

[0022] The target-like domain images generated by the FDT module not only reduce the domain shift between the source and target domains, but also provide better initial conditions for subsequent domain alignment, which helps to improve the cross-domain detection performance of the object detection model.

[0023] The coarse-to-fine domain alignment module is as follows: The goal of coarse-grained alignment is to reduce the overall distribution difference between the class target domain and the target domain by aligning the global feature distribution. The maximum mean difference is used to measure the global feature distribution difference between the class target domain and the target domain. The maximum mean difference is defined as: ; in: D ps and D t Represent the sample distribution of the class domain and the target domain respectively; n ps and n t are the number of samples in the class domain and the target domain, respectively; Φ ( x ) means that the sample x Mapping to feature space High-dimensional feature representation in ; Minimizing the maximum mean difference achieves coarse-grained alignment between the class target domain and the target domain. The optimization goal is: Min MMD D ps ，D t ); Fine-grained alignment aims to further align the feature distributions of different categories. We introduce the adversarial loss, an alignment loss function at the category level, and construct a domain discriminator DDD. The goal of this discriminator is to distinguish whether the sample comes from the class target domain or the target domain. The adversarial loss is optimized to deceive the domain discriminator so that the discriminator cannot effectively distinguish the two domains. That is, the feature distributions of the class target domain and the target domain at the category level become similar. The adversarial loss is defined as: ; Minimizing the adversarial loss achieves feature alignment between the class-target domain and the target domain at the category level; The optimization goal of the coarse-to-fine domain alignment module is: ; in: λ is a hyperparameter used to balance the weight between coarse-grained and fine-grained alignment.

[0024] Through the above coarse-grained and fine-grained alignment strategies, the CFDA module can effectively reduce the domain offset between the class-target domain and the target domain, so that the aligned class-target domain images can be better used in the target detection model, thereby improving the detection performance of the model in the target domain.

[0025] The present invention follows the standard practice of domain adaptive object detection, which includes a source (visible light) and a target (infrared) domain dataset. Specifically, the training phase includes a labeled visible light training dataset and an unlabeled infrared training dataset, while the validation set of the infrared dataset is exclusively used in the testing phase.

[0026] We empirically evaluate the proposed method by conducting experiments on two different datasets for visible-to-infrared scenes: the KAIST multispectral dataset and the FLIR ADAS dataset, to demonstrate its effectiveness.

[0027] In the experiment, the model of the present invention uses Faster R-CNN as the target detection network, built on the ResNet-101 architecture, and pre-trained on the ImageNet dataset. The sampling ratio of positive and negative samples for each image is set to 1:3, and a total of 128 anchor points are generated. The model is trained with the SGD optimizer, with an initial learning rate of 0.001, a momentum of 0.9, and a weight decay of 5*10 -4 . During the entire training process, the balance parameter is λ = 0.5. For the evaluation of the model, mean average precision (mAP) and a threshold of 0.5 were used during testing. All experiments were performed on three NVIDIA GeForce 1080Ti GPUs.

[0028] The present invention compares the results of the trained model with the existing advanced model. From the experimental results, it can be seen that the present invention greatly improves the accuracy of the target detection task in the domain adaptation scenario from visible light to infrared. As shown in Table 1, the performance comparison of the algorithm of the present invention with the existing advanced network in the FLIR data set is shown in Table 2, and the performance comparison with the existing advanced network in the KAIST data set is shown. The method of the present invention obtains excellent results.

[0029] Table 1 Average accuracy comparison of KAIST multispectral dataset

[0030] Table 2 Average accuracy comparison of FLIR ADAS dataset

[0031] Although the embodiments and drawings of the present invention are disclosed for illustrative purposes, those skilled in the art will appreciate that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A cross-domain infrared target detection method based on frequency domain style enhancement, characterized by: The detection method includes a Fourier transform domain migration module and a coarse-to-fine domain alignment module; The Fourier transform domain migration module converts the source domain image and the target domain image into the frequency domain space based on Fourier transform, extracts the amplitude and phase information of the source domain image and the target domain image respectively, and generates a target domain-like image by replacing the low-frequency amplitude component of the source domain image with the low-frequency amplitude component of the target domain image; After generating the class target domain image, the distribution difference between the class target domain and the target domain is aligned through the coarse-to-fine domain alignment module. The coarse-to-fine domain alignment module gradually reduces the difference between the class target domain and the target domain at the image level and the category level through coarse-grained alignment and fine-grained alignment, thereby ensuring that the final target detection model maintains efficient detection performance in the target domain.

2. The cross-domain infrared target detection method based on frequency domain style enhancement according to claim 1 is characterized in that: The Fourier transform domain migration module is specifically: For source domain images X s and the target domain image X t Perform Fourier transform respectively to get its frequency domain representation: ； in: u,v Represent the frequencies in the horizontal and vertical directions respectively; A s ( u,v )and A t ( u,v ) represent the amplitude components of the source domain and target domain images respectively; φ s ( u,v )and φ t ( u,v ) represent the phase components of the source domain and target domain images respectively; Preserve the phase information of the source domain image φ s ( u,v ), and the low-frequency amplitude component of the target domain image Replace the low-frequency amplitude components of the source domain image , generate frequency domain representation of target domain image F ( X ps ): ； Where: σ is a predefined low-frequency threshold used to distinguish low-frequency and high-frequency components in the frequency domain; After the low-frequency amplitude is replaced, the frequency domain representation of the target domain image is F ( X ps )( u,v ) is generated and converted back to the spatial domain by inverse Fourier transform to obtain the target domain image X ps : 。 3. The cross-domain infrared target detection method based on frequency domain style enhancement according to claim 1 is characterized in that: The coarse-to-fine domain alignment module is specifically: The goal of coarse-grained alignment is to reduce the overall distribution difference between the class target domain and the target domain by aligning the global feature distribution. The maximum mean difference is used to measure the global feature distribution difference between the class target domain and the target domain. The maximum mean difference is defined as: ； in: D ps and D t Represent the sample distribution of the class domain and the target domain respectively; n ps and n t are the number of samples in the class domain and the target domain, respectively; Φ ( x ) means that the sample x Mapping to feature space High-dimensional feature representation in ; Minimizing the maximum mean difference achieves coarse-grained alignment between the class target domain and the target domain. The optimization goal is: Min MMD( D ps ，D t ); Fine-grained alignment aims to further align the feature distributions of different categories, introduce category-level alignment loss function to counter the loss, and build a domain discriminator. D , the goal of the discriminator is to distinguish whether the sample comes from the class target domain or the target domain. The adversarial loss is optimized to deceive the domain discriminator so that the discriminator cannot effectively distinguish the two domains, that is, the feature distribution of the class target domain and the target domain at the category level becomes similar. The adversarial loss is defined as: ； Minimizing the adversarial loss achieves feature alignment between the class-target domain and the target domain at the category level; The optimization goal of the coarse-to-fine domain alignment module is: ； in: λ is a hyperparameter used to balance the weight between coarse-grained and fine-grained alignment.

Images