Multimodal micro-object detection method, apparatus and program product

Abstract

The application discloses a multi-modal micro-target detection method, device and program product. The method comprises the following steps: acquiring a visible light image and an infrared image, and extracting multiple levels of visible light features and infrared features from the visible light image and the infrared image; for other levels except the last level, performing band decoupling cross-modal attention processing on the visible light features and the infrared features respectively to obtain cross-modal features corresponding to the visible light features and the infrared features respectively, and performing enhancement processing to obtain enhanced features corresponding to the visible light features and the infrared features respectively; for the last level, performing attention processing on the visible light features and the infrared features respectively to obtain enhanced features corresponding to the visible light features and the infrared features respectively; and performing decoding fusion processing on the enhanced features corresponding to the visible light features and the infrared features at each level to obtain class information of a target to be detected and position information of the target to be detected in the visible light image or the infrared image. The application can improve the precision of micro-target detection.

Description

Technical Field

[0001] The application relates to the field of computer vision technology, and in particular to a multimodal small target detection method, device, and program product. Background Technology

[0002] End-to-end multimodal small target detection aims to achieve accurate identification and localization of small targets within a unified framework. Current methods can be broadly categorized into two types: single-modal small target detection methods and multimodal visible-infrared target detection methods. Specifically, single-modal small target detection methods primarily rely on a single sensing modality, such as visible light or infrared images, and employ different strategies to overcome the inherent limitation of pixel information in small targets. Multimodal visible-infrared target detection methods, on the other hand, improve small target detection performance in complex environments by fusing information from different sensing modalities. While these methods have achieved good results in small target detection tasks, most still focus on single-modal frequency domain enhancement, resulting in low detection accuracy for small targets in complex environments. Summary of the Invention

[0003] This application provides a multimodal micro-target detection method, apparatus, and program product to effectively improve the detection accuracy of micro-targets in complex environments.

[0004] On the one hand, embodiments of this application provide a multimodal small target detection method, including:

[0005] Acquire visible light and infrared images; wherein both the visible light and infrared images contain the target to be measured;

[0006] Feature extraction processing is performed on the visible light image and the infrared image to obtain multiple levels of visible light features and infrared features;

[0007] For all levels except the last one, the visible light features and infrared features of the current level are subjected to band decoupling cross-modal attention processing to obtain the cross-modal features corresponding to the visible light features and infrared features at the current level, respectively. Then, the cross-modal features corresponding to the visible light features and infrared features at the current level are subjected to enhancement processing to obtain the enhanced features corresponding to the visible light features and infrared features at the current level.

[0008] For the last level, attention processing is performed on the visible light features and infrared features of the current level to obtain the enhanced features corresponding to the visible light features and infrared features at the current level.

[0009] The enhanced features corresponding to the visible light features and the infrared features at each of the aforementioned levels are decoded and fused to obtain the category information of the target under test and the position information of the target under test in the visible light image or the infrared image.

[0010] Further, in one embodiment, the step of performing band-decoupling cross-modal attention processing on the visible light features and infrared features of the current level to obtain the cross-modal features corresponding to the visible light features and infrared features at the current level includes:

[0011] Vector generation is performed based on the first target feature to obtain query features, key features, and value features;

[0012] The query feature, the key feature, and the value feature are subjected to frequency band decoupling processing respectively to obtain the spectral features of the query feature, the key feature, and the value feature in multiple sub-frequency bands;

[0013] Based on the spectral features of the query feature, the key feature, and the value feature in multiple sub-bands, frequency band-level frequency domain attention processing is performed to obtain the cross-modal feature corresponding to the first target feature;

[0014] Wherein, the first target feature is either the visible light feature or the infrared feature.

[0015] Further, in one embodiment, the vector generation process based on the first target feature to obtain query features, key features, and value features includes:

[0016] If the first target feature is the visible light feature, then the query feature is generated based on the infrared feature, and the key feature and the value feature are generated based on the visible light feature;

[0017] Alternatively, if the first target feature is the infrared feature, then the query feature is generated based on the visible light feature, and the key feature and the value feature are generated based on the infrared feature.

[0018] Further, in one embodiment, the step of performing band-level frequency domain attention processing based on the spectral features of the query feature, the key feature, and the value feature in multiple sub-bands to obtain the cross-modal feature corresponding to the first target feature includes:

[0019] Based on the spectral characteristics of the query features and the key features in each of the sub-bands, calculate the cross-modal weights of each of the sub-bands;

[0020] Based on the cross-modal weights of each sub-band and the spectral characteristics of the value features in each sub-band, modulation processing is performed to obtain the response features of each sub-band;

[0021] The response features of each sub-band are fused to obtain the cross-modal features corresponding to the first target feature.

[0022] Further, in one embodiment, the enhancement processing of the cross-modal features corresponding to the visible light feature and the infrared feature at the current level to obtain the enhanced features corresponding to the visible light feature and the infrared feature at the current level includes:

[0023] The second target feature is smoothed by irrelevant background processing to obtain the noise-suppressed feature;

[0024] The noise suppression features are subjected to foreground enhancement processing to obtain foreground enhancement features;

[0025] The noise suppression feature and the foreground enhancement feature are integrated to obtain the enhancement feature corresponding to the second target feature;

[0026] Wherein, the second target feature is any one of the cross-modal features corresponding to the visible light features or the cross-modal features corresponding to the infrared features.

[0027] Further, in one embodiment, the process of smoothing the second target feature against an irrelevant background to obtain the noise-suppressed feature includes:

[0028] The second target feature is segmented to obtain background features and foreground features;

[0029] The foreground features and the background features are fused together to obtain the noise suppression features.

[0030] Further, in one embodiment, the foreground enhancement processing of the noise suppression features to obtain foreground enhancement features includes:

[0031] Based on the noise suppression features, a dynamic filter is generated;

[0032] The high-level features are modulated using the dynamic filter to obtain guided upsampling features; wherein, the high-level features are any one of the cross-modal features corresponding to the visible light features in the previous level, or the cross-modal features corresponding to the infrared features in the previous level, and the modal type of the high-level features is the same as the modal type of the second target features;

[0033] The high-level features and the guided upsampling features are fused together to obtain the foreground enhancement features.

[0034] Further, in one embodiment, the step of decoding and fusing the enhanced features corresponding to the visible light features and the infrared features at each of the said levels to obtain the category information of the target under test and the position information of the target under test in the visible light image or the infrared image includes:

[0035] For each of the aforementioned levels, the enhanced features corresponding to the visible light features and the infrared features at the current level are fused to obtain the multi-scale features of the current level.

[0036] The multi-scale features of each level are preprocessed to obtain a multi-scale feature sequence.

[0037] The multi-scale feature sequence is decoded to obtain the category information of the target under test and the position information of the target under test in the visible light image or the infrared image.

[0038] On the other hand, embodiments of this application provide a multimodal small target detection device, including:

[0039] An acquisition module is used to acquire visible light images and infrared images; wherein both the visible light images and the infrared images contain the target to be measured;

[0040] The first processing module is used to perform feature extraction processing on the visible light image and the infrared image to obtain multiple levels of visible light features and infrared features;

[0041] The second processing module is used to perform band-decoupling cross-modal attention processing on the visible light features and infrared features of the current level for all levels except the last level, to obtain the cross-modal features corresponding to the visible light features and infrared features at the current level, and to perform enhancement processing on the cross-modal features corresponding to the visible light features and infrared features at the current level, to obtain the enhanced features corresponding to the visible light features and infrared features at the current level.

[0042] The third processing module is used to perform attention processing on the visible light features and infrared features of the current level for the last level, respectively, to obtain the enhanced features corresponding to the visible light features and infrared features at the current level.

[0043] The fourth processing module is used to decode and fuse the enhanced features corresponding to the visible light features and the infrared features at each of the aforementioned levels to obtain the category information of the target under test and the position information of the target under test in the visible light image or the infrared image.

[0044] In another aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the above-described multimodal micro-target detection method.

[0045] According to the embodiments of this application, a multimodal micro-target detection method, apparatus, and program product are provided. First, feature extraction is performed on the input visible light image and infrared image to obtain visible light features and infrared features. Then, for layers other than the last layer, the visible light features and infrared features are processed by frequency domain decomposition, cross-modal interaction, and frequency band fusion through a dynamic frequency band decoupling cross-modal attention mechanism to enhance cross-modal complementary frequency information. A selective smoothing enhancement strategy is also used to enhance feature information to guide cross-modal feature fusion while suppressing background noise. For the last layer, only the visible light features and infrared features are enhanced based on the attention mechanism. Finally, decoding and fusion operations are performed on the enhanced features corresponding to the infrared features and visible light features at all layers to output the category information of the target and the position information of the target in the visible light image or infrared image. This fully utilizes cross-modal information to accurately locate the key features of micro-targets, thereby effectively improving the accuracy and robustness of micro-target detection in complex environments. Attached Figure Description

[0046] Figure 1 This is a flowchart of a multimodal small target detection method provided in this application;

[0047] Figure 2 This is a schematic diagram of a multimodal small target detection method provided in this application;

[0048] Figure 3 This is a schematic diagram of the principle of band decoupling cross-modal attention processing for infrared modes provided in this application;

[0049] Figure 4 This is a schematic diagram of the frequency band decoupling processing based on value features provided in this application;

[0050] Figure 5 This is a schematic diagram of a portion of the frequency band-level frequency domain attention processing provided in this application;

[0051] Figure 6 This is a schematic diagram of the irrelevant background smoothing process provided in this application;

[0052] Figure 7 This is a schematic diagram of the foreground enhancement processing provided in this application;

[0053] Figure 8 This is a comparison chart of the visualization results provided in this application;

[0054] Figure 9 This is a structural diagram of a multimodal micro-target detection device provided in this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] The present application will be further described below with reference to the accompanying drawings and specific embodiments. The described embodiments should not be considered as limitations on the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0057] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] With the rapid development of computer vision technology, multimodal small target detection has become an important research direction in this field. The core task of multimodal small target detection is to comprehensively utilize complementary information from different sensing modalities such as visible light and infrared to accurately identify and locate targets with extremely small size and weak features in complex environments. It has wide-ranging application value in scenarios such as video surveillance, remote sensing, maritime search and rescue, and disaster relief. For example, in video surveillance, fusing visible light and infrared information can effectively improve the detection capability of pedestrians or vehicles at long distances under complex environmental conditions such as nighttime and backlighting. In remote sensing, multimodal information helps improve the recognition of small vessels, buildings, and other targets under complex environmental conditions such as clouds, fog, and shadows. In maritime rescue, using multimodal information to quickly locate small floating objects or people in distress on the sea surface can help improve rescue efficiency and success rate.

[0060] However, multimodal small target detection is an extremely challenging task, with difficulties mainly manifested in the following aspects: First, small targets occupy a very small proportion of an image, and their pixel information is limited. Therefore, multiple downsampling and pooling operations are often required during feature extraction, which can easily weaken or even lose the discriminative information of small targets, thus affecting the detection performance. Second, visible light sensors and infrared sensors have significant differences in imaging mechanisms. Even after registration, spatial offsets may still exist between modalities. For small targets, even slight misalignment can affect the cross-modal feature fusion effect, thereby affecting the detection accuracy. Third, the fine-grained structure of small targets is highly dependent on frequency information. Especially under conditions such as complex backgrounds, dynamic lighting, and partial occlusion, their effective frequency components are easily submerged by noise, resulting in poor detection performance. Finally, traditional upsampling and fusion methods are usually difficult to effectively recover foreground-related details and easily introduce redundant background information, thus affecting the localization and classification accuracy of small targets.

[0061] End-to-end multimodal small target detection aims to achieve accurate identification and localization of small targets within a unified framework. Current methods can be broadly categorized into two types: single-modal small target detection methods and multimodal visible-infrared target detection methods. Specifically, single-modal small target detection methods mainly rely on a single sensing modality, such as visible light or infrared images, and overcome the inherent limitation of pixel information in small targets through different strategies. In contrast, multimodal visible-infrared target detection methods improve the performance of small target detection in complex environments by fusing information from different sensing modalities.

[0062] Although these methods have achieved good results in small target detection tasks, in multimodal small target detection scenarios, most existing methods still focus on single-modal frequency domain enhancement. For example, they only emphasize the importance of high-frequency components in visible light images and are usually designed only for a single visible light mode. They do not fully consider the frequency complementarity relationship under the differences in imaging mechanisms of different modes, and cannot make full use of cross-modal information, resulting in low detection accuracy of small targets in complex environments.

[0063] In view of this, embodiments of this application provide a multimodal micro-target detection method, apparatus, and program product. First, feature extraction is performed on the input visible light and infrared images to obtain visible light and infrared features. Then, for layers other than the last layer, a dynamic frequency band decoupling cross-modal attention mechanism is used to process the visible light and infrared features through frequency domain decomposition, cross-modal interaction, and frequency band fusion to enhance cross-modal complementary frequency information. A selective smoothing enhancement strategy is also used to enhance feature information, guiding cross-modal feature fusion while suppressing background noise. For the last layer, only the visible light and infrared features are enhanced using the attention mechanism. Finally, decoding and fusion operations are performed on the enhanced features corresponding to the infrared and visible light features at all layers, outputting the category information of the target and its position information in the visible light or infrared image. This achieves high-precision micro-target detection, fully utilizing cross-modal information to accurately locate the key features of micro-targets, thereby effectively improving the accuracy and robustness of micro-target detection in complex environments.

[0064] The implementation process of the embodiments of this application will be described in detail below.

[0065] Reference Figure 1 and Figure 2 , Figure 1 This is a flowchart of a multimodal small target detection method provided in this application. Figure 2 This is a schematic diagram of a multimodal small target detection method provided in this application, which may include the following steps S101-S105.

[0066] S101, acquire visible light and infrared images.

[0067] It should be noted that both visible light and infrared images contain the target to be measured, which refers to a small object to be measured. The type of target to be measured can be flexibly set according to the actual situation; for example, the target to be measured can be a ship, pedestrian, vehicle, etc., but is not limited to these.

[0068] In this step, during the detection of small targets, visible light (vis) and infrared (ir) images containing the target are first acquired to facilitate the detection of small targets based on multimodal information in subsequent steps. It is worth noting that the acquired visible light and infrared images are approximately registered or fully registered.

[0069] Optionally, the acquisition methods for visible light and infrared images can be flexibly set according to the actual situation. For example, visible light images can be acquired in real time through a visible light sensor, and infrared images can be acquired in real time through an infrared sensor, and then relevant registration operations can be performed. Alternatively, pre-stored visible light and infrared images can be acquired through a preset database, but this is not limited to these methods.

[0070] S102 performs feature extraction processing on visible light and infrared images to obtain multiple levels of visible light and infrared features.

[0071] In this step, feature extraction processing is performed on the visible light image to obtain multiple levels of visible light features, and feature extraction processing is performed on the infrared image to obtain multiple levels of infrared features. In this way, the small target features of the infrared mode and the visible light mode can be initially captured. Among them, the feature extraction method can be flexibly set according to the actual situation. For example, the multi-scale features of the visible light mode, that is, the multi-level visible light features, can be extracted from the visible light image by the first mode-specific feature extractor, and the multi-scale features of the infrared mode, that is, the multi-level infrared features, can be extracted from the infrared image by the second mode-specific feature extractor, as shown in the following formula (1):

[0072] , (1);

[0073] In equation (1), Represents a visible light image; Represents an infrared image; This indicates a first-mode-specific feature extractor, which is applicable to the visible light mode; This indicates a second-mode-specific feature extractor, which is applicable to infrared modes; Indicates the first Visible light characteristics at each level (scale level), Indicates the first Infrared characteristics of each scale layer Indicates the number of feature levels.

[0074] Optionally, the type of the first modality-specific feature extractor and the type of the second modality-specific feature extractor can be flexibly set according to the actual situation. For example, both the first modality-specific feature extractor and the second modality-specific feature extractor can be ResNet50, but it is not limited to this.

[0075] S103, for all levels except the last one, perform band-decoupled cross-modal attention processing on the visible light features and infrared features of the current level to obtain the cross-modal features corresponding to the visible light features and infrared features at the current level, and perform enhancement processing on the cross-modal features corresponding to the visible light features and infrared features at the current level to obtain the enhanced features corresponding to the visible light features and infrared features at the current level.

[0076] In this step, all levels except the last one are traversed, and the following operations are performed on the traversed levels:

[0077] First, frequency band decoupling cross-modal attention (DFCA) processing is performed on the visible light features at the current level to obtain the cross-modal features corresponding to the visible light features at the current level. Then, frequency band decoupling cross-modal attention processing is performed on the infrared features at the current level to obtain the corresponding cross-modal features. This operation can fully exploit the complementary frequency information between the infrared and visible light modes, thereby improving the feature representation capability of small targets.

[0078] Then, the cross-modal features corresponding to the visible light features at the current level are enhanced (SSE) to obtain the enhanced features corresponding to the visible light features at the current level. Similarly, the cross-modal features corresponding to the infrared features at the current level are enhanced to obtain the enhanced features corresponding to the infrared features at the current level. This operation can enhance foreground-related information while smoothing and suppressing irrelevant background noise, and guide multi-scale feature fusion, thereby further enhancing the feature representation capability of small targets.

[0079] S104. For the last level, attention processing is performed on the visible light features and infrared features of the current level to obtain the enhanced features corresponding to the visible light features and infrared features of the current level.

[0080] In this step, attention processing is applied to the visible light features of the last level to obtain the enhanced features corresponding to the visible light features at the last level. Similarly, attention processing is applied to the infrared features of the last level to obtain the enhanced features corresponding to the infrared features at the last level. This enhances the feature representation capability of small targets. The attention mechanism can be flexibly configured according to the actual situation; for example, it can be a self-attention mechanism, but it is not limited to this.

[0081] It is worth noting that since the enhancement process uses features from higher levels, and the last level does not have higher levels, the feature processing for the last level only uses attention processing, instead of the method shown in step S103 above.

[0082] S105, decode and fuse the enhanced features corresponding to the visible light features and infrared features at each level to obtain the category information of the target under test and the position information of the target under test in the visible light image or infrared image.

[0083] In this step, the enhanced features corresponding to the visible light features and infrared features at each level are fused, stitched, and flattened before being input into a preset decoder. The decoder and its subsequent detection head are then used to output the category information of the micro-target and the position information of the micro-target in the visible light image or infrared image, thereby achieving high-precision micro-target detection.

[0084] It is important to note that either a first dataset or a second dataset can be used during training. The first dataset can include several sample images and their corresponding first label information. The sample images refer to infrared and visible light sample images containing the target object, while the first label information includes the target object's category and its location in the infrared sample image. The second dataset can include several sample images and their corresponding second label information. The second label information includes the target object's category and its location in the visible light sample image. Therefore, in testing or practical applications, if the first dataset is used during training, the final output location information is the target object's location in the infrared image; if the second dataset is used, the final output location information is the target object's location in the visible light image.

[0085] Therefore, the embodiments of this application first perform feature extraction to obtain features of the visible light mode and the infrared mode. Then, for the non-last layer, the visible light features and infrared features are dynamically divided and interacted across modes through frequency band decoupling cross-modal attention processing to enhance the cross-modal complementary frequency information related to small targets. Furthermore, an enhancement strategy of selective smoothing is used to smooth and suppress irrelevant backgrounds and guide the upsampling of high-level features, aiming to improve the quality of multi-scale feature fusion in complex environments. For the last layer, only attention processing is used to enhance feature representation. Finally, the decoding fusion operation is performed to output the target detection result, thereby realizing infrared-visible light small target detection. This can effectively improve the accuracy and robustness of small target detection in complex environments.

[0086] In some embodiments, refer to Figure 3 In step S103 above, the visible light features and infrared features of the current level are subjected to frequency band decoupling cross-modal attention processing to obtain the cross-modal features corresponding to the visible light features and infrared features at the current level, including the following steps S201-S203, wherein the first target feature is any one of the visible light features or infrared features.

[0087] S201, based on the first target feature, perform vector generation processing to obtain query features, key features and value features.

[0088] In this step, corresponding query (q) features, key (k) features, and value (v) features are generated based on the first target features, thereby mapping data from different modalities to the same semantic matching space and providing a preliminary data foundation for subsequent processing.

[0089] Furthermore, in some embodiments, step S201 above includes:

[0090] If the first target feature is a visible light feature, then the query feature is generated based on the infrared feature, and the key feature and value feature are generated based on the visible light feature;

[0091] Alternatively, if the first target feature is an infrared feature, then query features are generated based on visible light features, and key and value features are generated based on infrared features.

[0092] Here, the process can be divided into an infrared modality branch and a visible light modality branch. In the visible light modality branch, where the first target feature is a visible light feature, the vector generation process involves generating query features based on infrared features, and then generating key and value features based on visible light features. In the infrared modality branch, where the first target feature is an infrared feature, the vector generation process involves generating query features based on visible light features, and then generating key and value features based on infrared features. The specific implementation of the generation can be flexibly configured according to the actual situation; for example, the generation method could be a 1×1 point convolution followed by a 3×3 depthwise convolution, but it is not limited to this.

[0093] S202, perform frequency band decoupling processing on the query feature, key feature and value feature respectively to obtain the spectral features of the query feature, key feature and value feature in multiple sub-frequency bands.

[0094] In this step, the following operations are performed on the query feature, key feature, and value feature: Frequency band decoupling (FBD) is applied to the current feature to obtain its spectral characteristics in different sub-bands. By traversing the query feature, key feature, and value feature, the spectral characteristics of the query feature, key feature, and value feature in multiple sub-bands can be obtained. This operation can fully capture the complementary frequency information of small targets in different modes.

[0095] Furthermore, in some embodiments, step S202 above includes:

[0096] Frequency domain transformation is performed on the query features, key features, and value features respectively to obtain the transformed query features, transformed value features, and transformed key features;

[0097] The transformed query feature, transformed value feature, and transformed key feature are subjected to frequency band decoupling processing to obtain the spectral features of the query feature, key feature, and value feature in multiple sub-frequency bands.

[0098] Here, frequency domain transformation is performed on the query features, key features, and value features respectively, and dynamic sub-band division is performed through radial binary mask, as shown in the following formula (2):

[0099] , (2);

[0100] In equation (2), For frequency band indexing; Indicates the first Query features, value features, or key features at each level. spectral characteristics in each sub-band; This represents the Fast Fourier Transform (FFT). This represents element-wise multiplication; Indicates the first Each level of query features, value features, or key features; Indicates the first The radial binary mask for each sub-band can be expressed as follows (3):

[0101] (3);

[0102] In equation (3), This represents the frequency index, which is the coordinate of the feature map after the Fast Fourier Transform on the frequency plane. This represents the radial frequency amplitude of the frequency component relative to the origin in the two-dimensional Fourier domain. Indicates the frequency band boundary parameters. Number of sub-bands, interval Indicates the first Sub-band.

[0103] Optionally, according to the Nyquist theorem, the normalized frequency range is: Based on this , However, it is not limited to this.

[0104] Alternatively, in order to achieve dynamic frequency band allocation, Set as a learnable parameter and initialize it using a frequency multiplication-based method, but it is not limited to this.

[0105] Alternatively, the number of sub-bands can be flexibly set according to the actual situation. For example, the number of sub-bands can be three, that is, the spectrum is divided into low frequency band, mid frequency band and high frequency band, but it is not limited to this.

[0106] In one example, refer to Figure 4 , Figure 4 This illustrates a cross-modal attention processing procedure for frequency band decoupling, using value features as an example. Three sub-bands are set: low-frequency, mid-frequency, and high-frequency. In the frequency band decoupling process, the value features are first processed using a Fast Fourier Transform (FFT) to obtain the transformed value features. Then, a radial binary mask corresponding to the low-frequency band is used. The transformed value features are processed to obtain their spectral features in the low-frequency band. And the radial binary mask corresponding to the intermediate frequency band. The transformed value features are processed to obtain their spectral characteristics in the mid-frequency band. And the radial binary mask corresponding to the high frequency band. The transformed value features are processed to obtain their spectral features in the high-frequency band. .

[0107] S203, based on the spectral features of query features, key features and value features in multiple sub-bands, perform band-level frequency domain attention processing to obtain the cross-modal features corresponding to the first target feature.

[0108] In this step, the spectral characteristics of the query feature, key feature, and value feature in multiple sub-bands are used as a benchmark to perform band-level frequency domain attention processing. The aim is to perform cross-modal correlation calculation and local modulation within the band, thereby obtaining the cross-modal feature corresponding to the first target feature.

[0109] Furthermore, in some embodiments, step S203 above includes:

[0110] Based on the spectral characteristics of query features and key features in each sub-band, calculate the cross-modal weights of each sub-band;

[0111] Modulation processing is performed based on the cross-modal weights of each sub-band and the spectral characteristics of the value features in each sub-band to obtain the response features of each sub-band;

[0112] The response features of each sub-band are fused to obtain the cross-modal features corresponding to the first target feature.

[0113] Here, as Figure 5 As shown, Figure 5 The following is a partial process of frequency band-level frequency domain attention processing. For each sub-band, the following operations are performed:

[0114] First, the cross-modal weights of the current sub-band are calculated based on the spectral characteristics of the query features and key features in the current sub-band, as shown in the following formula (4):

[0115] (4);

[0116] In equation (4), Indicates the first Under each level Cross-modal weights of sub-bands; This represents the Inverse Fast Fourier Transform (IFFT). Indicates the first Query features at each level in the first spectral characteristics in each sub-band; Indicates the first The key features at the 1st level are in the 1st... spectral characteristics in each sub-band; Indicates complex conjugation.

[0117] Then, the cross-modal weights of the current sub-band are spatially adaptively adjusted by local convolutional modulation, and multiplied with the spectral features of the value features in the current sub-band to obtain the response features of the current sub-band, as shown in the following formula (5):

[0118] (5);

[0119] In equation (5), Indicates the first Under each level The response characteristics of each sub-band; This represents a 3×3 convolution; This represents the Sigmoid activation function; Indicates the first The sub-value feature of the level in the th... Spectral characteristics of each sub-band.

[0120] By traversing all sub-bands, the response characteristics of each sub-band can be obtained.

[0121] Finally, the response features of each sub-band are fused to obtain the cross-modal features corresponding to the first target feature. The fusion process can be flexibly set according to the actual situation; for example, the fusion process can be aggregation, normalization, and linear projection, as shown in the following formula (6):

[0122] , (6);

[0123] In equation (6), Indicates the first Cross-modal features corresponding to visible light or infrared features at each level; Representation layer normalization; This indicates a linear projection layer.

[0124] Therefore, this embodiment, through frequency band decoupling and cross-attention processing, can perform frequency band modeling and interactive enhancement of visible light and infrared features at the frequency domain level. Specifically, by aligning and fusing information from different frequency bands (e.g., low-frequency structural information, mid-frequency transition information, and high-frequency detail information), frequency coupling interference caused by traditional full-frequency hybrid modeling can be avoided, thus more accurately depicting the complementary relationship between the two modes in different frequency bands. At the same time, combined with a dynamic partitioning mechanism of learnable frequency band boundaries, the sub-frequency band range can be adaptively adjusted according to the spectral distribution of different scenes and target scales, thereby improving the generalization and robustness of small target detection. In addition, by combining cross-modal correlation calculation within the frequency band with local convolutional modulation, a global correspondence between cross-modal features can be established, and fine-grained adaptive correction of spatial position can be performed, effectively enhancing the weak response features of small targets and suppressing interference caused by complex background textures, thermal noise, and illumination changes. Finally, cross-modal features with both semantic stability and detail expression capabilities are obtained, providing more discriminative input for subsequent enhancement processing. This helps to improve the recognition accuracy, positioning stability, and anti-interference capability of small target detection.

[0125] In some embodiments, step S103 above involves enhancing the cross-modal features corresponding to the visible light features and infrared features at the current level to obtain enhanced features corresponding to the visible light features and infrared features at the current level, including the following steps S301-S303, wherein the second target feature is any one of the cross-modal features corresponding to the visible light features or the cross-modal features corresponding to the infrared features.

[0126] S301, perform irrelevant background smoothing on the second target feature to obtain the noise-suppressed feature.

[0127] In this step, irrelevant background smoothing is performed on the second target feature to reduce the negative impact of background noise and make the foreground target more prominent, thereby obtaining noise-suppressed features.

[0128] Furthermore, in some embodiments, reference is made to Figure 6 The above step S301 includes:

[0129] The second target features are segmented to obtain background features and foreground features;

[0130] The foreground and background features are fused together to obtain noise-suppressed features.

[0131] Here, firstly, the second target feature is input into the convolutional layer to predict the binary foreground mask. The binary foreground mask can be optimized during training using a supervised loss. The type of supervised loss can be flexibly set according to the actual situation; for example, Focal Tversky Loss can be chosen as the supervised loss, but it is not limited to this. The second target feature is then divided into foreground features and background features using this binary foreground mask, as shown in the following formula (7):

[0132] , , (7);

[0133] In equation (7), Indicates the first Foreground features at each level; Indicates the first Background features at each level; This represents a binary foreground mask.

[0134] Then, in order to suppress noise interference in complex backgrounds, the background features are smoothed, specifically by channel compression and restoration, to achieve background smoothing. The smoothing method can be flexibly set according to the actual situation; for example, it can be achieved through two consecutive 3×3 convolutional layers, first compressing the channel dimensions and then restoring them, as shown in formula (8) below:

[0135] (8);

[0136] In equation (8), Indicates the first Smoothed background features at each level; This indicates that the number of output channels is 3×3 convolution; This indicates that the number of output channels is 3×3 convolution, For dimensionality reduction ratio.

[0137] Finally, the smoothed background features are fused with the foreground features to obtain the noise-suppressed features. The fusion method can be flexibly set according to the actual situation; for example, the fusion method can be element-wise addition, as shown in formula (9) below:

[0138] (9);

[0139] In equation (9), Indicates the first Noise suppression features at each level.

[0140] Therefore, this embodiment first separates foreground information from background information through foreground mask prediction and background feature separation, thereby reducing the interference of noise in complex backgrounds on target detection. Then, it performs channel compression and restoration processing on background features to achieve background smoothing, which can effectively remove high-frequency noise in the background and enhance the smoothness and consistency of the background area. Finally, after fusing the smoothed background with the foreground features, it can effectively reduce the influence of background noise and make the foreground target more prominent, thereby helping to improve the accuracy and robustness of small target detection.

[0141] S302, perform foreground enhancement processing on the noise-suppressed features to obtain foreground-enhanced features.

[0142] In this step, foreground enhancement processing is performed on the noise-suppressed features obtained in the previous step, aiming to further highlight the foreground target and thus obtain foreground enhanced features.

[0143] Furthermore, in some embodiments, reference is made to Figure 7 The above step S302 includes:

[0144] Based on noise suppression features, a dynamic filter is generated;

[0145] The high-level features are modulated using a dynamic filter to obtain guided upsampling features. The high-level features are any one of the cross-modal features corresponding to the visible light features in the previous level or the cross-modal features corresponding to the infrared features in the previous level. The modal type of the high-level features is the same as that of the second target features.

[0146] High-level features and guided upsampling features are fused together to obtain foreground enhancement features.

[0147] Here, firstly, to recover foreground details in high-level features during upsampling, a spatially adaptive dynamic filter is generated based on the noise-suppressing features. Specifically, filter weights are generated through convolutional layers to predict features, as shown in the following formula (10):

[0148] (10);

[0149] In equation (10), Indicates the first Filter weight prediction features at each level; This represents a 3×3 convolution.

[0150] Normalization is performed within the local neighborhood of each spatial location to generate a location-dependent dynamic filter, as shown in the following formula (11):

[0151] (11);

[0152] In equation (11), Indicates the first Dynamic filters at multiple levels; Indicates spatial location; Indicates size is The local neighborhood; Represents local neighborhood Internal relative spatial position The offset.

[0153] Then, modulation processing is performed. Specifically, to align with the resolution of the high-level features to be upsampled, pixel-unshuffle is used to spatially reorganize them, resulting in a reorganized dynamic filter, as shown in the following formula (12):

[0154] (12);

[0155] In equation (12), Indicates the first The recombined dynamic filter at each level.

[0156] The new dynamic filter is divided into four sub-filters. These four position-adaptive sub-filters are used to modulate the high-level features to obtain four guided responses, as shown in the following formula (13):

[0157] , , (13);

[0158] In equation (13), Indicates the first The first level Sub-filter; Indicates the first Under each level Response characteristics of sub-filters; Indicates the first The cross-modal features corresponding to visible light or infrared features at each level are called high-level features.

[0159] Spatial resolution is restored by pixel-shuffle, resulting in guided upsampling features, as shown in formula (14) below:

[0160] (14);

[0161] In equation (14), Indicates the first Guided upsampling features at each level.

[0162] Finally, the high-level features are upsampled and then fused with the guided upsampled features to obtain the foreground enhancement features. The fusion method can be flexibly set according to the actual situation; for example, the fusion method can be pixel-by-pixel addition, as shown in formula (15) below:

[0163] (15);

[0164] In equation (15), Indicates the first Foreground enhancement features at each level; This indicates upsampling.

[0165] S303 integrates the noise suppression features and foreground enhancement features to obtain the enhancement features corresponding to the second target features.

[0166] In this step, to further balance semantic information and spatial detail information, the foreground enhancement features and enhancement features obtained in the first two steps are fused together and integrated through the CSP (Cross Stage Partial) module to obtain the enhancement features corresponding to the second target feature. The fusion method can be flexibly set according to the actual situation; for example, the fusion method can be concatenation, as shown in the following formula (16):

[0167] (16);

[0168] In equation (16), Indicates the first The enhanced features corresponding to the second target features at each level; This indicates a CSP module.

[0169] Therefore, this embodiment introduces a foreground-related enhancement and fusion mechanism, which effectively recovers foreground details in high-level features through dynamic filter generation and guided upsampling. Specifically, the dynamic filter is generated based on noise suppression and enhancement features and can adaptively adjust filter weights to finely adjust the foreground target at different spatial locations. Through pixel rearrangement and spatial reorganization, the filter is aligned with high-level features to ensure accurate recovery of details during upsampling. Finally, the guided upsampling features are fused with the standard upsampling results, further improving spatial resolution and detail representation. Simultaneously, the CSP module integrates and balances semantic information and spatial details, enhancing detection accuracy and robustness. This method effectively improves the ability to recognize small targets, especially in the recovery of target details in complex scenes.

[0170] In some embodiments, step S105 above involves decoding and fusing the enhanced features corresponding to the visible light features and infrared features at each level to obtain the category information of the target under test and the position information of the target under test in the visible light image or infrared image, including the following steps S401-S403:

[0171] S401, for each level, the enhanced features corresponding to the visible light features and infrared features at the current level will be fused to obtain the multi-scale features of the current level;

[0172] S402, preprocess the multi-scale features at each level to obtain the multi-scale feature sequence;

[0173] S403 decodes the multi-scale feature sequence to obtain the category information of the target under test and the location information of the target under test in the visible light image or infrared image.

[0174] In this embodiment, firstly, for each level, the following operations are performed: the enhanced features corresponding to the visible light features and infrared features at the current level are fused to obtain the multi-scale features of the current level. The multi-scale features of each level can be obtained by traversing multiple levels. Optionally, the fusion method can be flexibly set according to the actual situation. For example, the enhanced features corresponding to the visible light features and infrared features at the current level can be stitched together and input into the CSP module for feature integration to obtain the corresponding multi-scale features.

[0175] Then, the multi-scale features at each level are flattened and formed into a multi-scale feature sequence, which is a token sequence.

[0176] Finally, the multi-scale feature sequence is input into a pre-defined decoder, which decodes it. The decoding result is then input into the detection head, which outputs the target's category information and its location information in the visible or infrared image, thus achieving infrared-visible light micro-target detection. The type of decoder can be flexibly configured according to the actual situation. For example, the decoder can be a Transformer-based decoder that uses a deformable attention mechanism to decode the multi-scale feature sequence, but it is not limited to this.

[0177] The following practical application scenario will be used to verify the effectiveness of the embodiments of this application. This application scenario is a scene text recognition scenario based on single vector query decoding, that is, the target to be tested is text, as follows:

[0178] This embodiment sets up a network model, which includes a backbone network, a frequency band decoupling cross-modal attention module, a selective smoothing enhancement module, and a decoder connected in sequence. Specifically, a pre-trained ResNet50 is used as the backbone network for the visible light and infrared branches to perform the feature extraction shown in step S102 above, with the number of feature layers L set to 3; the number of sub-frequency bands in the frequency band decoupling cross-modal attention module used to perform step S103 above is set to 3, that is, three frequency bands: low frequency, mid frequency, and high frequency; data augmentation adopts basic augmentation methods, including random scaling, random cropping, and random flipping; the mask supervision in the selective smoothing enhancement module used to perform step S104 above is optimized using a FocalLoss variant to stabilize the training process and improve the foreground / background separation accuracy; step S105 above is implemented using a Transformer-based decoder and detector head.

[0179] The infrared-visible target detection dataset used for training and testing is as follows:

[0180] RGBT-Tiny: This dataset is a large-scale benchmark dataset for infrared-visible light small target detection. It mainly comes from UAV remote sensing scenes and contains 115 pairs of sequences, about 93,000 frames of images and about 1.2 million manually labeled targets, covering 7 types of targets and 8 types of scene types. More than 81% of the targets are smaller than 16×16 pixels.

[0181] RGBT-DronePerson: This dataset is an infrared-visible light pedestrian detection dataset, containing 6125 pairs of images and 70880 labeled instances, covering 4 types of pedestrian targets. Approximately 98% of the targets are smaller than 20 pixels, making it quite challenging.

[0182] During training, this embodiment sets the number of training epochs to 20 for both the RGBT-Tiny and RGBT-DronePerson datasets, and the learning rate to a uniform 0.00025. This embodiment adds a modality branch to the RT-DETR baseline to construct a comparison baseline, and keeps all other settings (including the number of decoder layers, number of queries, etc.) consistent with the corresponding baseline to ensure fairness in the comparison.

[0183] The experimental results are as follows:

[0184] 1) Comparative experiment:

[0185] To verify the effectiveness of the embodiments of this application, a systematic comparison was conducted between the embodiments of this application and various mainstream object detection methods on the RGBT-Tiny dataset. Specifically, under the same or comparable backbone networks and training settings, the performance of the embodiments of this application was compared with single-modal detection methods including Faster R-CNN, DINO, DiffusionDet, and RT-DETR, as well as infrared-visible fusion methods such as QFDet, CALNet, MDPMFN, and RSDet. The results are shown in Table 1 below.

[0186] Table 1: Performance comparison of the embodiments of this application with various object detection methods on the RGBT-Tiny dataset.

[0187]

[0188] As shown in Table 1, the embodiments of this application achieved the best detection performance on the RGBT-Tiny dataset, with an overall AP of 48.2% and an AP50 of 69.1%. Compared with the currently superior single-modal method RT-DETR (42.7% AP), the embodiments of this application improved the AP by 5.5 percentage points; compared with DiffusionDet (36.9% AP), the improvement reached 11.3 percentage points. This indicates that in complex and small target scenarios, single-modal methods are difficult to fully utilize cross-modal complementary information. Compared with existing multimodal fusion methods, the embodiments of this application also have significant advantages. Compared with QFDet (33.2% AP, 50.3% AP50) and CALNet (29.4% AP, 50.7% AP50), the embodiments of this application improved the AP and AP50 by approximately 15.0 percentage points and 18.4 percentage points, respectively; compared with MDPMFN (37.8% AP), the AP improved by 10.4 percentage points. Furthermore, the embodiments of this application achieve optimal results across all target scales, reaching 41.3% for tiny targets (APt), significantly outperforming other methods. They also maintain a leading position across different scale metrics (APes, APs, APm, APl), demonstrating strong scale adaptability. Notably, the embodiments of this application achieve the highest recall rate (AR) of 63.2%, indicating a significant advantage in reducing false negative rates, particularly in scenarios involving extremely small targets.

[0189] Furthermore, to further verify the effectiveness of the embodiments of this application in pedestrian detection scenarios at extremely small scales, the embodiments of this application were systematically compared with the above-mentioned mainstream object detection methods on the RGBT-DronePerson dataset. This dataset has the characteristics of extremely small target size, severe occlusion, and complex background, which puts forward higher requirements on the model's cross-modal fusion capability and small target modeling capability. The results are shown in Table 2 below. In Table 2, the asterisk indicates that the method has been modified to the RGBT baseline detector (dual stream) version.

[0190] Table 2: Performance comparison of the embodiments of this application with various object detection methods on the RGBT-DronePerson dataset.

[0191]

[0192] As shown in Table 2, the embodiments of this application achieve significantly better detection performance than existing methods on the RGBT-DronePerson dataset, with AP50 reaching 61.0%, AP50_t reaching 62.4%, and AP50_s reaching 35.1%, all of which are the best results among all methods. Compared with single-modal baseline methods, the embodiments of this application have obvious advantages. For example, compared with Faster R-CNN* (28.4% AP50) and GFL* (38.8% AP50), the embodiments of this application improve AP50 by 32.6 percentage points and 22.2 percentage points, respectively, which fully demonstrates that even when a single-modal detector is extended to a two-stream structure, it is still difficult to fully model cross-modal complementary information. Compared with the current high-performing multimodal fusion methods, the embodiments of this application also show stable and significant performance improvements. Compared to QFDet (42.1% AP50) and IM-CMDET (43.6% AP50), the embodiments of this application show improvements of 18.9 percentage points and 17.4 percentage points, respectively; compared to the current best method COXNet (45.5% AP50), it still shows an improvement of 15.5 percentage points. On the AP50_t metric for extremely small targets, the embodiments of this application achieve 62.4%, significantly higher than QFDet (44.0%) and RSDet (44.6%), demonstrating a significant advantage in small target detection. Furthermore, on the AP50_s metric for small-scale targets, the embodiments of this application achieve a performance of 35.1%, basically on par with TINet (35.0%), but significantly outperform in both overall AP50 and AP50_t metrics, indicating that the embodiments of this application not only have strong modeling capabilities for extremely small targets, but also maintain good generalization performance across targets of different scales.

[0193] Finally, the visualization results of the embodiments of this application are compared with those of CLANet, QFDet, and UA-CMDet, as follows: Figure 8 As shown, Figure 8In this context, GroundTruth represents the real image, Ours represents the recognition result of this application embodiment, RGB represents the visible light mode, and IR represents the infrared mode. Figure 8 As can be seen, the recognition result of this application embodiment is closest to the real image, which proves that this application embodiment has superiority in the task of detecting small targets.

[0194] 2) Ablation experiment:

[0195] To further verify the effectiveness of each module in the embodiments of this application, ablation experiments were conducted on the RGBT-Tiny dataset, and each functional module was gradually introduced to analyze its impact on the overall performance. The results are shown in Table 3 below:

[0196] Table 3: Ablation Experiment Results

[0197]

[0198] Table 3 shows that the baseline model (row 1) achieves AP, AP50, and APt of 45.4%, 65.9%, and 36.6% without DFCA and SSE. First, by introducing only the Dynamic Frequency Band Decoupling Cross-Modal Attention Module (DFCA) (row 2), AP increases to 46.8%, AP50 to 67.5%, and APt to 37.8%, representing improvements of 1.4%, 1.6%, and 1.2% respectively compared to the baseline. This result indicates that DFCA can fully exploit the complementary information between infrared and visible light in different frequency subbands, thereby enhancing the discriminative feature representation of small targets. Second, by introducing only the Selective Smoothing Enhancement Module (SSE) (row 3), AP and AP50 increase to 46.9% and 67.4% respectively, both improving by 1.5% compared to the baseline, while APt increases to 39.7%, a 3.1% improvement. This result demonstrates that SSE, by smoothing complex backgrounds and enhancing foreground response, can significantly improve the detection performance of small targets, especially in scenarios with extremely small targets. When both DFCA and SSE are introduced simultaneously (row 5), the model achieves the best performance, with AP reaching 48.2%, AP50 reaching 69.1%, and APt reaching 41.3%. Compared with the baseline, the overall performance is improved by 2.8%, 3.2%, and 4.7%, respectively, indicating a significant synergistic gain effect between the two. DFCA provides rich cross-modal frequency information, while SSE effectively suppresses background noise, thereby further highlighting the features of small targets and achieving synergistic optimization of "frequency enhancement-noise suppression". In addition, when only background smoothing (IBS) is retained and the upsampling fusion module (FREF) is removed (row 4), the performance decreases by 0.8% (AP), 0.9% (AP50), and 1.2% (APt) compared with the complete model, respectively, indicating that both background smoothing and foreground-related upsampling fusion sub-modules contribute positively to the detection performance. In summary, the ablation experiments fully verified the effectiveness and synergistic effect of the DFCA and SSE modules and their substructures in cross-modal small target detection, proving that the embodiments of this application can significantly improve the accuracy of small target detection and reduce the false negative rate in complex environments.

[0199] In addition, refer to Figure 9 This application also provides a multimodal micro-target detection device, comprising:

[0200] The acquisition module 501 is used to acquire visible light images and infrared images; wherein, both the visible light images and the infrared images contain the target to be measured;

[0201] The first processing module 502 is used to perform feature extraction processing on visible light images and infrared images to obtain visible light features and infrared features at multiple levels.

[0202] The second processing module 503 is used to perform frequency band decoupling cross-modal attention processing on the visible light features and infrared features of the current level for all levels except the last level, to obtain the cross-modal features corresponding to the visible light features and infrared features at the current level, and to perform enhancement processing on the cross-modal features corresponding to the visible light features and infrared features at the current level, to obtain the enhanced features corresponding to the visible light features and infrared features at the current level.

[0203] The third processing module 504 is used to perform attention processing on the visible light features and infrared features of the current level for the last level, so as to obtain the enhanced features corresponding to the visible light features and infrared features of the current level respectively.

[0204] The fourth processing module 505 is used to decode and fuse the enhanced features corresponding to the visible light features and infrared features at each level to obtain the category information of the target under test and the position information of the target under test in the visible light image or infrared image.

[0205] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0206] Finally, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described multimodal small target detection method.

[0207] The content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0208] In summary, this application proposes an end-to-end detection method for multimodal small target detection in infrared and visible light, which can achieve cross-modal feature enhancement and small target detection result output within a unified framework. Specifically, this application first uses a dynamic frequency band decoupling cross-modal attention mechanism to perform frequency domain decomposition, cross-modal interaction, and frequency band fusion of multimodal features to mine complementary frequency information between infrared and visible light, thereby improving the feature representation capability of small targets. Then, a selective smoothing enhancement strategy is used to enhance feature expression, enhancing foreground-related information while smoothing and suppressing irrelevant background noise, and guiding multi-scale feature fusion. Finally, a decoding fusion operation is performed to output the small target detection result, which includes category information and location information. This effectively improves the accuracy and robustness of small target detection in complex environments.

[0209] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0210] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A multimodal small target detection method, characterized in that, The method includes: Acquire visible light and infrared images; wherein both the visible light and infrared images contain the target to be measured; Feature extraction processing is performed on the visible light image and the infrared image to obtain multiple levels of visible light features and infrared features; For all levels except the last one, the visible light features and infrared features of the current level are subjected to band decoupling cross-modal attention processing to obtain the cross-modal features corresponding to the visible light features and infrared features at the current level, respectively. Then, the cross-modal features corresponding to the visible light features and infrared features at the current level are subjected to enhancement processing to obtain the enhanced features corresponding to the visible light features and infrared features at the current level. For the last level, attention processing is performed on the visible light features and infrared features of the current level to obtain the enhanced features corresponding to the visible light features and infrared features at the current level. The enhanced features corresponding to the visible light features and the infrared features at each of the aforementioned levels are decoded and fused to obtain the category information of the target under test and the position information of the target under test in the visible light image or the infrared image.

2. The method according to claim 1, characterized in that, The step of performing band-decoupled cross-modal attention processing on the visible light and infrared features of the current level to obtain the cross-modal features corresponding to the visible light and infrared features at the current level includes: Vector generation is performed based on the first target feature to obtain query features, key features, and value features; The query feature, the key feature, and the value feature are subjected to frequency band decoupling processing respectively to obtain the spectral features of the query feature, the key feature, and the value feature in multiple sub-frequency bands; Based on the spectral features of the query feature, the key feature, and the value feature in multiple sub-bands, frequency band-level frequency domain attention processing is performed to obtain the cross-modal feature corresponding to the first target feature; Wherein, the first target feature is either the visible light feature or the infrared feature.

3. The method according to claim 2, characterized in that, The vector generation process based on the first target feature, yielding query features, key features, and value features, includes: If the first target feature is the visible light feature, then the query feature is generated based on the infrared feature, and the key feature and the value feature are generated based on the visible light feature; Alternatively, if the first target feature is the infrared feature, then the query feature is generated based on the visible light feature, and the key feature and the value feature are generated based on the infrared feature.

4. The method according to claim 2, characterized in that, The step of performing frequency band-level frequency domain attention processing based on the spectral features of the query feature, the key feature, and the value feature in multiple sub-frequency bands to obtain the cross-modal feature corresponding to the first target feature includes: Based on the spectral characteristics of the query features and the key features in each of the sub-bands, calculate the cross-modal weights of each of the sub-bands; Based on the cross-modal weights of each sub-band and the spectral characteristics of the value features in each sub-band, modulation processing is performed to obtain the response features of each sub-band; The response features of each sub-band are fused to obtain the cross-modal features corresponding to the first target feature.

5. The method according to claim 1, characterized in that, The step of enhancing the cross-modal features corresponding to the visible light features and the infrared features at the current level to obtain the enhanced features corresponding to the visible light features and the infrared features at the current level includes: The second target feature is smoothed by irrelevant background processing to obtain the noise-suppressed feature; The noise suppression features are subjected to foreground enhancement processing to obtain foreground enhancement features; The noise suppression feature and the foreground enhancement feature are integrated to obtain the enhancement feature corresponding to the second target feature; Wherein, the second target feature is any one of the cross-modal features corresponding to the visible light features or the cross-modal features corresponding to the infrared features.

6. The method according to claim 5, characterized in that, The process of smoothing the second target feature against an irrelevant background to obtain noise-suppressed features includes: The second target feature is segmented to obtain background features and foreground features; The foreground features and the background features are fused together to obtain the noise suppression features.

7. The method according to claim 5, characterized in that, The foreground enhancement processing of the noise suppression features to obtain foreground enhancement features includes: Based on the noise suppression features, a dynamic filter is generated; The high-level features are modulated using the dynamic filter to obtain guided upsampling features; wherein, the high-level features are any one of the cross-modal features corresponding to the visible light features in the previous level, or the cross-modal features corresponding to the infrared features in the previous level, and the modal type of the high-level features is the same as the modal type of the second target features; The high-level features and the guided upsampling features are fused together to obtain the foreground enhancement features.

8. The method according to claim 1, characterized in that, The step of decoding and fusing the enhanced features corresponding to the visible light features and the infrared features at each of the aforementioned levels to obtain the category information of the target under test and the position information of the target under test in the visible light image or the infrared image includes: For each of the aforementioned levels, the enhanced features corresponding to the visible light features and the infrared features at the current level are fused to obtain the multi-scale features of the current level. The multi-scale features of each level are preprocessed to obtain a multi-scale feature sequence. The multi-scale feature sequence is decoded to obtain the category information of the target under test and the position information of the target under test in the visible light image or the infrared image.

9. A multimodal micro-target detection device, characterized in that, include: An acquisition module is used to acquire visible light images and infrared images; wherein both the visible light images and the infrared images contain the target to be measured; The first processing module is used to perform feature extraction processing on the visible light image and the infrared image to obtain multiple levels of visible light features and infrared features; The second processing module is used to perform band-decoupling cross-modal attention processing on the visible light features and infrared features of the current level for all levels except the last level, to obtain the cross-modal features corresponding to the visible light features and infrared features at the current level, and to perform enhancement processing on the cross-modal features corresponding to the visible light features and infrared features at the current level, to obtain the enhanced features corresponding to the visible light features and infrared features at the current level. The third processing module is used to perform attention processing on the visible light features and infrared features of the current level for the last level, respectively, to obtain the enhanced features corresponding to the visible light features and infrared features at the current level. The fourth processing module is used to decode and fuse the enhanced features corresponding to the visible light features and the infrared features at each of the aforementioned levels to obtain the category information of the target under test and the position information of the target under test in the visible light image or the infrared image.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal small target detection method as described in any one of claims 1-8.

Images