Visible light infrared image target detection method and system based on modal common characteristics

Abstract

The application discloses a visible light infrared image target detection method and system based on modal common characteristics, and the method comprises the following steps: inputting a visible light image and an infrared image into a pre-trained improved weak alignment target detection network to obtain a target detection result, wherein the improved weak alignment target detection network is composed of a feature extraction network, a plurality of feature fusion branches and a target detection head; the feature extraction network comprises two parallel feature extraction branches, which are respectively used for aggregating and extracting modal common characteristics for the visible light image and the infrared image to obtain multi-scale features; the feature fusion branch is composed of a connected spatial alignment network and a bidirectional feature correction fusion module; and the target detection head is used for target detection to obtain the target detection result. The application aims to reduce the modal difference between the visible light and the infrared image, improve the performance of the weak alignment visible light and infrared target detection algorithm, and adapt to the weak alignment visible light and infrared image target detection task.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method and system for target detection in visible and infrared images based on modal common features. Background Technology

[0002] In recent years, visible light and infrared (VLS) image target detection has achieved remarkable results due to its ability to fuse visible light and infrared modal information, and has been widely applied in video surveillance, autonomous driving, UAV reconnaissance, and robotics. In UAV target detection research, target detection methods are typically designed based on the assumption of strict alignment between visible light and infrared images. Therefore, researchers often perform preprocessing steps such as image registration on visible light and infrared image data to obtain aligned VLS-IR image data. However, due to the different imaging times of multi-mode sensors and the complexity of target motion, it is difficult to achieve precise alignment between visible light and infrared images in practical applications, resulting in misalignment between the images—a weak alignment problem between visible light and infrared. This makes fusing target information from multi-mode images challenging and significantly limits the performance of detection models. Currently, target detection in VLS under weak alignment conditions typically employs pixel-level pre-alignment solutions, which learn a global mapping function from the visible light and infrared images to correct pixel coordinates. Many of these methods utilize deformable fields for supervision, thereby achieving strict pre-alignment. However, besides the spatial differences between visible light and infrared images, the intermodal gaps also make it difficult to correctly match cross-modal representations in space, leading to errors in the estimation of offsets between visible light and infrared images. Therefore, how to reduce the modal differences between visible light and infrared images to improve the performance of weakly aligned visible light and infrared target detection algorithms has become a key technical problem that urgently needs to be solved. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a visible light and infrared image target detection method and system based on modal common features, in order to address the above-mentioned problems of the prior art. The present invention aims to reduce the modal differences between visible light and infrared images and improve the performance of weakly aligned visible light and infrared target detection algorithms, so as to adapt to the target detection task of weakly aligned visible light and infrared images.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0005] A target detection method based on modal common features in visible light and infrared images includes the following steps: inputting visible light images and infrared images into a pre-trained improved weakly aligned target detection network to obtain target detection results. The improved weakly aligned target detection network consists of a feature extraction network, multiple feature fusion branches, and a target detection head. The feature extraction network includes two parallel feature extraction branches for aggregating and extracting modal common features from the visible light and infrared images respectively to obtain multi-scale features. The feature fusion branches consist of a connected spatial alignment network and a bidirectional feature correction fusion module. The spatial alignment network spatially aligns the visible light feature maps and infrared feature maps of corresponding scales extracted by the feature extraction network. The bidirectional feature correction fusion module performs bidirectional feature correction fusion on the spatially aligned visible light feature maps and infrared feature maps to obtain fused features of corresponding scales. The target detection head is used to perform target detection based on the fused features of each scale to obtain target detection results.

[0006] Optionally, the feature extraction branch has a four-layer structure. The first layer consists of a simple stem module and multiple cascaded target detection state space modules. The simple stem module is used to downsample the input visible light image or infrared image to a specified size. The multiple cascaded target detection state space modules are used to capture gradient information flow. The second to fourth layers each include a visual cue merging module, multiple cascaded target detection state space modules, a dual-mode aggregation module, and a target search correction module connected in sequence. The visual cue merging module is used to downsample the input feature map to a specified size. The two parallel feature extraction branches in the second to fourth layers share the dual-mode aggregation module and the target search correction module. The dual-mode aggregation module aggregates and extracts modal common features from the input visible light feature map and infrared feature map to enhance the visible light feature map and infrared feature map. The target search correction module performs target search correction on the aggregated and enhanced visible light feature or infrared feature to achieve feature correction of visible light feature and feature enhancement of infrared feature.

[0007] Optionally, the step of using a dual-mode aggregation module to aggregate and extract modal common features from the input visible light feature map and infrared feature map to enhance the visible light feature map and infrared feature map includes: [The text abruptly ends here, so the translation stops.] and infrared feature map The process involves three branches: the first branch processes the visible light feature map... and infrared feature map The visible light channel features are refined using a channel attention mechanism. and infrared channel features The second branch focuses on the visible light feature maps. and infrared feature map Encoding operations are performed using a 3×3 dilated convolutional layer and a C3 module to extract deeper features as unique features of visible light. and unique infrared characteristics The third branch will include visible light feature maps. and infrared feature map After connecting along the channel dimension, another dilated convolutional layer with a kernel size of 3×3 is used to perform a convolution operation to obtain the modal common features. Modal common features The image is divided into two equal parts along the channel dimension. Each part is then passed through a first convolutional layer (Conv1) and a second convolutional layer (Conv2) with shared parameters and a kernel size of 3×3. The output of the second convolutional layer (Conv2) is processed by the SiLU activation function and then combined with the unique characteristics of visible light. Visible light channel characteristics Multiplication results in enhanced visible light characteristics, as well as unique infrared characteristics. Infrared channel characteristics Multiplication yields enhanced infrared features. The output of the first convolutional layer Conv1 is processed by the SiLU activation function and then hop-connected with the enhanced visible light features to obtain aggregated enhanced visible light features. The enhanced infrared feature is obtained by skipping connections with the enhanced infrared feature. .

[0008] Optionally, the step of performing target search correction on the aggregated and enhanced visible light features or infrared features through the target search correction module to achieve feature correction of visible light features and feature enhancement of infrared features includes: performing global average pooling on the aggregated and enhanced visible light features and infrared features respectively, concatenating the obtained pooled features, adding a learnable positional code in the spatial dimension and inputting it into an encoder containing a cross-modal attention mechanism, and then upsampling the encoder's encoding result to obtain spatial-level features at multiple upsampling scales; performing global average pooling on the aggregated and enhanced visible light features and infrared features respectively, concatenating the obtained pooled features, adding a learnable positional code in the channel dimension and inputting it into an encoder containing a cross-modal attention mechanism, and then upsampling the encoder's encoding result to obtain channel-level features at multiple upsampling scales; and adding the spatial-level features and channel-level features to achieve feature correction of visible light features and feature enhancement of infrared features.

[0009] Optionally, the spatial alignment of the visible light feature map and infrared feature map of the corresponding scale extracted by the feature extraction network through the spatial alignment network includes: capturing the modal representations of the target and key background regions in the modal features of the visible light feature map and infrared feature map of the corresponding scale extracted by the feature extraction network through a spatial attention mechanism; subtracting the two modal representations to obtain differential features; and using the differential features through three parallel feature matchers to predict the offset estimates (x,y) in the x and y axes, the scale estimate h with respect to height, and the scale estimate w with respect to width between the visible light feature map and the infrared feature map, respectively. The feature matcher consists of an average pooling layer, a convolutional layer, and a fully connected layer. The offset estimates (x,y) in the x and y axes are used as positional bias factors, and the scale estimates h with respect to height and w with respect to width are used as scaling factors. The spatial alignment of the visible light feature map and the infrared feature map is completed based on the positional bias factors and the scaling factors.

[0010] Optionally, the step of performing bidirectional feature correction and fusion on the spatially aligned visible light feature map and infrared feature map through the bidirectional feature correction and fusion module to obtain fused features at the corresponding scale includes: performing average pooling and max pooling on the input visible light feature map, and summing the average pooled visible light feature map and the infrared feature map to obtain the average feature map. The maximum feature map is obtained by summing the visible light feature map and the infrared feature map after max pooling. ; average characteristics , the most significant feature These two feature sets are concatenated along the channel dimension, and a channel-level weighted vector is extracted using a channel attention mechanism. After separation along the channel dimension, weighted vectors for the visible light and infrared modes are obtained respectively. and The weighted vector and Application to visible light feature maps and infrared feature map To eliminate channel noise in visible light and infrared features, thus obtaining the visible light features after channel noise elimination. and infrared features Visible light characteristics after eliminating channel noise Spatial weight information of visible light modes is extracted using spatial attention mechanism. Spatial weight information of visible light modes Visible light characteristics after channel noise removal Multiplication yields visible light mode features with channel and spatial noise eliminated. Infrared features after eliminating channel noise Spatial weight information of infrared modes is extracted using spatial attention mechanism. Spatial weight information of infrared modes Infrared characteristics after channel noise elimination Multiplying them yields infrared mode features that have eliminated channel and spatial noise. This will eliminate the visible light mode characteristics of channel noise and spatial noise. and infrared modal characteristics Summation yields the fusion characteristics .

[0011] Optionally, the weighted vector and Application to visible light feature maps and infrared feature map The function expression is:

[0012] ;

[0013] in, The visible light characteristics after eliminating channel noise. Infrared characteristics after eliminating channel noise.

[0014] Furthermore, the present invention also provides a visible light infrared image target detection system based on modal common features, including a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the visible light infrared image target detection method based on modal common features.

[0015] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the visible-infrared image target detection method based on modal common features by a processor.

[0016] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the visible light infrared image target detection method based on modal common features by a processor.

[0017] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: The improved weakly aligned target detection network designed in this invention consists of a feature extraction network, multiple feature fusion branches, and a target detection head. The feature extraction network includes two parallel feature extraction branches for aggregating and extracting modal common features from visible light and infrared images respectively to obtain multi-scale features. The feature fusion branch consists of a connected spatial alignment network and a bidirectional feature correction fusion module. The spatial alignment network spatially aligns the visible light and infrared feature maps extracted by the feature extraction network at the corresponding scales. The bidirectional feature correction fusion module then performs bidirectional feature correction fusion on the spatially aligned visible light and infrared feature maps to obtain fused features at the corresponding scales. The structure of the feature extraction network used in this invention greatly expands the receptive field of the model and alleviates the problem of the secondary complexity of the self-attention mechanism increasing the computational burden on the model. Secondly, a dual-mode aggregation module is introduced, which uses multiple convolutional layers to map visible light and infrared features to the same subspace, obtaining common features of visible light and infrared modes. By combining modal common features with modal unique features, the modal gap between visible light and infrared features is effectively reduced. The improved weakly aligned target detection network of this invention is designed for the dual-light offset fusion detection problem. It can reduce the modal differences between visible light and infrared images, improve the performance of weakly aligned visible light and infrared target detection algorithms, adapt to the target detection task of weakly aligned visible light and infrared images, and achieve good dual-light fusion detection results for non-strictly aligned visible light and infrared images. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the network structure of the improved weakly aligned target detection network in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the network structure of the dual-mode aggregation module in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the network structure of the spatial alignment network in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of the network structure of the bidirectional feature correction and fusion module in an embodiment of the present invention.

[0022] Figure 5 This is a partial example of a weakly aligned visible light and infrared image dataset constructed in an embodiment of the present invention, wherein (a) and (b) are a pair of visible light and infrared images with positional offset, respectively, and (c) and (d) are a pair of visible light and infrared images with rotational transformation, respectively. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0024] This embodiment of the visible light and infrared image target detection method based on modal common features includes the following steps: inputting visible light images and infrared images into a pre-trained improved weakly aligned target detection network to obtain target detection results, such as... Figure 1 As shown, the improved weakly aligned target detection network consists of a feature extraction network, multiple feature fusion branches, and a target detection head. The feature extraction network includes two parallel feature extraction branches for aggregating and extracting modal common features from visible light and infrared images to obtain multi-scale features. The feature fusion branches consist of a connected spatial alignment network and a bidirectional feature correction fusion module. The spatial alignment network spatially aligns the visible light and infrared feature maps extracted by the feature extraction network at the corresponding scales. The bidirectional feature correction fusion module then performs bidirectional feature correction fusion on the spatially aligned visible light and infrared feature maps to obtain fused features at the corresponding scales. The target detection head is used to perform target detection based on the fused features at each scale to obtain the target detection result. In this embodiment, when the visible light image and infrared image are input into the pre-trained improved weakly aligned target detection network, the size of the visible light image and infrared image is 640×640. First, two parallel feature extraction branches in the feature extraction network are used to aggregate and extract modal common features for the visible light image and infrared image respectively to obtain multi-scale features. For example, in this embodiment, image features of three scales are extracted. Then, they are sent to the feature fusion branch and processed sequentially through the dual-mode aggregation module and the target search and correction module to obtain visible light and infrared feature maps of sizes 80×80, 40×40, and 20×20. Then, the features of these three scales are input into the spatial feature alignment network for feature alignment, and then through the bidirectional feature correction and fusion module to achieve feature fusion. Finally, the fused features are input into the target detection head for target detection. The target detection head sets three different detection layers according to the different sizes of the feature maps to detect small, medium, and large targets respectively. "Weak alignment" refers to the problem of partial or complete positional offset, scaling, and rotation between visible light and infrared images. The improved weak alignment target detection network in this embodiment is designed for the dual-light offset fusion detection problem. It can reduce the modal differences between visible light and infrared images and improve the performance of weak alignment visible light and infrared target detection algorithms to adapt to target detection tasks of weak alignment visible light and infrared images. It achieves good dual-light fusion detection results for non-strictly aligned visible light and infrared images, and solves the problem that the target detection performance of visible light and infrared images is limited by non-strict alignment in real-world scenarios.

[0025] The feature extraction network in this embodiment is an extension of the Mamba-YOLO backbone network, using two parallel backbones to extract multi-scale features from visible light and infrared images respectively. For example... Figure 1 As shown, the feature extraction branch in this embodiment has a four-layer structure. The first layer consists of a simple stem module and multiple cascaded target detection state space modules (ODSSBlock) (×3 in the figure indicates the number of cascaded modules is 3). The simple stem module is used to downsample the input visible light image or infrared image to a specified size. The multiple cascaded target detection state space modules are used to capture gradient information flow. The second to fourth layers each include a visual cue merging module, multiple cascaded target detection state space modules, a dual-mode aggregation module, and a target search correction module connected in sequence. The visual cue merging module is used to downsample the input feature map to a specified size. The two parallel feature extraction branches in the second to fourth layers share the dual-mode aggregation module and the target search correction module. The dual-mode aggregation module aggregates and extracts modal common features from the input visible light feature map and infrared feature map to enhance the visible light feature map and infrared feature map. The target search correction module performs target search correction on the aggregated and enhanced visible light feature or infrared feature to achieve feature correction of visible light feature and feature enhancement of infrared feature. It should be noted that the simple stem module is an existing, well-known module, so its implementation details will not be described in detail here.

[0026] like Figure 2 As shown, in this embodiment, a dual-mode aggregation module is used to aggregate and extract common modal features from the input visible light feature map and infrared feature map to enhance the aggregation of the visible light feature map and infrared feature map. This aims to combine the common features of visible light and infrared modes, further reduce the modal gap between visible light and infrared features, and improve the accuracy of subsequent offset estimation. In this embodiment, the dual-mode aggregation module's aggregation and extraction of common modal features from the input visible light feature map and infrared feature map to enhance the aggregation of the visible light feature map and infrared feature map includes: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] and infrared feature map The process involves three branches: the first branch processes the visible light feature map... and infrared feature map The visible light channel features are refined using a channel attention mechanism. and infrared channel features The second branch focuses on the visible light feature maps. and infrared feature map Encoding operations are performed using a dilated convolutional layer with a kernel size of 3×3 and a C3 module (an existing, well-known module) to extract deeper features as unique features of visible light. and unique infrared characteristics The third branch will include visible light feature maps. and infrared feature map After connecting along the channel dimension, another dilated convolutional layer with a kernel size of 3×3 is used to perform a convolution operation to obtain the modal common features. , can be represented as:

[0027] ;

[0028] in, , and For different convolutions, , Modal unique features were extracted from visible light and infrared features respectively. This is used to extract common features between visible and infrared modes. This indicates that the unique features of visible light and infrared modes are spliced ​​along the channel dimension; common features of modes are also spliced. Dividing the channel into two equal parts along its dimension can be represented as:

[0029] ;

[0030] in, The feature obtained by dividing it into two equal parts This represents the operation of segmenting common features along the channel dimension; passing the two parts through two convolutional layers Conv1 and Conv2 with shared parameters and kernel size of 3×3 can be represented as:

[0031] ; ;

[0032] ; ;

[0033] in, Indicates the SiLU activation function;

[0034] The output of the second convolutional layer Conv2 is processed by the SiLU activation function and then combined with the unique characteristics of visible light. Visible light channel characteristics Multiplication results in enhanced visible light characteristics, as well as unique infrared characteristics. Infrared channel characteristics Multiplication yields enhanced infrared features. The output of the first convolutional layer Conv1 is processed by the SiLU activation function and then hop-connected with the enhanced visible light features to obtain aggregated enhanced visible light features. The enhanced infrared feature is obtained by skipping connections with the enhanced infrared feature. , can be represented as:

[0035] ;

[0036] .

[0037] The aggregated image features need to be processed by an object search and correction module to correct redundancy in visible light features and enhance infrared features. First, global average pooling is performed on the input visible light and infrared images to reduce subsequent workload. The resulting pooled features are concatenated, and a learnable positional code is added sequentially in the spatial and channel dimensions, then input into the encoder. The result is then upsampled. This module uses a cross-modal attention mechanism to better capture intermodal correlations. Finally, the spatial and channel features are added to achieve feature correction for visible light features and feature enhancement for infrared features. In this embodiment, the target search correction module performs target search correction on the aggregated and enhanced visible light features or infrared features to achieve feature correction of visible light features and feature enhancement of infrared features. This includes: performing global average pooling on the aggregated and enhanced visible light features and infrared features respectively; concatenating the resulting pooled features; adding a learnable positional code in the spatial dimension and inputting it into an encoder containing a cross-modal attention mechanism; then upsampling the encoder's encoding result to obtain spatial-level features at multiple upsampling scales; performing global average pooling on the aggregated and enhanced visible light features and infrared features respectively; concatenating the resulting pooled features; adding a learnable positional code in the channel dimension and inputting it into an encoder containing a cross-modal attention mechanism; then upsampling the encoder's encoding result to obtain channel-level features at multiple upsampling scales; and finally, adding the spatial-level features to the channel-level features to achieve feature correction of visible light features and feature enhancement of infrared features. The target search and correction module performs global average pooling on the input visible light and infrared images to reduce subsequent workload. The obtained pooled features are concatenated, and a learnable position code is added sequentially in the spatial dimension and the channel dimension. The code is then input into the encoder, and the result is upsampled. Finally, the spatial features and channel features are added to achieve feature correction of visible light features and feature enhancement of infrared features.

[0038] like Figure 4As shown, in this embodiment, spatial alignment of the visible light feature map and infrared feature map of the corresponding scale extracted by the feature extraction network through a spatial alignment network includes: capturing the modal representations of the target and key background regions in the modal features of the visible light feature map and infrared feature map of the corresponding scale extracted by the feature extraction network through a spatial attention mechanism; subtracting the two modal representations to obtain differential features to reduce computational burden; and using the differential features through three parallel feature matchers to predict the offset estimates (x,y) in the x and y axes, the scale estimate h with respect to height, and the scale estimate w with respect to width between the visible light feature map and the infrared feature map, respectively. The feature matcher consists of an average pooling layer, a convolutional layer, and a fully connected layer. The offset estimates (x,y) in the x and y axes are used as positional bias factors, and the scale estimates h with respect to height and w with respect to width are used as scaling factors. The positional bias factors and scaling factors can be expressed as:

[0039] ;

[0040] Spatial alignment of visible light feature maps and infrared feature maps is completed based on positional deviation factors and scaling factors. Specifically, the translation magnitude in the x and y axes is the offset estimate (x,y) in the x and y axes, the scaling factor in the height direction is the scale estimate h about the height, and the scaling factor in the width direction is the scale estimate w about the width.

[0041] like Figure 3 As shown, in this embodiment, the bidirectional feature correction and fusion module performs bidirectional feature correction and fusion on the spatially aligned visible light feature map and infrared feature map to obtain fused features at the corresponding scale. This includes: performing average pooling and max pooling on the input visible light feature map, and summing the average pooled visible light feature map and the infrared feature map to obtain the average feature. The maximum feature map is obtained by summing the visible light feature map and the infrared feature map after max pooling. ; average characteristics , the most significant feature These two feature sets are concatenated along the channel dimension, and a channel-level weighted vector is extracted using a channel attention mechanism. After separation along the channel dimension, weighted vectors for the visible light and infrared modes are obtained respectively. and , can be represented as:

[0042] ;

[0043] in, This indicates a feature separation operation. This represents the Sigmoid activation function, and CA represents the channel attention mechanism. This indicates splicing along the channel dimension. By introducing channel attention and spatial attention mechanisms, visible light and infrared features are refined, computational load is reduced, and channel and spatial noise are suppressed. The weighted vectors... and Application to visible light feature maps and infrared feature map To eliminate channel noise in visible light and infrared features, thus obtaining the visible light features after channel noise elimination. and infrared features Visible light characteristics after eliminating channel noise Spatial weight information of visible light modes is extracted using spatial attention mechanism. Spatial weight information of visible light modes Visible light characteristics after channel noise removal Multiplication yields visible light mode features with channel and spatial noise eliminated. Infrared features after eliminating channel noise Spatial weight information of infrared modes is extracted using spatial attention mechanism. Spatial weight information of infrared modes Infrared characteristics after channel noise elimination Multiplying them yields infrared mode features that have eliminated channel and spatial noise. This will eliminate the visible light mode characteristics of channel noise and spatial noise. and infrared modal characteristics Summation yields the fusion characteristics In this embodiment, the weighted vector... and Application to visible light feature maps and infrared feature map The function expression is:

[0044] ;

[0045] in, The visible light characteristics after eliminating channel noise. Infrared characteristics after eliminating channel noise.

[0046] The training of the improved weakly aligned target detection network in this embodiment includes: Step 1: preparing a dataset of weakly aligned visible and infrared images, using infrared images as reference images and visible images as sensing images; Step 2: constructing an improved weakly aligned target detection network based on the CMA-Det algorithm; Step 3: training the improved weakly aligned target detection network using the training dataset, loading the best weights after training into the improved weakly aligned target detection network to obtain the improved weakly aligned target detection network compared to the pre-trained network; based on this, the improved weakly aligned target detection network is used to obtain the final detection result for the weakly aligned visible and infrared image pairs to be detected.

[0047] In step one, the weakly aligned visible and infrared image dataset was primarily collected from parking lots and roads using drones equipped with visible and infrared sensors. The visible and infrared images were acquired by the drones at a flight altitude of 110 meters with a camera pitch angle of 90 degrees and at a flight altitude of 80 meters with a camera pitch angle of 60 degrees. The dataset includes four categories: car, bus, truck, and van. Existing visible and infrared target detection methods are based on aligned data; therefore, this embodiment constructs a weakly aligned visible and infrared image dataset, addressing weak alignment issues such as positional offset, scale scaling, and rotation, making it suitable for training and validating weakly aligned visible and infrared image target detection methods. Specifically, the weakly aligned visible light and infrared image dataset in this embodiment is mainly collected using a DJI Mavic 3T drone, which is equipped with visible light and infrared cameras. It can simultaneously acquire visible light and infrared image pairs of parking lots and main roads. However, due to drone flight vibrations and the spacing between camera lenses, there is a certain positional offset between the acquired visible light and infrared image pairs. Furthermore, to address weak alignment issues such as scale transformation, rotation, and projection between the dual-mode images, the drone captures visible light and infrared image pairs at a flight altitude of 110 meters and a camera pitch angle of 90 degrees, and at a flight altitude of 80 meters and a camera pitch angle of 60 degrees. The resulting weakly aligned visible light and infrared image dataset can be used for training and testing the proposed method. Some image examples are shown below. Figure 5 As shown.

[0048] To verify the effectiveness of the visible light and infrared image target detection method based on modal common features in this embodiment, the test results of the method on the constructed weakly aligned visible light and infrared image dataset are shown in Table 1. The dataset contains a total of 15,878 visible light and infrared image pairs, covering 7 main road scenes and 11 parking lot scenes. The dataset was divided according to different scenes. The training set covers 5 main road scenes and 9 parking lot scenes, totaling 12,037 image pairs, while the remaining 3,841 image pairs were used for testing.

[0049] Table 1: Test results of the method in this embodiment on a self-constructed weakly aligned visible light and infrared image dataset.

[0050]

[0051] As shown in Table 1, the detection performance metrics mAP@0.5, mAP@0.75, and mAP@0.5:0.95 of the method on the self-constructed weakly aligned visible light and infrared image dataset can reach 79%, 72.5%, and 59.1%, respectively. The experimental results show that the method has good detection performance in the visible light and infrared image fusion detection task under weakly aligned scenarios.

[0052] This embodiment uses the DroneVehicle dataset for comparative experiments. The performance metrics used include precision (P), recall (R), and mean precision (mAP) at different intersection-to-union (IU) ratios (IU), specifically mAP@0.5 at an IU of 0.5, mAP@0.75 at an IU of 0.75, and mAP@0.5:0.95 at IUs ranging from 0.5 to 0.95. The method in this embodiment is compared with five existing state-of-the-art methods (GM-DETR, DaFF, MFPF, RGBXFusion, and CMA-Det). The test results on the DroneVehicle dataset are shown in Table 2. The DroneVehicle dataset is a large-scale dataset constructed from drone images captured in various scenarios, providing 28,439 pairs of visible and infrared images for both daytime and nighttime scenes. The dataset includes five target categories: cars, trucks, buses, railcars, and vans. This experiment used 17,990 image pairs as training data and 1,469 image pairs as test data, with the labeled infrared images used as the true values ​​for both training and testing.

[0053] Table 2: Test results of different methods on the DroneVehicle dataset

[0054]

[0055] As shown in Table 2, the method in this embodiment achieved the best results on the mAP50 test metric, reaching 85.4%, which is higher than other methods in terms of detection accuracy. These experimental results demonstrate the effectiveness of the method in this embodiment. By combining common modal features, it effectively reduces the modal gap between visible light and infrared modes, improves the accuracy of subsequent feature alignment transformation matrix estimation, and can better adapt to target detection tasks in weak alignment scenarios.

[0056] In this embodiment, the ablation experiment was conducted by extending YOLOv5 and introducing a target search correction module as a baseline method. The results are shown in Table 3.

[0057] Table 3: Ablation test results of the method in this embodiment on the DroneVehicle dataset

[0058]

[0059] Table 3 shows the test results of the baseline method on the dataset in the first row. The impact of the spatial alignment network is shown in the second row; compared to the first row, the detection accuracy metrics mAP@0.5, mAP@0.75, and mAP@0.5:0.95 increased by 0.6%, 0.7%, and 1.2%, respectively. Experimental results show that this network can effectively eliminate the impact of spatial misalignment on the detection algorithm performance and achieve alignment of spatial features between visible and infrared modalities. The impact of the bidirectional feature fusion correction module is shown in the third row; compared to the second row, the detection accuracy metrics mAP@0.5, mAP@0.75, and mAP@0.5:0.95 increased by 2.2%, 2.8%, and 2.2%, respectively. Experimental results show that this module further aligns and corrects the visible and infrared modalities, eliminating noise interference during the visible and infrared feature fusion process. The impact of the dual-mode aggregation module is shown in the fourth row. Compared with the third row, the detection accuracy indices mAP@0.5, mAP@0.75, and mAP@0.5:0.95 increased by 3.4%, 2.6%, and 2.5%, respectively. Experimental results show that this module further eliminates the gap between the visible light and infrared modes, improving the accuracy of subsequent feature alignment.

[0060] Furthermore, this embodiment also provides a visible light infrared image target detection system based on modal common features, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the visible light infrared image target detection method based on modal common features.

[0061] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the visible-infrared image target detection method based on modal common features by a processor.

[0062] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the visible light infrared image target detection method based on modal common features by a processor.

[0063] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0064] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for target detection in visible and infrared images based on modal common features, characterized in that, The process includes the following steps: inputting visible light images and infrared images into a pre-trained improved weakly aligned target detection network to obtain target detection results. The improved weakly aligned target detection network consists of a feature extraction network, multiple feature fusion branches, and a target detection head. The feature extraction network includes two parallel feature extraction branches used to aggregate and extract modal common features from the visible light and infrared images respectively to obtain multi-scale features. The feature fusion branch consists of a connected spatial alignment network and a bidirectional feature correction fusion module. The spatial alignment network spatially aligns the visible light and infrared feature maps extracted by the feature extraction network at corresponding scales. The bidirectional feature correction fusion module then performs bidirectional feature correction fusion on the spatially aligned visible light and infrared feature maps to obtain fused features at the corresponding scale. The target detection head is used to perform target detection based on the fused features at each scale to obtain target detection results. The feature extraction branch has a four-layer structure, where the first... The first layer consists of a simple stem module and multiple cascaded target detection state space modules. The simple stem module is used to downsample the input visible light or infrared image to a specified size. The multiple cascaded target detection state space modules are used to capture gradient information flow. The second to fourth layers each include a visual cue merging module, multiple cascaded target detection state space modules, a dual-mode aggregation module, and a target search correction module connected in sequence. The visual cue merging module is used to downsample the input feature map to a specified size. The two parallel feature extraction branches in the second to fourth layers share the dual-mode aggregation module and the target search correction module. The dual-mode aggregation module aggregates and extracts modal common features from the input visible light and infrared feature maps to enhance them. The target search correction module performs target search correction on the aggregated and enhanced visible light or infrared features to achieve feature correction of visible light features and feature enhancement of infrared features.

2. The visible-infrared image target detection method based on modal common features according to claim 1, characterized in that, The step of using a dual-mode aggregation module to aggregate and extract modal common features from the input visible light feature map and infrared feature map to enhance the visible light feature map and infrared feature map includes: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] and infrared feature map The process involves three branches: the first branch processes the visible light feature map... and infrared feature map The visible light channel features are refined using a channel attention mechanism. and infrared channel features The second branch focuses on the visible light feature maps. and infrared feature map Encoding operations are performed using a 3×3 dilated convolutional layer and a C3 module to extract deeper features as unique features of visible light. and unique infrared characteristics The third branch will include visible light feature maps. and infrared feature map After connecting along the channel dimension, another dilated convolutional layer with a kernel size of 3×3 is used to perform a convolution operation to obtain the modal common features. Modal common features The image is divided into two equal parts along the channel dimension. Each part is then passed through a first convolutional layer (Conv1) and a second convolutional layer (Conv2) with shared parameters and a kernel size of 3×3. The output of the second convolutional layer (Conv2) is processed by the SiLU activation function and then combined with the unique characteristics of visible light. Visible light channel characteristics Multiplication results in enhanced visible light characteristics, as well as unique infrared characteristics. Infrared channel characteristics Multiplication yields enhanced infrared features. The output of the first convolutional layer Conv1 is processed by the SiLU activation function and then hop-connected with the enhanced visible light features to obtain aggregated enhanced visible light features. The enhanced infrared feature is obtained by skipping connections with the enhanced infrared feature. .

3. The visible-infrared image target detection method based on modal common features according to claim 1, characterized in that, The step of using a target search correction module to perform target search correction on the aggregated and enhanced visible light or infrared features to achieve feature correction of visible light features and feature enhancement of infrared features includes: performing global average pooling on the aggregated and enhanced visible light features and infrared features respectively; concatenating the pooled features and adding a learnable positional code in the spatial dimension, which is then input into an encoder containing a cross-modal attention mechanism; upsampling the encoder's encoding result to obtain spatial-level features at multiple upsampling scales; performing global average pooling on the aggregated and enhanced visible light features and infrared features respectively; concatenating the pooled features and adding a learnable positional code in the channel dimension, which is then input into an encoder containing a cross-modal attention mechanism; upsampling the encoder's encoding result to obtain channel-level features at multiple upsampling scales; and summing the spatial-level features and channel-level features to achieve feature correction of visible light features and feature enhancement of infrared features.

4. The visible-infrared image target detection method based on modal common features according to claim 1, characterized in that, The spatial alignment of the visible light feature map and infrared feature map of the corresponding scale extracted by the feature extraction network through the spatial alignment network includes: capturing the modal representations of the target and key background regions in the modal features of the visible light feature map and infrared feature map of the corresponding scale extracted by the feature extraction network through a spatial attention mechanism; subtracting the two modal representations to obtain differential features; and using the differential features through three parallel feature matchers to predict the offset estimates (x,y) in the x and y axes, the scale estimate h with respect to height, and the scale estimate w with respect to width between the visible light feature map and the infrared feature map, respectively. The feature matcher consists of an average pooling layer, a convolutional layer, and a fully connected layer. The offset estimates (x,y) in the x and y axes are used as positional bias factors, and the scale estimates h with respect to height and w with respect to width are used as scaling factors. The spatial alignment of the visible light feature map and the infrared feature map is completed based on the positional bias factors and the scaling factors.

5. The visible-infrared image target detection method based on modal common features according to claim 1, characterized in that, The step of performing bidirectional feature correction and fusion on the spatially aligned visible light feature map and infrared feature map through the bidirectional feature correction and fusion module to obtain fused features at the corresponding scale includes: performing average pooling and max pooling on the input visible light feature map, and summing the average pooled visible light feature map and the infrared feature map to obtain the average feature. The maximum feature map is obtained by summing the visible light feature map and the infrared feature map after max pooling. ; average characteristics , the most significant feature These two feature sets are concatenated along the channel dimension, and a channel-level weighted vector is extracted using a channel attention mechanism. After separation along the channel dimension, weighted vectors for the visible light and infrared modes are obtained respectively. and The weighted vector and Application to visible light feature maps and infrared feature map To eliminate channel noise in visible light and infrared features, thus obtaining the visible light features after channel noise elimination. and infrared features Visible light characteristics after eliminating channel noise Spatial weight information of visible light modes is extracted using spatial attention mechanism. Spatial weight information of visible light modes Visible light characteristics after channel noise removal Multiplication yields visible light mode features with channel and spatial noise eliminated. Infrared features after eliminating channel noise Spatial weight information of infrared modes is extracted using spatial attention mechanism. Spatial weight information of infrared modes Infrared characteristics after channel noise elimination Multiplying them yields infrared mode features that have eliminated channel and spatial noise. This will eliminate the visible light mode characteristics of channel noise and spatial noise. and infrared modal characteristics Summation yields the fusion characteristics .

6. The visible-infrared image target detection method based on modal common features according to claim 5, characterized in that, The weighted vector and Application to visible light feature maps and infrared feature map The function expression is: ； in, The visible light characteristics after eliminating channel noise. Infrared characteristics after eliminating channel noise.

7. A visible-infrared image target detection system based on modal common features, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the visible-infrared image target detection method based on modal common features as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the visible-infrared image target detection method based on modal common features as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the visible-infrared image target detection method based on modal common features as described in any one of claims 1 to 6.

Images