Small target detection method and system based on aligned visible light and infrared images

Abstract

The invention provides a small target detection method and system based on aligned visible light and infrared images, and relates to the field of target detection, and the method comprises the steps: obtaining a visible light image and an infrared image of a to-be-detected small target; inputting a visible light image and an infrared image into the trained detection model, firstly, respectively performing multi-scale feature extraction on the visible light image and the infrared image by adopting a double-branch structure, and in the extraction process, performing multi-scale feature extraction on the visible light image and the infrared image through interactive collaborative learning of the visible light image and the infrared image; the method comprises the following steps: firstly, carrying out feature alignment, modal interaction correction and multi-scale feature enhancement between two modals to obtain enhanced visible light features and infrared features, then carrying out feature fusion, and finally, carrying out small target positioning and classification by utilizing the fused features. According to the method, feature level alignment of image pairs is realized by using deformable convolution and modal interaction correction, so that the features of visible light and infrared images are effectively and fully interacted and fused, and the accuracy of a detection algorithm is improved.

Description

Technical Field

[0001] This invention relates to the field of target detection, and more specifically to a method and system for small target detection based on aligned visible light and infrared images. Background Technology

[0002] The rapid development of the Internet and digital media technologies has made digital images and videos important mediums for people to transmit and obtain information. Small object detection is a subfield of general object detection, focusing on detecting small-sized objects. It has important theoretical and practical significance in various scenarios such as surveillance, drone scene analysis, remote sensing images, pedestrian detection, and traffic sign detection in autonomous driving.

[0003] Thanks to massive amounts of training data and the powerful learning capabilities of deep convolutional neural networks, small object detection has achieved remarkable success. However, most deep learning-based object detection algorithms are designed and applied to single-modal visible light images, which present limitations in low-light and adverse weather conditions. Infrared images can effectively compensate for this limitation; therefore, detection algorithms that fuse visible light and infrared images have gradually become a research hotspot. Infrared images provide the thermal radiation characteristics of targets in low-light and occluded environments, while visible light images retain rich texture and color details. By effectively combining the complementary information of the two modalities, the accuracy of detection is improved.

[0004] Existing small target detection algorithms that fuse visible light and infrared images have the following shortcomings: Feature fusion mechanisms have poor adaptability and are prone to losing key information of small targets: Existing methods either have redundant interference or adopt fixed strategies when fusing multimodal features, and cannot dynamically adjust the fusion weights according to the image content, resulting in the dilution or loss of detailed features that are crucial to small targets during the fusion process.

[0005] Insufficient cross-modal alignment capability restricts the effectiveness of fusion: Most methods do not fully consider the geometric misalignment between visible light and infrared images, and existing alignment strategies (such as spatial alignment or modal conversion) have limited flexibility and are difficult to achieve accurate alignment at the feature level, thus affecting the accuracy of subsequent fusion and detection.

[0006] In summary, the aforementioned shortcomings limit the accuracy, robustness, and practicality of existing technologies in small target detection tasks using visible light and infrared images. Summary of the Invention

[0007] To address the aforementioned problems, this invention proposes a method and system for small target detection based on aligned visible light and infrared images. By utilizing deformable convolution and modal interaction correction, feature-level alignment of image pairs is achieved, thereby effectively enabling full interaction and fusion of features from visible light and infrared images, and improving the accuracy of the detection algorithm.

[0008] According to some embodiments, the present invention adopts the following technical solution: Small target detection methods based on aligned visible and infrared images include: Acquire visible light and infrared images of the small target to be detected; Visible light and infrared images are input into the trained detection model to detect small targets and obtain the localization and classification results of small targets. The detection model first employs a dual-branch structure to extract multi-scale features from visible light and infrared images. During the extraction process, through interactive collaborative learning between visible light and infrared images, feature alignment, modal interaction correction, and multi-scale feature enhancement between the two modalities are achieved, resulting in enhanced visible light and infrared features. Then, feature fusion is performed, and finally, the fused features are used to locate and classify small targets.

[0009] According to some embodiments, the present invention adopts the following technical solution: Small target detection systems based on aligned visible and infrared images include: The image acquisition module is configured to acquire visible light and infrared images of the small target to be detected; The target detection module is configured to input visible light images and infrared images into the trained detection model to detect small targets and obtain the localization and classification results of the small targets. The detection model first employs a dual-branch structure to extract multi-scale features from visible light and infrared images. During the extraction process, through interactive collaborative learning between visible light and infrared images, feature alignment, modal interaction correction, and multi-scale feature enhancement between the two modalities are achieved, resulting in enhanced visible light and infrared features. Then, feature fusion is performed, and finally, the fused features are used to locate and classify small targets.

[0010] According to some embodiments, the present invention adopts the following technical solution: A computer program product includes a computer program that, when executed by a processor, implements the small target detection method based on aligned visible and infrared images.

[0011] According to some embodiments, the present invention adopts the following technical solution: A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the small target detection method based on aligned visible light and infrared images.

[0012] According to some embodiments, the present invention adopts the following technical solution: An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the small target detection method based on aligned visible light and infrared images.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proactively addresses the geometric misalignment problem between visible light and infrared images at the feature level through a specially designed cross-modal alignment and adaptive resampling module (including spatial resampling alignment and modal interaction correction). This module not only achieves preliminary geometric alignment through spatial resampling but also further corrects it through intermodal interaction and information complementarity, achieving more refined and flexible feature-level alignment than existing methods. This improvement lays a precise and high-quality feature foundation for subsequent feature fusion and detection, significantly enhancing the effectiveness of fusion and the accuracy of final detection.

[0014] This invention enhances the ability to extract multi-scale features, especially small target features, from images by introducing a multi-scale feature enhancement module and combining it with an attention mechanism. The attention mechanism helps the model focus on key regions and features in the data and suppress irrelevant information, thereby making the feature representation of small targets more significant and robust. This design works synergistically with the aforementioned fusion and alignment mechanisms to improve the accuracy, robustness, and practicality of the model in detecting small targets under various complex environments (such as low light and severe weather).

[0015] This invention introduces an adaptive fusion module (emphasizing a dynamic gating mechanism) that dynamically and adaptively adjusts the fusion weights of visible light and infrared modal features based on the specific content of the input image. This allows the model to selectively enhance modal features that are more important to the current detection task for different scenes and targets (e.g., relying more on infrared features in low light and more on visible light features when the texture is complex). This ensures that detailed information crucial to small targets is effectively preserved and enhanced during the fusion process, avoiding the loss of key features due to fixed fusion strategies or redundant interference. Attached Figure Description

[0016] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0017] Figure 1 This is a flowchart of the small target detection method based on aligned visible light and infrared images in Example 1. Figure 1 This is a structural diagram of the detection model in Example 1. Figure 2 This is a structural diagram of the cross-modal alignment and adaptive resampling module in Example 1. Figure 3 This is a structural diagram of the multi-head cross-attention module in Example 1. Figure 4 This is a structural diagram of the multi-scale feature enhancement module in Example 1.

[0018] Figure 5 This is a structural diagram of the adaptive fusion module in Example 1. Figure 6 This is a diagram of the dynamic gating mechanism in Example 1. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0020] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0022] Example 1 One embodiment of the present invention provides a small target detection method based on aligned visible light and infrared images, comprising: Step S1: Acquire visible light and infrared images of the small target to be detected; Step S2: Input the visible light image and infrared image into the trained detection model to perform small target detection and obtain the localization and classification results of the small targets; The detection model first employs a dual-branch structure to extract multi-scale features from visible light and infrared images. During the extraction process, through interactive collaborative learning between visible light and infrared images, feature alignment, modal interaction correction, and multi-scale feature enhancement between the two modalities are achieved, resulting in enhanced visible light and infrared features. Then, feature fusion is performed, and finally, the fused features are used to locate and classify small targets.

[0023] As one embodiment, the small target detection method based on aligned visible light and infrared images of the present invention constructs a detection model and utilizes deformable convolution and modal interaction correction to achieve feature-level alignment of image pairs. This effectively enables full interaction and fusion of features from visible light and infrared images, improving the accuracy of the detection algorithm. The specific implementation process is as follows: like Figure 1 As shown, the detection model consists of seven parts: visible light branch, infrared branch, cross-modal alignment and adaptive resampling module, multi-scale feature enhancement module, adaptive fusion module, neck and detection head. The specific steps for model construction, training and inference are as follows: Step 1: Construct a visible light branch for extracting multi-scale features from visible light images. The visible light branch uses ResNet50 as the backbone network, which generates visible light features of different dimensions through multiple convolutional layers.

[0024] The input of the visible light branch is The visible light image is output as visible light features in different dimensions. , , , and .

[0025] Step 2: Construct an infrared branch for extracting infrared image features. The infrared branch also uses ResNet50 as the backbone network, and after multiple convolutional layers, infrared features of different dimensions are obtained.

[0026] The input of the infrared branch is The infrared image is aligned with the visible light image, and the output is infrared features in different dimensions. , , , and .

[0027] Step 3: Build a cross-modal alignment and adaptive resampling module The module consists of two parts: a spatial resampling alignment module and a modal interaction correction module, as described below. Figure 2As shown in (a) and (b), the input to the spatial resampling alignment module is visible light features. and infrared features The output is the aligned visible light features. and infrared features Then, it is used as input to the modal interaction correction module to obtain the corrected alignment features. and Specifically: Step 3-1 Spatial Resampling Alignment Module First, the visible light characteristics and infrared features The input is fed into the spatial resampling and alignment module, which performs channel-level concatenation of the features from the two modalities to obtain the concatenated features. , represented as: (1) Concat represents feature concatenation along the channel dimension.

[0028] Then, convolutional layers and activation functions are introduced to obtain offsets and modulation coefficients, achieving feature alignment. The calculation is expressed as: (2) in, express The convolutional layer, where ReLU represents the ReLU activation function.

[0029] Modulation coefficient K The calculation expression is shown in equation (3): (3) in, This represents the Sigmold activation function.

[0030] Finally, deformable convolution is applied to both modalities to obtain aligned features.

[0031] For the visible light branch, the offset, modulation coefficient, and original visible light features are used as inputs to deformable convolution to obtain aligned visible light features. , represented as: (4) Wherein, DFConv represents deformable convolution.

[0032] Similarly, for the infrared branch, the same processing is used to obtain aligned infrared features. , represented as: (5) Step 3-2 Modal Interaction Correction Module The aligned visible light features obtained in step 3-1 and infrared features The input is fed into the modal interaction correction module, and the corrected alignment features are output. and Specifically: First, the aligned visible light features obtained in step 3-1 and infrared features The input is fed into a multi-head cross-attention module to obtain differential complementary features. and .

[0033] Multi-head cross-attention modules, such as Figure 3 As shown, the aligned visible light features Through a 1×1 convolutional layer, it is mapped to visible light query features. Visible optical bond characteristics and visible light value characteristics Furthermore, a channel dimensionality reduction factor is introduced during the mapping process. r To reduce computational complexity while retaining sufficient key information to maintain the accuracy and effectiveness of feature mapping, a mask is introduced. M This is used to filter out irrelevant positions, thereby limiting the computational cost of attention and making the attention matrix sparse.

[0034] Regarding the mask M The computation first preserves the dependencies between adjacent positions through local attention, introducing a sparsity factor. a Calculate the window size for local attention. k for: (6) in, and These represent the width and height of the feature map, respectively, and the sparsity. a Set it to 0.1.

[0035] Initialization mask M for To improve local attention, a zero matrix is ​​obtained for each location. i The mask is set as a strip matrix, that is, the area near the diagonal is assigned a value of 1, while the other areas are assigned 0, represented as: (7) Then, global connectivity is used to improve global attention to features, and the stride of the calculated window is determined. s for: (8) Every s,set up and To ensure long-distance information exchange.

[0036] Visible light query features and infrared bond features Multiplication and attention graph At the same time, a mask is introduced. M To reduce attention computation, it can be represented as: (9) Here, Softmax represents the Softmax activation function.

[0037] Visible light value characteristics With attention map Multiply, and use Convolution reshapes the feature size to obtain differential complementary features. , represented as: (10) Similarly, the infrared branch is processed using the same operation to process the input infrared features. Mapped to infrared query features Infrared key features and infrared value characteristics .

[0038] Using infrared query features and visible light bond characteristics Multiplication and attention graph At the same time, a mask is introduced. M To reduce attention computation, it can be represented as: (11) infrared value characteristics With attention map Multiply, and use Convolution reshapes the feature size to obtain differential complementary features. , represented as: (12) Complementary features of visible light differences Cross-enhanced visible light features are obtained through layer normalization and residual connection operations. , represented as: (13) Where LN represents layer normalization.

[0039] Then, deep features are further extracted using a multilayer perceptron, and after normalization, residual connections are introduced to obtain a corrected and aligned feature representation. , represented as: (14) MLP is a multilayer perceptron consisting of two linear layers.

[0040] Similarly, by processing the infrared branches using the same operation, cross-enhanced infrared features are obtained. , represented as: (15) Then, deep features are further extracted using a multilayer perceptron, and after normalization, residual connections are introduced to obtain a corrected and aligned feature representation. , represented as: (16) Step 4: Build a multi-scale feature enhancement module like Figure 4 As shown, this module utilizes channel attention and multi-scale enhancement to achieve enhanced feature extraction, with the input being corrected and aligned visible light features. and infrared features The output is the enhanced visible light feature. and infrared features Specifically: Step 4-1 First, correct and align the input visible light features. Further deep features are extracted through operations such as convolution, batch normalization, and activation functions. , represented as: (17) Wherein, CBR indicates sequential execution. Operations of convolution, batch normalization, and RuLU activation functions.

[0041] Similarly, the infrared branch is processed using the same operation to further extract its depth features. , represented as: (18) Step 4-2: Utilize the visible light depth features obtained in Step 4-1. Channel attention assigns different weights to features in different channels, enabling the network to focus on more important features. The new weights are then multiplied by the original features, improving the network's ability to capture key information. This process can be represented as: (19) Here, GAP represents the Global Average Pooling operation. This indicates element-wise multiplication.

[0042] Then, residual connections are added to the original feature map, allowing the network to deeply mine key features while preserving the original feature information, as shown below: (20) Similarly, by processing the infrared branch using the same operation, channel attention is employed to improve the ability to capture key information, as shown below: (twenty one) Then, the residual connection is added to the original feature map, as shown below: (twenty two) Step 4-3: To fuse the multi-scale information of the features, the channel-enhanced visible light features obtained in Step 4-2 are... Multi-scale enhancement is performed to obtain the result of multi-scale feature merging. .

[0043] This process is accomplished using three convolutional kernels of different scales, as follows: (twenty three) in , and They represent , and The convolutional kernels are set to stride 0, 1, and 2 respectively to keep the feature map size constant.

[0044] Finally, operations such as convolution, batch normalization, and activation functions are used to adjust the feature map dimensions, perform feature fusion and enhancement, and obtain the final enhanced features. , represented as: (twenty four) Similarly, the infrared branch is processed using the same operation, and multi-scale feature merging is achieved using three different convolutional kernels. , represented as: (25) The final enhanced features are obtained by using operations such as convolution, batch normalization, and activation functions. , represented as: (26) Step 5: Build the adaptive fusion module like Figure 5 As shown, this module adjusts the weights of the fused features through a dynamic gating mechanism, with the input being the enhanced features. and Dynamic gating weights are introduced during the fusion process to adjust the fusion of the two modalities, and the output is the fused feature. Specifically: First, the fusion weights are calculated, and the dynamic gating mechanism is as follows: Figure 6 As shown. To improve the fusion effect of the two modes, dynamic gating is introduced to adjust the fusion weights. W .

[0045] Visible light characteristics Context information is extracted using a context encoder for subsequent gating calculations, and is represented as follows: (27) Here, Flatten represents feature flattening, Linear represents a linear layer, Dropout represents regularization, and ReLU represents the ReLU activation function.

[0046] Simultaneously, visible light characteristics The signal-to-noise ratio (SNR) of the input features is estimated using a signal-to-noise ratio estimator, which is then used to adjust the gating weights. This is expressed as: (28) Similarly, by processing the infrared branch using the same operation, the outputs of the context encoder and the signal-to-noise ratio estimator are shown in equations (29) and (30), respectively: (29) (30) The two outputs of the two modes are concatenated by channel, as follows: (31) (32) Then, through operations such as linear layers and activation functions, the final fusion weights are obtained. W , represented as: (33) The obtained fusion weights W Multiplying the two modalities yields the fused features, represented as: (34) Step 6: Construct the neck pyramid network.

[0047] Considering that the last three layers of the feature extraction network contain richer and more useful information, the designed multi-scale feature enhancement module and adaptive fusion module are applied to these last three layers to perform feature fusion on the extracted high-order features. Since the scales of these three feature layers are different, upsampling is necessary to ensure size consistency during the stitching operation. Finally, the detection head outputs the target prediction result.

[0048] Step 7: Encapsulate the branches and modules mentioned above into a single network for end-to-end training and testing.

[0049] Step 8: Train and test the overall network encapsulated in Step 7.

[0050] Training and testing were conducted using the LLVIP and FLIR datasets, both of which contain images in pairs of visible light and infrared images. LLVIP is a large-scale visible light-infrared paired dataset developed by the Artificial Intelligence Research Team at Beijing University of Posts and Telecommunications, specifically designed for low-light vision research. This dataset contains 30,976 images, or 15,488 pairs, most of which were taken in very dark scenes, and pedestrians are labeled in the dataset. FLIR is a publicly available thermal imaging dataset provided by Teledyne FLIR, containing both daytime and nighttime scenes, and offering a large number of visible light and infrared image pairs. However, the image pairs in the original dataset were severely misaligned. Researchers have released aligned versions, which are widely used. This paper also uses the aligned version of the FLIR dataset, containing 5,142 visible light-infrared image pairs, of which 4,129 pairs were used for training and 1,013 pairs for testing. This dataset covers three object categories: "people," "vehicles," and "bicycles."

[0051] The overall loss function used by the network during training It can be represented as: (35) in, This represents the alignment loss of the CMAAR module. The bounding box regression loss represents the target location and size. This represents the classification loss for the target category.

[0052] Next, we will explain in detail the composition of these three sub-losses.

[0053] Alignment loss It is designed to optimize the alignment effect of the CMAAR module, and the input is the corrected alignment feature obtained by the CMAAR module. and The loss function consists of structural difference loss. and local structural consistency loss It consists of two parts, represented as: (36) Among them, hyperparameters and Control the balance between the two losses.

[0054] Structural difference loss The difference between two structural feature maps is calculated using the mean squared error loss, expressed as: (37) in, and These represent the corresponding pixel values ​​for visible light and infrared images, respectively. N Indicates the total number of pixels.

[0055] Local structural consistency loss Using cosine similarity To measure, it is expressed as: (38) about The calculation first divides the input feature map into small blocks, and then calculates the standardized cosine similarity of the visible light and infrared features for each block, expressed as: (39) in, B Indicates the total number of blocks. and They are respectively for the first i The block visible / infrared features, after standardization, can be represented as: (40) in, and They are the first i Block visible light / infrared features, and These are the average values ​​of the visible light and infrared features for each patch, respectively. and These are the standard deviations of the visible light and infrared features for each piece. It is a tiny positive number, to prevent the denominator from being zero, set .

[0056] Bounding box regression loss The purpose is to constrain the difference between the predicted bounding box and the ground truth bounding box, and it consists of L1 loss and GIoU loss, expressed as: (41) Among them, hyperparameters and Used to control the balance between two losses. b g , b p and b c These represent the true value, the predicted value, and the area of ​​the minimum bounding box, respectively.

[0057] Classification loss Using cross-entropy loss for calculation, it is expressed as: (42) in, p ( c ) indicates that the real sample is c The probability of a class Indicates that the network prediction sample is c The probability of a class.

[0058] Other training parameter settings are shown in Table 1. Save the trained network parameters.

[0059] Table 1. Parameter settings for the entire network training

[0060] At this point, the detection model has been built and trained, and the weight file is saved in step 8.

[0061] Step 9: When detecting both visible light and infrared images, after... Figure 1 The complete process shown, namely steps 1 to 6, uses the weight parameters trained in step 8 to detect the input image.

[0062] To demonstrate the effectiveness of the method in this embodiment, tests were conducted using the LLVIP and FLIR datasets to verify its performance in detecting small targets. The method in this embodiment was compared with other deep neural network-based algorithms (three unimodal algorithms: YOLOv5, DINO, and YOLOv11; six multimodal algorithms: CFT, YOLOFusion, ICAFSusion, LRAF-Net, DACFusion, and DAMSDet). To ensure fairness in the comparison of experimental results, the experimental data for these nine algorithms were divided using the same dataset, and the network training parameters were set identically. Tables 2 and 3 show the detection accuracy of the method in this embodiment compared to the other seven algorithms on the LLVIP and FLIR datasets, where RGB and IR represent the visible light and infrared modal images, respectively. As shown in Tables 2 and 3, the average precision mAP50 with an intersection-over-union (IoU) threshold of 0.5 achieved the best results on both the LLVIP and FLIR datasets, the average precision mAP achieved the second-best result on the LLVIP dataset, and the best result on the FLIR dataset.

[0063] Table 2 shows the detection results on the LLVIP dataset.

[0064] Table 3 shows the detection results on the FLIR dataset.

[0065] Example 2 One embodiment of the present invention provides a small target detection system based on aligned visible light and infrared images, comprising: The image acquisition module is configured to acquire visible light and infrared images of the small target to be detected; The target detection module is configured to input visible light images and infrared images into the trained detection model to detect small targets and obtain the localization and classification results of the small targets. The detection model first employs a dual-branch structure to extract multi-scale features from visible light and infrared images. During the extraction process, through interactive collaborative learning between visible light and infrared images, feature alignment, modal interaction correction, and multi-scale feature enhancement between the two modalities are achieved, resulting in enhanced visible light and infrared features. Then, feature fusion is performed, and finally, the fused features are used to locate and classify small targets.

[0066] Example 3 One embodiment of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the small target detection method based on aligned visible light and infrared images.

[0067] Example 4 In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the small target detection method based on aligned visible light and infrared images.

[0068] Example 5 One embodiment of the present invention provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the small target detection method based on aligned visible light and infrared images.

[0069] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0071] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for small object detection based on aligned visible and infrared images, characterized in that, The method comprises the following steps: acquiring a visible light image and an infrared image of a small target to be detected; inputting the visible light image and the infrared image into a trained detection model to detect the small target, and obtaining a positioning and classification result of the small target; wherein the detection model first adopts a double-branch structure to perform multi-scale feature extraction on the visible light image and the infrared image respectively, in the process of extraction, through interactive learning of the visible light image and the infrared image, feature alignment between the two modalities, modal interaction correction and multi-scale feature enhancement are realized, and then enhanced visible light features and infrared features are obtained, and finally, the enhanced features are fused, and the positioning and classification of the small target are performed.

2. The alignment-based small target detection method of visible and infrared images as claimed in claim 1, wherein, The detection model comprises a visible light branch, an infrared branch, a cross-modality alignment and adaptive resampling module, a multi-scale feature enhancement module, an adaptive fusion module, a neck and a detection head. The visible light branch and the infrared branch are used to extract multi-scale features. The cross-modality alignment and adaptive resampling module is used to perform feature alignment between the two modalities and modal interaction correction to obtain corrected and aligned single-modality features. The multi-scale feature enhancement module is used to perform multi-scale feature enhancement on the corrected and aligned single-modality features. The adaptive fusion module is used to fuse the enhanced single-modality features to obtain fused multi-modality features. The neck is used to fuse multi-modality features of different scales to obtain fused features. The detection head is used to perform positioning and classification of the small target on the fused features.

3. The alignment-based small target detection method of visible and infrared images as claimed in claim 1, wherein, The cross-modality alignment and adaptive resampling module comprises the following steps: based on the channel-level spliced visible light features and infrared features, offset and modulation coefficient are calculated through convolution layer and activation function; based on the offset and modulation coefficient, feature alignment is performed on the visible light features and infrared features through deformable convolution; multi-head cross attention is used to calculate the attention map of the visible light features and infrared features respectively, and the difference complementary features of the visible light and infrared are obtained through the attention map respectively; layer normalization, residual connection and multi-layer perception are performed on the difference complementary features to obtain corrected and aligned visible light features and infrared features.

4. The alignment-based small target detection method of visible and infrared images as claimed in claim 1, wherein, The multi-scale feature enhancement module comprises the following steps: deep features in the corrected and aligned single-modality features are extracted through convolution, batch normalization and activation function operation; channel attention and residual connection are used to enhance the deep features in the channel; multi-scale feature extraction, fusion and enhancement are performed on the channel-enhanced deep features through convolution kernels of different scales to obtain enhanced single-modality features.

5. The alignment-based small target detection method of visible and infrared images as claimed in claim 1, wherein, The adaptive fusion module comprises the following steps: context information of the single-modality features is extracted through a context encoder; a signal-to-noise ratio estimator is used to estimate the signal-to-noise ratio of the single-modality features; based on the context information and the signal-to-noise ratio, the weight for fusion is calculated; based on the weight, the two enhanced single-modality features are fused.

6. The alignment-based small target detection method of visible and infrared images as claimed in claim 1, wherein, In the training process of the detection model, a loss function is composed of an alignment loss, a bounding box regression loss and a classification loss, wherein the alignment loss includes a structure difference loss and a local structure consistency loss, the structure difference loss uses a mean square error loss, and the local structure consistency loss uses a cosine similarity to measure.

7. A small target detection system based on aligned visible and infrared images, characterized in that, The method comprises: an image acquisition module configured to acquire a visible light image and an infrared image of a small target to be detected; a target detection module configured to input the visible light image and the infrared image into a trained detection model to perform small target detection and obtain a positioning and classification result of the small target. The detection model first adopts a double-branch structure to perform multi-scale feature extraction on the visible light image and the infrared image, in the process of which, through interactive learning of the visible light image and the infrared image, feature alignment between the two modalities, modal interaction correction and multi-scale feature enhancement are realized to obtain enhanced visible light features and infrared features, then feature fusion is performed, and finally, the fused features are used to perform positioning and classification of the small target.

8. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the small target detection method based on alignment of visible light and infrared images according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium is used to store computer instructions, which, when executed by a processor, implement the small target detection method based on alignment of visible light and infrared images according to any one of claims 1-6.

10. An electronic device, comprising: The method comprises: a processor, a memory and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device performs the small target detection method based on alignment of visible light and infrared images according to any one of claims 1-6.

Images