Dynamic feature alignment bimodal fusion target detection method based on programmable gradient guidance

Abstract

The invention relates to a dynamic feature alignment bimodal fusion target detection method based on programmable gradient guidance, and the method comprises the steps: constructing a target detection network model which comprises a coarse registration module and a visible light-infrared target detection network; the coarse registration module is used for converting the visible light-infrared image pair which is obviously staggered into a visible light-infrared image pair which is weakly staggered; the visible light-infrared target detection network is composed of a double-flow feature extraction network, a cross-modal dynamic feature coupling interaction module and a depth supervision network based on gradient guidance. The double-flow feature extraction network is used for acquiring feature information of visible light and infrared images; the cross-modal dynamic feature coupling interaction module is used for dynamically aligning and fusing weak dislocation multi-scale features of the visible light-infrared image pair; the deep supervision network is used for realizing network gradient path optimization and accelerating model convergence; performing training optimization on the model; and inputting the visible light-infrared image pair into the trained model to obtain a target detection result.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a dual-modal fusion target detection method based on dynamic feature alignment guided by programmable gradients. Background Technology

[0002] Target detection technology is an important research direction in the field of computer vision and has been widely applied in various fields such as transportation, surveillance, and aviation. Single-modal detection methods are often limited by factors such as weather and occlusion, making it difficult to achieve reliable separation between the target and the background. Furthermore, their target detection performance is not outstanding in complex environments. Visible-infrared cross-modal fusion target detection technology, due to its ability to fully utilize complementary information and achieve robust all-weather detection, has gradually become a research hotspot.

[0003] Most existing visible-infrared cross-modal fusion target detection techniques are based on the assumption that visible and infrared image pairs are strictly aligned. However, due to the imaging time differences between visible and infrared sensors and the complex motion of targets in the field of view, accurate image alignment is difficult to achieve. This leads to local inconsistencies in the multimodal spatial distribution, i.e., weak misalignment. Therefore, real-world visible-infrared image pairs are often weakly misaligned. Weak misalignment of image pairs usually causes problems such as difficult model training, poor convergence, and low accuracy. However, most current bimodal detection methods assume that the image pairs are strictly aligned, which is clearly inappropriate. Furthermore, as the complexity of visible-infrared image cross-modal fusion networks increases, simple fusion methods easily lead to insufficient fusion of complementary bimodal information and often ignore the information loss that may occur during the feedforward process. This information loss will further lead to biased gradient flows being used to update the model. This will cause the deep network to establish incorrect associations between the target and the input, resulting in incorrect predictions from the model. Summary of the Invention

[0004] The purpose of this invention is to provide a dynamic feature alignment dual-modal fusion target detection method based on programmable gradient guidance, so as to solve the shortcomings of existing methods in terms of efficient and accurate cross-modal fusion and information loss, and significantly improve the performance of visible light-infrared cross-modal fusion target detection.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a dual-modal fusion target detection method based on programmable gradient-guided dynamic feature alignment, comprising the following steps:

[0006] A programmable gradient-guided dynamic feature alignment dual-modal fusion target detection network model is constructed, including a coarse registration module and a visible-infrared target detection network. The coarse registration module is used to convert significantly misaligned visible-infrared image pairs into weakly misaligned visible-infrared image pairs. The visible-infrared target detection network consists of a two-stream feature extraction network, a cross-modal dynamic feature coupling interaction module, and a gradient-guided deep supervision network. The two-stream feature extraction network is used to acquire feature information from visible and infrared images. The cross-modal dynamic feature coupling interaction module consists of a cross-modal feature enhancement component, a dynamic feature alignment component, and a bidirectional coordinate attention fusion component, used for dynamic alignment and fusion of weakly misaligned multi-scale features of visible-infrared image pairs to achieve accurate fusion and complementarity of visible and infrared dual-modal feature information. The deep supervision network introduces auxiliary branches and supervision signals to optimize network gradient paths and accelerate model convergence.

[0007] The constructed dynamic feature-aligned bimodal fusion object detection network model was trained using the training dataset to obtain network parameters, and the network parameters were optimized by constructing a loss function.

[0008] The visible light-infrared image pairs are input into a trained dynamic feature-aligned dual-modal fusion target detection network model to obtain the predicted target detection results.

[0009] Furthermore, the implementation method of the coarse registration module includes:

[0010] Step 1.1: Obtain clearly misaligned image pairs acquired using a visible-infrared dual-light device;

[0011] Step 1.2: Construct a coarse registration module based on the principle of affine transformation; specifically: first, manually select 8 pairs of relatively obvious feature correspondence points between the visible light image and the infrared image, and then use the feature points to estimate the parameters of an affine transformation matrix that includes basic geometric transformations of rotation, translation and scaling;

[0012] Step 1.3: Using visible light as the perceived image and infrared image as the reference image, apply the affine transformation matrix parameters to the visible light image to perform the corresponding affine transformation operation, thereby achieving a coarse alignment transformation from the visible light modal image to the infrared modal image.

[0013] Furthermore, the implementation method of the dual-stream feature extraction network includes:

[0014] Step 3.1: Obtain weak misalignment image pairs between the visible light image and the infrared image after coarse registration;

[0015] Step 3.2: Construct a two-stream feature extraction network for extracting features from visible light and infrared images; specifically, use convolutional blocks for feature extraction and downsampling, then utilize several residual structures to halve the feature map spatial size while doubling the number of channels, thereby obtaining feature maps of different depths; specifically including:

[0016] Step 3.2.1: Use convolutional modules with the same structure as the initial feature extractors for visible light and infrared images; each branch first passes through a convolutional block containing convolutional layers, normalization and activation functions to achieve preliminary feature mapping and spatial downsampling of the input image;

[0017] Step 3.2.2: Introduce a multi-level residual module for deep feature extraction; each branch stacks multiple residual structures sequentially to generate feature maps of different depths.

[0018] Step 3.3: Input the image pairs of the three-channel visible light image and the single-channel infrared image into the two-stream feature extraction network respectively; specifically including:

[0019] Step 3.3.1: Perform normalization and standardization operations on the visible light image and infrared image pair; specifically: normalize the pixel values ​​of the visible light and infrared images to [0,1] and perform standardization processing so that the mean pixel value is 0 and the standard deviation of the pixel value is 1, in order to meet the network input requirements;

[0020] Step 3.3.2: Perform Mosaic data enhancement on the normalized and standardized visible light and infrared images, including adjusting the image contrast, color temperature, sharpness, and using random cropping and stitching techniques to increase data diversity;

[0021] Step 3.4: Obtain the visible light multi-level features extracted by the dual-stream feature extraction network , , With infrared multi-level features , , .

[0022] Furthermore, the implementation method of the cross-modal dynamic feature coupling interaction module includes:

[0023] Step 4.1: Construct a cross-modal feature enhancement component to perform global feature coarse interaction on visible light and infrared image feature information at the same scale, obtaining globally enhanced visible light and infrared image features; specifically including:

[0024] Step 4.1.1: Obtain the visible light feature map after feature extraction by the dual-stream feature extraction network. and infrared feature map Each feature map is mapped to a set of tockens respectively. , ;

[0025] Step 4.1.2: Obtain the... and The sequence is cross-input to the CrossAtt attention mechanism as shown in Equation (1) to perform feature interaction and capture complementary information between the two modalities from a global perspective.

[0026] (1)

[0027] Where softmax() is the normalized activation function, for Projected to Query vector in dimension They are respectively Projected to Value vectors and key vectors in dimensions The dimensions of the query vector and value vector;

[0028] Similarly, Projected to In terms of dimensions, it will be used as a value vector and a key vector. Projected to The visible light image features are obtained by using the dimension as the query vector and following the calculation method of equation (1). ;

[0029] Step 4.1.3: Obtain visible light image features after global feature enhancement and infrared image features ;

[0030] Step 4.2: Input the visible light image features and infrared feature information at the same scale after coarse interaction into the dynamic feature alignment component for feature alignment to obtain the aligned visible light image features; specifically including:

[0031] Step 4.2.1: Obtain visible light image features after coarse interaction and infrared features ;

[0032] Step 4.2.2: Set the visible light mode as the sensing mode and the infrared mode as the reference mode, and set the visible light features... infrared features Subtraction, as shown in equation (2), yields the residual vector. To reduce the amount of computation;

[0033] (2)

[0034] Step 4.2.3: Input the residual vector into the convolution module and perform a 4x downsampling to reduce the dimension of the residual vector;

[0035] Step 4.2.4: Input the feature residual map after dimensionality reduction into the three parallel branches of Equation (3). Each branch consists of a global average pooling layer and a convolutional layer. The three branches regress the position offset, width scaling factor and height scaling factor respectively, which together constitute the parameters of the affine transformation matrix.

[0036] (3)

[0037] in, This is the residual vector after downsampling. This refers to the displacement along the horizontal and vertical directions. This represents the scaling ratio along the width direction. The scaling factor is along the height direction, and Avgpool() is the average pooling operation. For a 1×1 convolution with 16 input channels and 2 output channels;

[0038] Step 4.2.5: Perform affine transformation on the visible light sensing image features according to Equation (4) based on the obtained homography matrix parameters;

[0039] (4)

[0040] in, Let the coordinates of a certain pixel be . Pixel coordinates after performing an affine transformation to perceive image features;

[0041] Step 4.3: Input the aligned visible light and infrared features at the same scale into the bidirectional coordinate attention fusion component for dual-modal feature enhancement and feature fusion; specifically including:

[0042] Step 4.3.1: Apply global average pooling to each channel along the horizontal direction X and vertical direction Y to aggregate the features of each modality, and then stitch the two modality features together along a unified direction to enable the network to more accurately locate the region of interest in each modality;

[0043] (5)

[0044] (6)

[0045] in, and These are respectively the visible light and infrared characteristics following the signal. To perform average pooling along the x-direction, Averaging is performed along the y-direction; then, according to equation (6), features from the two modalities are aggregated along the same direction to obtain... and Aggregation features in direction and ;

[0046] Step 4.3.2: Perform transformation operations on the fusion features as shown in Equation (7) to generate intermediate feature maps of the fusion features in the horizontal and vertical directions in terms of spatial information;

[0047] (7)

[0048] Where F1 is a 1×1 convolution transformation function, and [ , ] is a concatenation operation along the spatial dimensions. It is a non-linear activation function. This is an intermediate feature mapping for encoding spatial information in the horizontal and vertical directions;

[0049] Step 4.3.3: Along the spatial dimension It is divided into two separate tensors along the x and y directions. and This is used to capture the long-distance dependencies of the fused feature maps in various spatial directions; then, two 1×1 convolutions are used to calculate the fusion attention weights.

[0050] (8)

[0051] in, It is the sigmoid activation function;

[0052] Step 4.3.4: After performing attention-weighted feature transformation on the fusion attention weights and the visible light input features and infrared input features respectively, the fusion is performed to obtain the final fusion feature output. This enhanced fusion method captures the orientation and position information of the target from multiple angles through cross-modal and cross-channel feature interaction. This not only enhances the local feature expression of the fusion features, but also suppresses redundant information, providing a more robust feature representation and attention guidance for subsequent target recognition and localization in the model.

[0053] Furthermore, the implementation method of the gradient-guided deep supervised network includes:

[0054] Step 5.1: Embed the cross-modal dynamic feature coupling and interaction module into the dual-stream feature extraction network to achieve alignment and fusion of shallow and deep information of multi-scale features in visible light and infrared images;

[0055] Step 5.1.1: Utilize the visible light multi-level features obtained in Step 3.4 , , With infrared multi-level features , , As input to the cross-modal dynamic feature coupling interaction module;

[0056] Step 5.1.2: Input the visible light feature map and infrared feature map at the same scale into the cross-modal dynamic feature coupling interaction module to obtain the fused features at different scales. , , ;

[0057] Step 5.1.3: Fusion output features of cross-scale channels under the visible light branch , , Cross-scale channel fusion output features under infrared branch , , The same operation as above is used to obtain fusion features at different scales. , , ;

[0058] Step 5.2: Merge features at the same scale , , and , , Perform the Concat operation to efficiently aggregate bimodal feature information at various scales;

[0059] Step 5.3: Input the total features of the dual-modal fusion at each scale into a detection head containing IOU queries to form a dual-modal fusion branch for visible light and infrared images; connect the dual-stream feature extraction networks of the original visible light and infrared images to a detection head containing an IOU query mechanism to form a visible light auxiliary branch and an infrared auxiliary branch, thereby generating programmable gradient information. By programming the gradient information at different semantic levels, information loss caused by dual-modal data during the feedforward process can be avoided.

[0060] Furthermore, the method of training the constructed dynamic feature-aligned bimodal fusion object detection network model using the training dataset to obtain network parameters, and optimizing the network parameters by constructing a loss function, includes:

[0061] Step 6.1: Construct the loss function as shown in Equation (9) for the visible light auxiliary branch, infrared auxiliary branch, and fusion branch using the RT-DETR-based loss function method:

[0062] (9)

[0063] in, For IOU-aware classification loss in RT-DETR, Loss from bounding box regression The loss consists of the loss and the generalized IOU loss. This is the loss used for noise reduction training;

[0064] Step 6.2: Construct the overall weighted loss function as shown in Equation (10) according to the contribution of each branch, based on the losses of the visible light auxiliary branch, infrared auxiliary branch and fusion branch, so as to achieve efficient and dense supervision of the entire network;

[0065] (10)

[0066] (11)

[0067] in, and These represent the total loss values ​​on the infrared and visible light auxiliary branches, respectively. They are mainly responsible for the dense supervision of the encoder on the auxiliary branches, optimizing the gradient calculation on the auxiliary branches, and can also perform dual-path coordination and guidance with the fusion main branch to enhance the fusion representation. This represents the loss value on the fusion branch, which is mainly responsible for the dense supervision and optimization of the fusion performance of the encoder on the main branch.

[0068] The present invention also provides a computer device, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which implement the above-described method when executed by the processor.

[0069] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the above-described method.

[0070] Compared with existing technologies, this invention has the following advantages: This invention combines visible light image target detection with infrared image target detection, obtaining dual-modal complementary features of visible light and infrared images. This overcomes the limitations of single-modal target detection in complex and occluded scenes, achieving robust real-time detection in all weather conditions. Simultaneously, by introducing a cross-modal dynamic feature coupling and interaction module, this invention achieves efficient alignment of misaligned features between visible light and infrared images during network training, significantly reducing the impact of image misalignment on the dual-modal network detection results and effectively improving the model's detection accuracy. Furthermore, by introducing a gradient-guided deep supervised network, this invention effectively avoids information loss in both visible light and infrared images within the dual-modal network, further enhancing network training stability and accelerating the convergence speed of the dual-modal network. Attached Figure Description

[0071] Figure 1 This is an overall flowchart of the method of the present invention;

[0072] Figure 2 This is an overall network structure diagram of an embodiment of the present invention;

[0073] Figure 3 This is a structural diagram of the cross-modal dynamic feature coupling interaction module in an embodiment of the present invention;

[0074] Figure 4 This is a structural diagram of the bidirectional coordinate attention fusion module in an embodiment of the present invention. Detailed Implementation

[0075] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0076] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0077] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0078] like Figure 1As shown in the figure, this embodiment provides a target detection method based on programmable gradient-guided dynamic feature alignment dual-modal fusion. It is mainly implemented by constructing and training a target detection network model based on programmable gradient-guided dynamic feature alignment dual-modal fusion. The specific implementation process is as follows.

[0079] Step 1: Construct a coarse registration module to convert obviously misaligned visible-infrared image pairs into weakly misaligned visible-infrared image pairs.

[0080] Step 1.1: Obtain clearly misaligned image pairs based on visible light-infrared dual-light equipment.

[0081] Step 1.2: Construct a coarse registration module based on the principle of affine transformation; specifically, firstly, manually select 8 pairs of relatively obvious feature pairs from the visible light image and infrared image pair, and then use the feature points to estimate the parameters of an affine transformation matrix that includes basic geometric transformations of rotation, translation and scaling.

[0082] Step 1.3: Using the visible light image as the sensing image and the infrared image as the reference image, apply the affine transformation matrix parameters to the visible light image to perform the corresponding affine transformation operation, thereby achieving a coarse alignment transformation from the visible light modal image to the infrared modal image.

[0083] In this embodiment, since the image pairs directly acquired from the visible-infrared dual-spectrum device have significant resolution differences, directly inputting them into the dual-modal network may cause problems such as training instability. Therefore, a coarse registration module based on the affine transformation principle is designed to achieve coarse alignment with the same resolution. The coarse registration module includes a series of steps: reading the visible-infrared image pair, calculating matrix parameters, performing affine transformation, and outputting the transformed image. The matrix parameters are calculated by manually selecting more than four feature point pairs to calculate the homography matrix parameters. Since the image pairs are all acquired from the same dual-spectrum device, their resolution differences and field-of-view differences are consistent, and the matrix parameters calculated earlier can be applied to the subsequent image pairs for processing. Due to the low accuracy of manual registration and the possible time difference between the visible light camera and the infrared camera when acquiring images, there is a slight misalignment phenomenon in the coarsely registered image pairs, which lays the data foundation for the subsequent feature alignment network design.

[0084] Step 2: Construct a visible-infrared target detection network consisting of a two-stream feature extraction network, a cross-modal dynamic feature coupling and interaction module, and a gradient-guided deep supervision network. The specific implementation steps are as follows.

[0085] Step 3: Construct a dual-stream feature extraction network for extracting feature information from visible light and infrared images, and obtain feature information from visible light and infrared images respectively.

[0086] Step 3.1: Obtain the weak misalignment image pair of the visible light image and infrared image after coarse registration.

[0087] Step 3.2: Construct a two-stream feature extraction network for extracting visible light and infrared images; specifically, use convolutional blocks for feature extraction and downsampling, and then use several residual structures to halve the feature map spatial size while doubling the number of channels, thereby obtaining feature maps of different depths.

[0088] Step 3.2.1: Use convolutional modules with the same structure as the initial feature extractors for visible light and infrared images; each branch first passes through a convolutional block containing convolutional layers, normalization and activation functions to achieve preliminary feature mapping and spatial downsampling of the input image.

[0089] Step 3.2.2: Introduce a multi-level residual module for deep feature extraction; each branch stacks multiple residual structures in sequence to finally generate feature maps of different depths.

[0090] In this embodiment, to ensure network depth while avoiding gradient explosion or vanishing problems, and considering the real-time performance of the bimodal network model, the bimodal network model should be as lightweight as possible without significant loss of accuracy. Therefore, the ResNet-18 structure is used as the initial feature extractor. Figure 2 As shown. Specifically, ResNet-18 is a deep residual network containing 18 convolutional layers. The first layer is a convolutional operation with 64 7×7 kernels and a stride of 2, used to extract primary features and halve the size of the input image. A 3×3 max-pooling layer further halves the feature map size. The second, third, fourth, and fifth layers are all composed of stacked residual modules with the same structure, but the number of modules in each layer varies. Its core residual module consists of two 3×3 convolutional layers and residual connections. By directly superimposing the residual mapping between the input and the convolutional output, it effectively alleviates the gradient vanishing and network degradation problems in deep network training, while preserving the integrity of feature information. The residual module in the second layer does not perform downsampling, and the residual connections use identity mapping to ensure that the input and output dimensions are consistent. The first residual module in the third, fourth, and fifth layers needs to be downsampled: the main path halves the feature map size through a 3×3 convolution with a stride of 2, while a 1×1 convolution with a stride of 2 is introduced at the residual connection to adjust the number of channels and spatial size of the input features to match the dimension of the output features. This achieves the deepening of the number of feature channels and the halving of the size, enabling the network to extract rich multi-level features.

[0091] Step 3.3: Input the three-channel visible light image and the single-channel infrared image into the two-stream feature extraction network respectively.

[0092] Step 3.3.1: Perform normalization and standardization operations on the visible light image and infrared image pair; specifically, normalize the pixel values ​​of the visible light and infrared images to [0,1] and perform standardization processing so that the mean pixel value is 0 and the standard deviation of the pixel value is 1, in order to meet the network input requirements.

[0093] Step 3.3.2: Perform Mosaic data enhancement on the normalized and standardized visible light and infrared images; specific enhancement methods include adjusting the image contrast, color temperature, sharpness, and random cropping and stitching techniques to increase data diversity;

[0094] In this embodiment, paired visible light and infrared weak misalignment images are first acquired. Each pixel in the visible light image is composed of 3 channels, while each pixel in the infrared image is composed of a single channel. The infrared image is copied to 3 channels, and the dimensions of the paired visible light and infrared images are normalized to a format of 3×640×640 (channel × length × width) before being input into the feature extraction network.

[0095] Step 3.4: Obtain the visible light multi-level features extracted by the dual-stream feature extraction network , , With infrared multi-level features , , .

[0096] In this embodiment, multi-level features , , and , , These are the features from the third to the fifth layers of the two-stream feature extraction network, respectively, using the output of the ResNet-18 initial feature extractor.

[0097] Step 4: Construct a cross-modal dynamic feature coupling and interaction module. This module consists of a cross-modal feature enhancement component, a dynamic feature alignment component, and a bidirectional coordinate attention fusion component. It is used to dynamically align and fuse weakly misaligned multi-scale features of visible light and infrared images, thereby achieving accurate fusion and complementarity of visible light and infrared dual-modal feature information.

[0098] Step 4.1: Construct a cross-modal feature enhancement component to perform global feature coarse interaction on visible light and infrared image feature information at the same scale to obtain globally enhanced visible light image features and infrared image features.

[0099] Step 4.1.1: Obtain the visible light feature map after feature extraction by the dual-stream feature extraction network. and infrared feature map Each feature map is mapped to a set of tockens respectively. , .

[0100] Step 4.1.2: Obtain the... and The sequence is cross-input into the CrossAtt attention mechanism as shown in Equation (1) for feature interaction, capturing complementary information between the two modalities from a global perspective;

[0101] (1)

[0102] Where softmax() is the normalized activation function, for Projected to Query vector in dimension They are respectively Projected to Value vectors and key vectors in dimensions To query the dimensions of the vector and value vector.

[0103] Similarly, Projected to In terms of dimensions, it will be used as a value vector and a key vector. Projected to The visible light image features are obtained by using the dimension as the query vector and following the calculation method of equation (1). .

[0104] Step 4.1.3: Obtain visible light image features after global feature enhancement and infrared image features .

[0105] In this embodiment, as Figure 3 The left side of the diagram illustrates the specific structure of the cross-modal feature enhancement component. Due to the difference in field of view between visible light and infrared, and the possibility that targets may only appear in a single modality under certain occlusion or low-light conditions, directly aligning and fusing the two modalities not only affects alignment accuracy but may also introduce noise. Therefore, designing a cross-modal feature enhancement component allows for the capture of complementary information between the two modalities from a global perspective. The visible light image branch and the infrared image branch achieve coarse interaction and enhancement of cross-modal features through a parallel cross-modal attention mechanism.

[0106] Step 4.2: Input the visible light image features and infrared feature information at the same scale after coarse interaction into the dynamic feature alignment component for feature alignment; obtain the aligned visible light image features.

[0107] Step 4.2.1: Obtain visible light image features after coarse interaction and infrared features .

[0108] Step 4.2.2: Set the visible light mode as the sensing mode and the infrared mode as the reference mode, and set the visible light features... infrared features Subtracting them yields the residual vector. With less computation;

[0109] (2)

[0110] In this embodiment, infrared image mode imaging is chosen because it is more stable than visible light image mode imaging and less affected by weather and day / night light conditions. As a reference mode, while the visible light image mode To perceive the image, the stable infrared image mode is used as the reference mode, and the visible light image mode is transformed into the coordinate system of the reference mode through affine transformation, thereby ensuring the quality of fusion detection in all weather conditions.

[0111] Step 4.2.3: Input the residual vector into the convolution module and perform a 4x downsampling to reduce the dimension of the residual vector.

[0112] Step 4.2.4: Input the feature residual map after dimensionality reduction into the three parallel branches of Equation (3). Each branch consists of a global average pooling layer and a convolutional layer. The three branches regress the position offset, width scaling factor and height scaling factor, which together constitute the parameters of the affine transformation matrix.

[0113] (3)

[0114] in, This is the residual vector after downsampling. This refers to the displacement along the horizontal and vertical directions. This represents the scaling ratio along the width direction. The scaling factor is along the height direction, and Avgpool() is the average pooling operation. It is a 1×1 convolution with 16 input channels and 2 output channels.

[0115] Step 4.2.5: Perform affine transformation on the visible light sensing image features according to Equation (4) based on the obtained homography matrix parameters.

[0116] (4)

[0117] in, Let the coordinates of a certain pixel be . The pixel coordinates after performing an affine transformation to perceive image features.

[0118] In this embodiment, as Figure 3 The right side shows the specific structure of the dynamic feature alignment component. It first subtracts the corresponding visible light image features and infrared image features at the same scale to obtain a new feature map. To reduce computational burden, the transformation matrix parameters, including the horizontal offset parameter, are then calculated through a three-branch structure containing a global average pooling layer and a convolutional layer. Vertical offset parameter Scaling parameters along the horizontal direction Scaling parameters along the vertical direction Finally, the transformation matrix parameters are used to map the perceived features from the original coordinate system to the reference feature coordinate system.

[0119] Step 4.3: Input the aligned visible light features and infrared features at the same scale into the bidirectional coordinate attention fusion component for bimodal feature enhancement and feature fusion.

[0120] Step 4.3.1: The aligned visible light features and the reference modal infrared features are aggregated by global average pooling along the horizontal direction X and the vertical direction Y for each channel. Then, the two modal features are stitched together along a unified direction. This helps the network to more accurately locate the region of interest in each modality.

[0121] (5)

[0122] (6)

[0123] in, and These are respectively the visible light and infrared characteristics following the signal. To perform average pooling along the x-direction, Averaging is performed along the y-direction; then, according to equation (6), features from the two modalities are aggregated along the same direction to obtain... and Aggregation features in direction and .

[0124] Step 4.3.2: Perform transformation operations on the fusion features as shown in Equation (7) to generate intermediate feature maps of the fusion features in the horizontal and vertical directions in terms of spatial information;

[0125] (7)

[0126] Where F1 is a 1×1 convolution transformation function, and [ , ] is a concatenation operation along the spatial dimensions. It is a non-linear activation function. This is an intermediate feature mapping for encoding spatial information in the horizontal and vertical directions.

[0127] Step 4.3.3: Along the spatial dimension It is divided into two separate tensors along the x and y directions. and This is used to capture the long-distance dependencies of the fused feature maps in various spatial directions; then, two 1×1 convolutions are used to calculate the fusion attention weights.

[0128] (8)

[0129] In the formula It is the sigmoid activation function.

[0130] In this embodiment, as Figure 4 The diagram shows the specific structure of the bidirectional coordinate attention fusion component. The dynamic feature alignment component implements the mapping and alignment operation from the visible light image feature coordinate system to the infrared image feature coordinate system, while the bidirectional coordinate attention component aims to achieve deep fusion of aligned visible light and infrared features from both horizontal and vertical dimensions. First, the aligned features... With reference modal features Global average pooling is applied to each channel along both the horizontal (X) and vertical (Y) coordinates to aggregate features from each modality. This helps the network more accurately locate regions of interest within each modality. Then, the two modalities are added together along the same direction to obtain the desired feature. , Next, the initial fusion was... , Perform a concatenation operation, then use a 1×1 convolutional transform function F1 to transform it, generating fused feature maps in the horizontal and vertical directions of spatial information. Next, along the spatial dimensions... Split into two separate tensors and This is used to capture long-range dependencies in the fused feature maps across different spatial directions. Finally, the fused attention weights are respectively applied to the input features. and Attention-weighted feature transformation is performed before fusion. This enhanced fusion method captures the orientation and location information of the target from multiple angles through cross-modal and cross-channel feature interaction. This not only enhances the local feature expression of the fused features but also suppresses redundant information, providing more robust feature representation and attention guidance for subsequent target recognition and localization in the model.

[0131] Step 5: Construct a gradient-guided deep supervised network, introducing auxiliary branches and supervisory signals; this will enable network gradient path optimization and accelerate model convergence.

[0132] Step 5.1: Embed the cross-modal dynamic feature coupling and interaction module into the dual-stream feature extraction network to achieve alignment and fusion of shallow and deep information of multi-scale features in visible light and infrared images.

[0133] Step 5.1.1: Utilize the visible light multi-level features obtained in Step 3.4 , , With infrared multi-level features , , As input to the cross-modal dynamic feature coupling interaction module.

[0134] Step 5.1.2: Input the visible light feature map and infrared feature map at the same scale into the cross-modal dynamic feature coupling interaction module to obtain the fused features at different scales. , , .

[0135] Step 5.1.3: Fusion output features of cross-scale channels under the visible light branch , , Cross-scale channel fusion output features under infrared branch , , The same operations described above are used to obtain fusion features at different scales. , , .

[0136] In this embodiment, as Figure 2 The diagram shows the specific structure of a programmable gradient-guided dynamic feature alignment dual-modal fusion target detection network. Both the visible light image branch and the infrared image feature extraction branch in the dual-modal network are built based on the single-modal network model RT-DETR, mainly including an initial feature extractor based on ResNet-18 and cross-scale channel fusion. In the dual-stream feature extraction stage, the visible light image branch and the infrared image branch generate visible light feature maps at various scales. ) and infrared feature map ( The visible light image features and infrared image features at the same scale are then input into the cross-modal dynamic feature coupling interaction module (CDFC) for feature alignment and feature fusion. Next, in the single-modal branch, to combine features from different scales and maintain a balance between high-level semantic information and low-level detail features, the feature maps from each stage of the initial feature extractor output are also input into the cross-scale channel fusion module for single-modal feature enhancement. Finally, the output is a visible light feature map corresponding to the scale of the feature map output from the initial feature extractor. ) and infrared feature map ( Meanwhile, visible light image features and infrared image features at the same scale are input to the cross-modal dynamic feature coupling interaction module (CDFC) for feature alignment and feature fusion.

[0137] Step 5.2: Merge features at the same scale , , and , , Perform the Concat operation to efficiently aggregate bimodal feature information at various scales.

[0138] Step 5.3: Input the total features of the dual-modal fusion at each scale into a detection head containing IOU queries to form a dual-modal fusion branch for visible light and infrared images; connect the original dual-stream feature extraction networks of visible light and infrared images to a detection head containing an IOU query mechanism to form a visible light auxiliary branch and an infrared auxiliary branch, thereby generating programmable gradient information. By programming the gradient information at different semantic levels, information loss caused by dual-modal data during the feedforward process can be avoided.

[0139] In this embodiment, as Figure 2The diagram illustrates the specific structure of a dynamic feature alignment dual-modal fusion target detection network based on programmable gradient guidance. This dual-modal fusion network comprises three detection heads, corresponding to a visible light and infrared image dual-modal fusion branch, a visible light single-modal branch, and an infrared single-modal branch, respectively. Each branch employs the same detection head based on the RT-DETR model and incorporating an IOU query mechanism. The input to the fusion branch detection head is a feature map fused from the visible light and infrared features output from the main feature extraction stage and the cross-scale channel fusion stage. The input to the visible light and infrared single-modal branch detection heads is the cross-scale channel fusion features. Notably, during network training, the visible light and infrared branches are innovatively used as reversible auxiliary branches: these branches can generate programmable gradient information (PIG). By programming gradient information at different semantic levels, information loss during the feedforward process of dual-modal data can be avoided. By assisting the main branch in coordinating and guiding the correction of misalignment in image pairs, the convergence speed of the dual-modal detector training is accelerated. It is worth noting that reparameterization techniques can be used to remove auxiliary checks during the inference testing phase.

[0140] Step 6: Train the dynamic feature alignment bimodal fusion target detection network based on programmable gradient guidance using the training dataset to obtain the network training parameters; and design an overall loss function to optimize the detection network parameters.

[0141] Step 6.1: The loss function based on RT-DETR is used to design the loss function for the visible light auxiliary branch, the infrared auxiliary branch and the fusion branch respectively, as shown in equation (9);

[0142] (9)

[0143] in, For IOU-aware classification loss in RT-DETR, Loss from bounding box regression The loss consists of the loss and the generalized IOU loss. This is the loss used for noise reduction training.

[0144] In this embodiment, the total loss function is designed primarily based on the loss function in the RT-DETR model. Specifically, the classification loss... Cross-entropy loss is used to measure the difference between the model's predicted class labels and the true labels. Each predicted bounding box has a corresponding class score vector, representing the probability that the box belongs to each class. Bounding box loss. Using generalized IOU loss can more accurately reflect the degree of overlap and positional relationship between bounding boxes. Denoising training loss can improve the quality of matched samples and accelerate the convergence speed of training.

[0145] Step 6.2: Design the overall weighted loss function as shown in equation (10) according to the contribution of each branch, based on the losses of the visible light auxiliary branch, infrared auxiliary branch and fusion branch, so as to achieve efficient and dense supervision of the entire network;

[0146] (10)

[0147] (11)

[0148] in, and It represents the total loss value on the infrared and visible light auxiliary branches, and is mainly responsible for the dense supervision of the encoder on the auxiliary branches. While optimizing the gradient calculation on the auxiliary branches, it can also perform dual-path coordination and guidance with the fusion main branch to enhance the fusion representation. This represents the loss value on the fusion branch, which is mainly responsible for the dense supervision and optimization of the fusion performance of the encoder on the main branch.

[0149] Step 7: Input the image pairs of the visible light image and the infrared image into the trained and optimized dynamic feature alignment dual-modal fusion target detection network model to obtain the predicted target detection results.

[0150] In this embodiment, after the bimodal network is trained, the overall network parameters are saved. A pair of visible light and infrared images containing the target are input for testing, and the target bounding box coordinates and category information are obtained as the detection results.

[0151] Table 1. Comparison of pedestrian target detection results under visible light-infrared dual-mode imaging.

[0152] Methods Modality mAP50 mAP75 mAP RT-DETR-r18 RGB 58.4 22.1 27.3 Yolov5 RGB 57.1 20.6 25.5 Yolov9 RGB 62.6 23.5 29.5 RT-DETR-r18 IR 73 34.3 38.3 Yolov5 IR 74.8 32.6 36.5 Yolov9 IR 74.1 33.1 38 SLBAF-Net RGB+IR 71.5 27.4 34.4 CFT RGB+IR 78.5 35.5 39.6 CMA-Det RGB+IR 75.8 32.2 37.9 ICAFusion RGB+IR 79.2 35.8 40.3 MambaDFuse RGB+IR 78.3 34.9 40.1 CMADet RGB+IR 75.4 31.5 36.8 Example 1 RGB+IR 81.5 37.8 42.4

[0153] To further demonstrate the superiority of this invention, it was experimentally compared and evaluated with existing mainstream single-modal target detection methods and visible-infrared dual-modal target detection methods on the typical visible-infrared dual-modal weak misalignment public dataset FLIR. The comparison results are shown in Table 1. This invention uses a ResNet-18 architecture. The method was compared with three typical single-modal target detection methods, seven mainstream dual-modal methods, and inverse methods. The evaluation metrics used were three target detection metrics introduced by MS-COCO: mean Average Precision (mAP), mAP50, and mAP75, with higher values ​​being better. The results show that this method not only significantly outperforms single-modal methods but also has significant advantages over mainstream visible-infrared dual-modal methods, resulting in a significant improvement in target detection accuracy.

[0154] This embodiment also provides a computer device, including: at least one processor, at least one memory, and computer program instructions stored in the memory, which implement the above-described method when executed by the processor.

[0155] This embodiment also provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method.

[0156] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0158] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0160] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A dynamic feature alignment bimodal fusion object detection method based on programmable gradient guidance, characterized in that, The method comprises the following steps: A dynamic feature alignment bimodal fusion target detection network model based on programmable gradient guidance is constructed, comprising a coarse registration module and a visible light-infrared target detection network; the coarse registration module is used for converting a visible light-infrared image pair with obvious misalignment into a visible light-infrared image pair with weak misalignment; the visible light-infrared target detection network is composed of a double-flow feature extraction network, a cross-modal dynamic feature coupling interaction module and a gradient guidance-based deep supervision network; the double-flow feature extraction network is used for acquiring feature information of visible light images and infrared images; the cross-modal dynamic feature coupling interaction module is composed of a cross-modal feature enhancement component, a dynamic feature alignment component and a bidirectional coordinate attention fusion component, and is used for dynamically aligning and fusing weak misalignment multi-scale features of the visible light-infrared image pair, so as to realize fusion and complementation of visible light and infrared bimodal feature information; the deep supervision network introduces an auxiliary branch and a supervision signal, so as to realize network gradient path optimization and accelerate model convergence; The dynamic feature alignment bimodal fusion target detection network model constructed is trained by using a training data set, so as to obtain network parameters, and the network parameters are optimized by constructing a loss function; The visible light-infrared image pair is input into the trained dynamic feature alignment bimodal fusion target detection network model, so as to obtain a predicted target detection result.

2. The dynamic feature alignment dual-modal fusion object detection method based on programmable gradient guidance according to claim 1, characterized in that, The implementation method of the coarse registration module comprises: Step 1.1: acquiring a visible light-infrared double-light equipment-based image pair with obvious misalignment; Step 1.2: constructing a coarse registration module based on an affine transformation principle; specifically, first, 8 relatively obvious feature point pairs in the visible light image and the infrared image pair are manually selected, and then an affine transformation matrix parameter containing rotation, translation and scaling basic geometric transformation is estimated by using the feature points; Step 1.3: taking the visible light image as a perception image and the infrared image as a reference image, applying the affine transformation matrix parameter to the visible light image to perform corresponding affine transformation operation, so as to realize coarse alignment transformation of the visible light modal image to the infrared modal image.

3. The programmable gradient guidance based dynamic feature alignment bimodal fusion object detection method according to claim 1, characterized in that, The implementation method of the double-flow feature extraction network comprises: Step 3.1: acquiring a weak misalignment image pair of the visible light image and the infrared image after image coarse registration; Step 3.2: constructing a double-flow feature extraction network for extracting features of the visible light image and the infrared image; specifically, a convolution block is used for feature extraction and down-sampling, and then a plurality of residual structures are used to realize halving of the spatial size of the feature map and doubling of the channel number, so as to obtain feature maps of different depths; specifically, it comprises: Step 3.2.1: using a convolution module with the same structure as the initial feature extractor for the visible light image and the infrared image; each branch first passes through a convolution block containing a convolution layer, normalization and an activation function in series, so as to realize preliminary feature mapping and spatial down-sampling of the input image; Step 3.2.2: introducing a multi-level residual module for deep feature extraction; each branch is sequentially stacked with a plurality of residual structures, and finally generates feature maps of different depths; Step 3.3: input the image pair of the visible light image with three channels and the infrared image with a single channel into the dual-flow feature extraction network respectively; specifically including: Step 3.3.1: perform normalization and standardization operations on the visible light image and the infrared image pair; specifically, normalize the pixel values of the visible light and infrared images to [0, 1] and perform standardization processing to make the pixel value mean 0 and the pixel value standard deviation 1, to adapt to the network input requirements; Step 3.3.2: perform Mosaic data enhancement on the normalized and standardized visible light image and infrared image, including adjusting the contrast, color temperature, sharpness of the image, and using random cropping and splicing technology to increase data diversity; Step 3.4: Obtain the visible multi-level features extracted by the dual-stream feature extraction network 、 、 and the infrared multi-level features 、 、 .

4. The dynamic feature alignment dual-modal fusion object detection method based on programmable gradient guidance according to claim 3, characterized in that, The implementation method of the cross-modal dynamic feature coupling interaction module includes: Step 4.1: construct a cross-modal feature enhancement component to perform global feature coarse interaction on the visible light and infrared image feature information at the same scale, and obtain the visible light image feature and the infrared image feature after global feature enhancement; specifically including: Step 4.1.1: Obtain the visible light feature map after feature extraction by the dual-flow feature extraction network and the infrared feature map , map each feature map to a set of tockens respectively , ; Step 4.1.2: The obtained and The sequence is cross-input to the CrossAtt attention mechanism as formula (1) to interact features and capture complementary information between the two modalities from a global perspective. (1) where softmax() is a normalization activation function, is projected onto dimension of the query vector, are projected onto dimension of the value vector and the key vector, is the dimension of the query vector and the value vector; Similarly, the visible light image feature is obtained according to the calculation method of formula (1) as a query vector projected onto the dimension as the value vector and the key vector . . . . ; Step 4.1.3: Obtain the visible light image features enhanced by global features and infrared image features ; Step 4.2: input the visible light image feature and the infrared feature information at the same scale after completing the coarse interaction into the dynamic feature alignment component for feature alignment, and obtain the aligned visible light image feature; specifically including: Step 4.2.1 : Obtain visible light image features after coarse interaction and infrared features ; Step 4.2.2: Set the visible light modality as the perceptual modality, the infrared modality as the reference modality, subtract the visible light features from the infrared features as shown in equation (2) to obtain the residual vector to reduce the amount of computation; (2) Step 4.2.3: input the residual vector into the convolution module for 4 times down-sampling to reduce the dimension of the residual vector; Step 4.2.4: input the reduced feature residual map into three parallel branches as formula (3), each branch is composed of a global average pooling layer and a convolution layer; the three branches regress the position offset, width scaling factor and height scaling factor respectively to form the parameters of the affine transformation matrix; (3) wherein, is a down-sampled residual vector, is a displacement amount along a horizontal direction and a vertical direction, is a scaling ratio along a width direction, is a scaling ratio along a height direction, and Avgpool() is an average pooling operation, is a 1x1 convolution with 16 input channels and 2 output channels; Step 4.2.5: perform affine transformation on the visible light perception image feature according to the obtained homography matrix parameters according to formula (4); (4) wherein, is a pixel position coordinate for a certain pixel position, is a pixel coordinate after performing an affine transformation for the perceived image feature; Step 4.3: input the visible light feature and the infrared feature at the same scale after completing the alignment into the bidirectional coordinate attention fusion component for dual-modal feature enhancement and feature fusion; specifically including: Step 4.3.1: use global average pooling to aggregate each channel of the aligned visible light feature and the reference modal infrared feature along the horizontal direction X and the vertical direction Y, and then concatenate the two modal features along the unified direction, so that the network can more accurately locate the region of interest in each modal; (5) (6) wherein, and are the visible and infrared features respectively after the following operations, is the average pooling along the x direction, is the average pooling operation along the y direction; then according to formula (6), the features of the two modalities are aggregated along the same direction to obtain and aggregated features in the direction and ; Step 4.3.2: transform the fused feature to generate the intermediate feature map in the horizontal direction and the vertical direction according to formula (7); (7) wherein F1 is a 1x1 convolutional transformation function, [, ] is a concatenate operation along the spatial dimension, is a nonlinear activation function, is an intermediate feature map encoding the spatial information in horizontal and vertical directions; Step 4.3.3: split along the spatial dimension into two separate tensors in the x and y directions and to capture long-range dependencies of fused feature maps in each spatial direction; and two 1x1 convolutions are used to compute the fused attention weights; (8) wherein, sigmoid is a sigmoid activation function; Step 4.3.4: fuse the fused attention weight with the visible light input feature and the infrared input feature after attention weighted feature transformation, to obtain the final fused feature output.

5. The programmable gradient guidance based dynamic feature alignment dual-modal fusion object detection method according to claim 4, characterized in that, The implementation method of the gradient-guided deep supervision network includes: Step 5.1: embed the cross-modal dynamic feature coupling interaction module into the dual-flow feature extraction network to realize the alignment and fusion of visible light and infrared image multi-scale feature shallow and deep information; Step 5.1.1: Visible light multi-level features from the visible light images acquired in Step 3.4 , , and infrared multi-level features , , as inputs to the cross-modal dynamic feature coupling interaction module; Step 5.1.2: input the visible light feature map and the infrared feature map under the same scale into the cross-modal dynamic feature coupling interaction module to obtain the fused feature under different scales , , ; Step 5.1.3: fusing the output features under the visible light branch and the output features under the infrared branch across scales , , Step 5.1.3: fusing the output features under the visible light branch and the output features under the infrared branch across scales , , Step 5.1.3: fusing the output features under the visible light branch and the output features under the infrared branch across scales , , ; Step 5.2: fusing features of the same scale , , with , , performing a Concat operation to efficiently aggregate the bimodal feature information of each scale; Step 5.3: The total feature of the bimodal fusion under each scale is input into the detection head containing the IOU query to form the visible light image and infrared image bimodal fusion branch; the original visible light image and infrared image double-flow feature extraction network is connected to a detection head containing an IOU query mechanism to form a visible light auxiliary branch and an infrared auxiliary branch to generate programmable gradient information, and by programming the gradient information of different semantic levels, the information loss generated in the feedforward process of the bimodal data is avoided.

6. The programmable gradient guidance based dynamic feature alignment dual-modal fusion object detection method according to claim 1, characterized in that, The constructed dynamic feature alignment bimodal fusion target detection network model is trained by using the training data set to obtain network parameters, and the network parameters are optimized by constructing a loss function, and the implementation method comprises: Step 6.1: The loss function method based on RT-DETR is used to construct the loss function of the visible light auxiliary branch, the infrared auxiliary branch and the fusion branch as formula (9): (9) wherein, is an IOU-aware classification loss in RT-DETR, is a bounding box regression loss consists of a loss and a generalized IOU loss, is a loss for denoising training; Step 6.2: The losses of the visible light auxiliary branch, the infrared auxiliary branch and the fusion branch are constructed according to the contribution degree of each branch to construct the overall weighted loss function as formula (10) to realize efficient dense supervision of the overall network; (10) (11) wherein, and respectively represent the total loss value on the infrared and visible light auxiliary branch, mainly responsible for the dense supervision of the encoder on the auxiliary branch, and can also be guided by the dual-channel coordination of the fusion main branch to enhance the fusion representation while optimizing the gradient calculation on the auxiliary branch; represents the loss value on the fusion branch, mainly responsible for the dense supervision of the encoder on the main branch and the optimization of the fusion efficiency.

7. A computer device, comprising: comprise: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of any one of claims 1-6.

8. A computer-readable storage medium having stored thereon computer program instructions, wherein, when the computer program instructions are executed by the processor, the method of any one of claims 1-6 is implemented.

Images