Multimodal image feature enhancement fusion target detection method and device of joint pre-training model

Abstract

The application discloses a kind of multi-modal image feature enhancement fusion target detection method and device of combined pre-training model, belong to the field of target detection in the field of computer vision, including: the general features of two kinds of images are extracted using the image encoder of pre-training model, and the modal specific features of each are extracted by depth learning backbone network;To infrared image, in the forward transmission process of each level feature of backbone network, introduce adaptive gradient feature encoder to extract edge information;The extracted general features and specific features are interactively fused;The specific features extracted at different levels of different modalities are input into the multi-modal attention fusion module for hierarchical fusion;The last layer of general features and the last layer of specific features from the same modality are input into the general feature and specific feature fusion module;The multi-layer features formed after fusion are input into the target detection algorithm to obtain the final multi-modal target detection result.The application improves the accuracy and robustness of target detection.

Description

Technical Field

[0001] This invention relates to the field of target detection in computer vision, and more specifically, to a multimodal image feature enhancement and fusion target detection method and apparatus using a joint pre-trained model. Background Technology

[0002] Object detection is one of the core tasks in the field of computer vision, aiming to automatically identify and locate objects of interest from images or videos. Traditional object detection methods often rely on a single modality (such as visible light images) for feature extraction. However, in complex environments, such as low light, inclement weather, or occluded scenes, relying solely on image information from a single modality often fails to achieve robust detection results.

[0003] Multimodal target detection technology, by fusing complementary information from images of different modalities, can provide more comprehensive and stable target feature descriptions in various environments. Visible light images offer rich texture, shape, and color information, while infrared images reflect the thermal radiation characteristics of objects, exhibiting stronger target discriminability at night or under low-light conditions. The combined use of these two modalities has become an important research direction for improving target detection performance.

[0004] Meanwhile, the development of deep learning, especially the emergence of large-scale pre-trained models, has provided new breakthroughs for image feature extraction. Pre-trained models are typically trained on massive datasets, enabling them to learn more generalizable multi-level semantic features and significantly improve detection accuracy and robustness in downstream tasks. However, the utilization of pre-trained models in existing multimodal object detection methods remains relatively limited. This results in room for improvement in the fused feature representation capabilities when facing complex backgrounds, dynamic lighting, and changes in object appearance. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal image feature enhancement and fusion target detection method and device with joint pre-trained model, which significantly improves the accuracy and robustness of target detection and has broad application prospects in video surveillance, autonomous driving and other fields, especially suitable for the field of target detection technology in computer vision.

[0006] The objective of this invention is achieved through the following approach: A multimodal image feature enhancement and fusion target detection method using a joint pre-trained model includes: Feature extraction: For visible light and infrared images, a pre-trained model image encoder is used to extract general features, and a deep learning backbone network is used to extract modality-specific features. For infrared images, an adaptive gradient feature encoder is introduced to extract edge information during the forward propagation of features at each level of the backbone network, so as to guide the deep learning backbone network to pay more attention to the contours of instances and enhance the perception of edge details. Feature fusion: The extracted general features and specific features are interactively fused; Specifically, specific features extracted from different modalities at different levels are input into the multimodal attention fusion module for hierarchical fusion, which is used to integrate information from different levels between modalities; The last layer of general features and the last layer of specific features from the same modality are input into the general feature and specific feature fusion module for deep interaction and fusion of general information and specific information; Target detection result output: The multi-layer features formed after fusion are input into the target detection algorithm to obtain the final multimodal target detection result.

[0007] Furthermore, the deep learning backbone network is a CNN backbone network.

[0008] Furthermore, the adaptive gradient feature encoder specifically includes: a gradient processor, a hybrid gradient equation, and a multi-scale feature encoder; for infrared image features, firstly, two filters are used to process the edge information on each channel, and then the gradient processor is used to perform channel stitching on the processed edge information; then, the hybrid gradient equation is used to mix the edge information of the infrared image feature map in different directions to obtain edge features; finally, the multi-scale feature encoder is used to encode the edge features obtained by the hybrid gradient equation into multi-scale edge features that match the features at each level of the backbone network.

[0009] Furthermore, the parameters of the filter are learned based on the features of the input infrared image.

[0010] Furthermore, the multi-scale feature encoder uses a convolutional layer (Conv), a normalization layer (Norm), and a ReLU activation function as a basic unit (ConvU) to perform convolution, layer normalization, and activation operations on the input features. The feature map is then processed by different numbers of ConvUs and max pooling operations to obtain feature maps of different scales.

[0011] Furthermore, the multimodal attention fusion module specifically includes the following sub-steps: For the two modal feature maps input, the two modal features are first directly added together to form a preliminary fused feature; Next, calculate the channel attention and spatial attention, and add them together to obtain the initial pixel weights; The initial pixel weights are then concatenated with the initial fused features, and then fine-grained pixel attention is generated through depthwise separable convolution. Cross-modal attention is used to model the relationship between the two modalities. The final output features are obtained by weighted summation of the original features from the two modalities, the preliminary fused features, and the fine-grained pixel attention.

[0012] Furthermore, the general feature and specific feature fusion module specifically includes the following sub-steps: First, the general features extracted by the image encoder of the pre-trained model are reshaped into features with the same size as the specific features through the basic unit ConvU; Then, the two features are mapped to the query, key, and value space, and multi-head cross attention is calculated to obtain the attention score matrix; Then, the attention score matrices are concatenated and fed into subsequent basic units ConvU, fully connected layers, ReLU activation functions, and Drop layers to obtain local enhanced features. Finally, the local enhanced features are added to the specific features to obtain the features after the fusion of general features and specific features.

[0013] Furthermore, the multi-layer feature input target detection algorithm formed after fusion specifically includes the following sub-steps: The total loss function is composed of the localization loss, classification loss, and confidence loss of the predicted bounding boxes. The localization loss uses the generalized intersection-union (GIoU) ​​loss; the classification loss uses the cross-entropy loss; and the confidence loss uses the mean squared error loss to calculate the distance between the predicted value and the true value of each predicted bounding box.

[0014] A multimodal image feature enhancement and fusion target detection device with a joint pre-trained model includes a processor and a memory, wherein the memory stores a computer program that, when loaded by the processor, executes the method described in any of the preceding methods.

[0015] The beneficial effects of this invention include: This invention proposes a multimodal image feature enhancement and fusion target detection method that combines a pre-trained model. It fully utilizes the feature extraction advantages of the pre-trained model in two modalities and significantly improves the accuracy and robustness of target detection through feature enhancement and multimodal fusion strategies. It is especially suitable for target detection tasks in low light, strong interference and variable environments.

[0016] The overall framework of this invention is both scalable and practical, and can be applied to a variety of visible light-infrared image target detection tasks, while maintaining stable detection accuracy even in complex environments.

[0017] This invention introduces a pre-trained model image encoder, providing a high-starting-point, robust, and generalizable visual feature foundation, enabling multimodal object detection to achieve excellent feature representation and fusion results even in small sample and complex environments.

[0018] This invention designs an adaptive gradient feature encoder, which realizes the denoising and enhancement of infrared image features and significantly improves the intermodal information complementarity capability.

[0019] This invention designs a module for fusing general and specific features. By utilizing a multi-head attention mechanism, it effectively integrates two features with different attributes, achieving a better fusion effect between general and specific features.

[0020] This invention designs a multimodal attention fusion module, which utilizes multiple attention mechanisms to effectively fuse cross-modal features from different levels, achieving stronger feature representation and helping the model achieve better detection performance. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a block diagram of the overall model of the multimodal image feature enhancement and fusion target detection method based on the joint pre-trained model in this invention. Figure 2 This is a network structure diagram of the adaptive gradient feature encoder in an embodiment of the present invention; Figure 3 This is a structural diagram of the general feature and specific feature fusion module in an embodiment of the present invention; wherein, Conv, ReLU, Norm, and Drop layers represent convolutional layers, activation layers, normalization layers, and dropout layers, respectively; Figure 4 This is a structural diagram of the multimodal attention fusion module in an embodiment of the present invention; Figure 5 These are visual effect comparison diagrams of the methods in the embodiments of the present invention. Detailed Implementation

[0023] All features disclosed in all embodiments of this specification, or steps in all methods or processes implied in the disclosure, may be combined and / or extended or replaced in any way, except for mutually exclusive features and / or steps.

[0024] The specific implementation process of this invention is as follows: This invention particularly relates to the use of pre-trained models and CNN backbone networks to extract features from visible light and infrared images respectively, and on this basis, to fuse general features and multimodal features of the pre-trained model to improve target detection performance.

[0025] In a preferred embodiment, a multimodal image feature enhancement and fusion target detection method using a joint pre-trained model is provided, comprising the following steps: Feature extraction: For visible light and infrared images, a pre-trained model image encoder is used to extract general features, and a deep learning backbone network is used to extract modality-specific features. For infrared images, an adaptive gradient feature encoder is introduced to extract edge information during the forward propagation of features at each level of the backbone network, so as to guide the deep learning backbone network to pay more attention to the contours of instances and enhance the perception of edge details. Feature fusion: The extracted general features and specific features are interactively fused; Specifically, specific features extracted from different modalities at different levels are input into the multimodal attention fusion module for hierarchical fusion, which is used to integrate information from different levels between modalities; The last layer of general features and the last layer of specific features from the same modality are input into the general feature and specific feature fusion module for deep interaction and fusion of general information and specific information; Target detection result output: The multi-layer features formed after fusion are input into the target detection algorithm to obtain the final multimodal target detection result.

[0026] In other embodiments, based on the above embodiments, the following steps are further included: like Figure 1 As shown in this embodiment, the present invention provides a multimodal image feature enhancement and fusion target detection method that combines a pre-trained model, mainly involving a pre-trained model text encoder, a CNN feature extraction backbone network, an adaptive gradient feature encoder (AGFE), a general feature and unique feature fusion module (GUFFM), and a multimodal attention fusion module (MMAFusion).

[0027] The overall network framework and processing procedure in this embodiment are as follows: For the two input images (visible light image and infrared image), the image encoder of the pre-trained model is used to extract general features, and the CNN backbone network is used to extract modality-specific features. For the infrared image, during the forward propagation of features at each level of the CNN backbone network, edge information extracted by the Adaptive Gradient Feature Encoder (AGFE) is introduced to guide the backbone network to pay more attention to the instance contour and enhance the model's perception of edge details. Subsequently, the modality-specific features of each layer are fused hierarchically through the Multimodal Attention Fusion Module (MMAFusion) to integrate modality information at different levels; at the same time, the last layer of general features and the last layer of specific features of the same modality are input into the General and Specific Feature Fusion Module (GUFFM) for fusion to achieve deep interaction between general and specific information. Finally, the fused multi-layer features are sent to the YOLOv8 detection head to obtain the target detection result. The design of the above modules enables the present invention to achieve excellent performance in the visible light-infrared image fusion target detection task.

[0028] It should be noted that the specific implementation process of the Adaptive Gradient Feature Encoder (AGFE) is as follows: Figure 2 As shown, the adaptive gradient feature encoder mainly consists of a gradient processor, a hybrid gradient equation, and a multi-scale feature encoder. For the input infrared image features, two filters are first used to process the edge information in each channel. The filter parameters are learned based on the input infrared image features and can be represented by matrices as follows: ; Based on the above matrix and The gradient processor's processing can be represented as: ; ; in, This indicates that the input feature map has undergone... Processed feature map, This indicates that the input feature map has undergone... Processed feature map, The number of channels in the infrared image feature map. Indicates the first Characteristics of each channel Indicating the infrared image feature map Each channel passes through a filter. Process the obtained edge information, Indicating the infrared image feature map Each channel passes through a filter. Process the obtained edge information, This indicates a channel splicing operation.

[0029] Then, the edge information from different directions of the infrared image feature map is mixed using a hybrid gradient equation to obtain a more robust edge representation. This process can be described as follows: ; ; in, Indicates the first The feature map contains mixed edge information across multiple channels, where H represents the height of the feature map and W represents the width of the feature map. Represents the dimension of the feature space. It represents a very small non-zero positive number, used to prevent numerical instability when calculating the square root; This indicates a channel splicing operation; The edge feature representation of the hybrid gradient equation output is filtered out by non-structural features such as clutter and noise, and is used as input for the subsequent multi-scale feature encoder.

[0030] Finally, a multi-scale feature encoder is used to encode the edge features obtained from the hybrid gradient equation into multi-scale edge features that match the features at each level of the backbone. The multi-scale encoder uses a convolutional layer (Conv), a normalization layer (Norm), and a ReLU activation function as a basic unit (ConvU) to perform convolution, layer normalization, and activation operations on the input features. The feature map is then processed through different numbers of ConvUs and a max pooling operation (MaxPool) to obtain feature maps of different scales.

[0031] It should be noted that the specific implementation process of the General Feature and Specific Feature Fusion Module (GUFFM) is as follows: Figure 3 As shown, assuming the general features extracted by the pre-trained model image encoder are... ,in , , These represent the number of feature channels, height, and width, respectively. First, they are reshaped into features of the same size as the specific features using convolutional layers, normalization layers, and the ReLU activation function. ; Where Norm represents normalization; Then, the two features are mapped to the query, key, and value space to calculate the cross-attention score, i.e., multiplied by . , and Matrix generation for querying general and specific features ( and ),key( and ) and value ( and )matrix: ; The multi-head cross-attention calculation method is as follows: ; ; in, yes The transpose of , where G is the general feature and U is the specific feature. This indicates that the query uses general features as the basis to focus on information related to specific features. This indicates that the query uses specific features as the search term to focus on information related to general features. yes The dimensions of the matrix are then determined, and the attention score matrices are concatenated and fed into subsequent convolutional layers, normalization layers, ReLU activation functions, fully connected layers, ReLU activation functions, and Drop layers to obtain local enhanced features. Finally The feature obtained by adding the general feature to the feature-specific feature is the feature fused together with the feature-specific feature.

[0032] It should be noted that the specific implementation process of the multimodal attention fusion module (MMAFusion) is as follows: Figure 4 As shown, the feature maps for the two input modalities The multimodal attention fusion module first adds the features of the two modalities directly to form preliminary fused features: ; Based on this, channel attention is calculated. and spatial attention : ; ; in , For 1×1 convolution, This is a global average pooling operation. It is the ReLU activation function. , , and These represent the operations of taking the maximum value, taking the average value, concatenation, and convolution, respectively. Adding these two together yields the initial pixel weights: ; Again with The concatenation is then processed through depthwise separable convolutions to generate fine-grained pixel attention. Simultaneously utilize cross-modal attention Model the correlation between the two modalities, where For the sigmoid function, This is a depthwise separable convolution operation. The final output features are derived from two modalities of the original features. Preliminary fusion characteristics and fine-grained pixel attention The weighted sum is: ; in This indicates element-wise multiplication. The multimodal attention fusion module achieves deep modeling of modal differences and complementarities through the synergistic effect of channel, spatial, pixel, and cross-modal attention, thereby outputting fusion features that are both discriminative and robust.

[0033] It should be noted that the specific design of the loss function is as follows: The method of this invention uses the localization loss, classification loss, and confidence loss of the predicted bounding box as the total loss, as shown in the following formula: ; in , , These represent regression loss, classification loss, and confidence loss of the predicted bounding box, respectively.

[0034] First, the method of this invention uses Generalized Intersection over Union (GIoU) ​​loss as the localization loss. Unlike IoU loss, which only focuses on overlapping areas, GIoU loss considers not only overlapping areas but also other non-overlapping areas, thus more comprehensively reflecting the degree of overlap between the two bounding boxes. The formula for calculating the localization loss is as follows: ; in, This indicates the number of grid cells in the predicted image. This indicates the number of prediction boxes in each grid cell. Indicates the first In the grid, the th Is each predicted bounding box a positive sample? Indicates the prediction box. Represents the true bounding box. Indicates by and The area of ​​the minimum bounding box formed. Indicates the first Generalized Cross-Union Comparison (GUC) between each predicted bounding box and its corresponding ground truth bounding box.

[0035] The classification loss uses cross-entropy loss to optimize the classification of predicted bounding boxes. The calculation formula is as follows: ; in, and These represent the categories to which the network predicts the samples. The probabilities and true probabilities are denoted by , where c represents the number of categories. Indicates category, Indicates the first In the grid, the th Whether each predicted bounding box is a positive sample.

[0036] Confidence loss measures the difference between the model's predicted confidence level for detected objects and the true label. Mean squared error loss is used to calculate the distance between the predicted value and the true value for each bounding box. The calculation formula is as follows: ; in, and These represent the true confidence score and the network prediction score, respectively. Indicates the first In the grid, the th The loss function is designed to effectively balance localization accuracy, target detection confidence, and classification accuracy, thereby improving the model's accuracy and robustness. This design also helps to determine whether each predicted bounding box is a negative sample.

[0037] To verify the effectiveness of the multimodal image feature enhancement and fusion target detection method provided by the joint pre-trained model in this embodiment of the invention, detailed comparative experiments will be provided below.

[0038] The experimental environment consisted of an Ubuntu 20.04 operating system, PyTorch 1.10 deep learning framework, and Python version 3.7. This invention compared the model's mAP50 results (a higher mAP50 value indicates better performance) on the mainstream visible-infrared image fusion object detection dataset DVTOD, and visualized some of the detection results. Mainstream object detection methods from recent years were selected for comparison, specifically: YOLOv5, YOLOv11, CFT, CMA-Det, and F3M-Det.

[0039] The test results are shown in Table 1. Compared with other methods, it can be found that the method of the present invention has better detection performance. Figure 5The diagram shows an objective visual comparison of the present invention with other algorithms on the DVTOD dataset. The columns from left to right are the visualization results of visible light image input, infrared image input, YOLOv8, CMA-Det, F3M-Det, and the method of the present invention. It can be seen that the method of the present invention can more effectively detect objects located at the edge of the image and smaller objects, and outline them with appropriate bounding boxes.

[0040] Table 1 Comparison of detection performance on the DVTOD dataset

[0041] In Table 1, AP represents the detection accuracy, and mAP50 represents the average detection accuracy of all categories when the cross-union threshold is 50%.

[0042] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0043] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0044] In another aspect, embodiments of the present invention also provide a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

Claims

1. A multimodal image feature enhancement and fusion target detection method using a joint pre-trained model, characterized in that, include: Feature extraction: For visible light and infrared images, a pre-trained model image encoder is used to extract general features, and a deep learning backbone network is used to extract modality-specific features. For infrared images, an adaptive gradient feature encoder is introduced to extract edge information during the forward propagation of features at each level of the backbone network, so as to guide the deep learning backbone network to pay more attention to the contours of instances and enhance the perception of edge details. Feature fusion: The extracted general features and specific features are interactively fused; Specifically, specific features extracted from different modalities at different levels are input into the multimodal attention fusion module for hierarchical fusion, which is used to integrate information from different levels between modalities; The last layer of general features and the last layer of specific features from the same modality are input into the general feature and specific feature fusion module for deep interaction and fusion of general information and specific information; Target detection result output: The multi-layer features formed after fusion are input into the target detection algorithm to obtain the final multimodal target detection result; The adaptive gradient feature encoder specifically includes: a gradient processor, a hybrid gradient equation, and a multi-scale feature encoder. For infrared image features, firstly, two filters are used to process the edge information on each channel, and then the gradient processor is used to perform channel stitching on the processed edge information. Then, the hybrid gradient equation is used to mix the edge information of the infrared image feature map in different directions to obtain edge features. Finally, the multi-scale feature encoder is used to encode the edge features obtained by the hybrid gradient equation into multi-scale edge features that match the features at each level of the backbone network.

2. The multimodal image feature enhancement and fusion target detection method according to claim 1, characterized in that, The deep learning backbone network is a CNN backbone network.

3. The multimodal image feature enhancement and fusion target detection method according to claim 1, characterized in that, The parameters of the filter are learned based on the features of the input infrared image.

4. The multimodal image feature enhancement and fusion target detection method according to claim 1, characterized in that, The multi-scale feature encoder uses a convolutional layer (Conv), a normalization layer (Norm), and a ReLU activation function as a basic unit (ConvU) to perform convolution, layer normalization, and activation operations on the input features. The feature map is then processed by different numbers of ConvUs and max pooling operations to obtain feature maps of different scales.

5. The multimodal image feature enhancement and fusion target detection method according to claim 1, characterized in that, The multimodal attention fusion module specifically includes the following sub-steps: For the two modal feature maps input, the two modal features are first directly added together to form a preliminary fused feature; Next, calculate the channel attention and spatial attention, and add them together to obtain the initial pixel weights; The initial pixel weights are then concatenated with the initial fused features, and then fine-grained pixel attention is generated through depthwise separable convolution. Cross-modal attention is used to model the relationship between the two modalities. The final output features are obtained by weighted summation of the original features from the two modalities, the preliminary fused features, and the fine-grained pixel attention.

6. The multimodal image feature enhancement and fusion target detection method according to claim 4, characterized in that, The general feature and specific feature fusion module specifically includes the following sub-steps: First, the general features extracted by the image encoder of the pre-trained model are reshaped into features with the same size as the specific features through the basic unit ConvU; Then, the two features are mapped to the query, key, and value space, and multi-head cross attention is calculated to obtain the attention score matrix; Then, the attention score matrices are concatenated and fed into subsequent basic units ConvU, fully connected layers, ReLU activation functions, and Drop layers to obtain local enhanced features. Finally, the local enhanced features are added to the specific features to obtain the features after the fusion of general features and specific features.

7. The multimodal image feature enhancement and fusion target detection method according to claim 1, characterized in that, The multi-layer feature input target detection algorithm formed after fusion specifically includes the following sub-steps: The total loss function is composed of the localization loss, classification loss, and confidence loss of the predicted bounding boxes. The localization loss uses the generalized intersection-union (GIoU) ​​loss; the classification loss uses the cross-entropy loss; and the confidence loss uses the mean squared error loss to calculate the distance between the predicted value and the true value of each predicted bounding box.

8. A multimodal image feature enhancement and fusion target detection device with a joint pre-trained model, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when loaded by the processor, executes the method as described in any one of claims 1 to 7.

Images