Identification method and system based on RGB-NIR multi-modal image fusion

Through the multi-step optimization of RGB-NIR feature fusion method, the bottleneck module and the dual-modal fusion module are used to solve the accuracy and real-time problems of object detection in complex environments, and efficient object recognition in night scenes is achieved.

CN118968239APending Publication Date: 2024-11-15REALMAN ROBOT CO LTD

Patent Information

Application Number
CN202410973465.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In complex environments, especially in night scenes, RGB-NIR multimodal image fusion based on deep convolutional networks is difficult to achieve accurate object detection, and the existing methods have shortcomings in real-time inference speed.

Method used

Using the multi-step optimization RGB-NIR feature fusion method, feature extraction and fusion are performed through the bottleneck module and the dual-modal fusion module, and feature enhancement and correction are achieved by using depth-separable convolution and channel-level attention vectors to achieve RGB and NIR images. effective integration.

Benefits of technology

It improves the accuracy of object detection in similar color objects and night scenes, while maintaining real-time inference speed, improving the recognition performance and feature representation capabilities of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968239A_ABST
    Figure CN118968239A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of target detection, and aims to solve the problem that existing similar color objects and target objects in a night scene are inaccurately recognized. In order to achieve the purpose, the invention discloses a recognition method and system based on RGB-NIR multi-modal image fusion, and the method comprises the following steps: obtaining an RGB image and an NIR image of a to-be-recognized object; respectively processing the RGB image and the NIR image through a convolution layer and a maximum pooling layer in sequence to obtain an RGB feature map and an NIR feature map; performing feature extraction and bimodal image fusion on the RGB feature map and the NIR feature map for multiple times, and determining an object to be recognized according to a final fused feature map; wherein the feature extraction is carried out through a Bottleneck module, and the bimodal image fusion is carried out through a bimodal fusion module. According to the method, the problem of low detection accuracy of the target object in complex environments such as darkness and close color texture of the object to be detected can be solved, and the real-time reasoning speed can be achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a recognition method and system based on RGB-NIR multimodal image fusion. Background Art

[0002] In recent years, with the computing power support of high-performance hardware and the convenience of obtaining large-scale visible light image (RGB) data sets, computer vision technology based on deep neural networks can guide robots to show good recognition performance under normal lighting conditions. However, even with large-scale RGB images of complex environments, accurately detecting targets in complex environments such as darkness and close color and texture of the objects to be detected is still a challenging problem.

[0003] Intuitively, in daytime scenes, objects in RGB images present clearer texture information than in infrared images. In contrast, in nighttime scenes, infrared images can provide clearer object shape information than RGB images. Therefore, the fusion of RGB-near infrared (NIR) multimodal images based on deep convolutional networks has been widely studied.

[0004] According to the different fusion regions of multimodal features, multimodal fusion can be divided into early fusion, late fusion and mid-term fusion. The fusion of multimodal images before being input into the backbone network is called early fusion. Existing literature has proposed a novel convolutional neural network with aligned regions for the weak alignment problem between two modal images. This method enhances the feature saliency of the target in the RGB image with the assistance of near-infrared thermal images only. However, when there is pixel offset in different modal images, the early fusion method is prone to produce too much noise. Intermediate fusion is a feature-level fusion method that fuses feature maps between modalities at different feature extraction stages of the backbone network. There are also literature reports that RGB, infrared and polarization images are provided as input to the three backbone feature extraction networks of Faster R-CNN (algorithm), and then the output feature maps are cascaded for modal fusion. The results show that the model with multimodal images as input has a good recognition effect on difficult-to-identify targets in complex environments, but the introduction of multiple branches will lead to a decrease in detection speed. In the late fusion method, the parameters of the backbone feature extraction networks of the two modalities are completely independent, which makes the entire network ultimately unable to learn the complementary information between the modalities and the unique attributes of each modality, thus limiting the model effect. Another literature reported that based on the shared Faster R-CNN feature extraction network, the detection results of RGB-NIR multimodal images were fused at the decision layer. Compared with the unfused results, the detection accuracy of fruits with similar colors increased from 81.6% to 83.8%. However, the late fusion method linearly fuses the target information output by each independent network model prediction head. In addition, the introduction of multiple branches will lead to a decrease in detection speed.

[0005] RGB-NIR multimodal fusion for target detection utilizes complementary information between modalities, which brings certain advantages to many application fields. However, in some existing methods, RGB and NIR images are not aligned, and the same target has different positions in the two modal images, so partially misaligned RGB-NIR images increase the difficulty of fusion.

[0006] Therefore, there is an urgent need for a network that can resolve objects of similar colors and accurately identify them in night scenes while achieving real-time inference speed. Summary of the invention

[0007] (I) Purpose of the invention

[0008] The purpose of the present invention is to provide a recognition method and system based on RGB-NIR multimodal image fusion. The method adopts a two-step optimization fusion RGB-NIR feature design to improve the model recognition performance. It inherits the real-time reasoning speed of the RGB network structure and has the powerful feature representation performance of multimodal image fusion. The method solves the problem of accurately identifying objects of similar colors and targets in night scenes, and can also achieve real-time reasoning speed.

[0009] (II) Technical solution

[0010] To solve the above problems, the first aspect of the present invention provides a recognition method based on RGB-NIR multimodal image fusion, comprising the following steps:

[0011] Obtain RGB image and NIR image of the object to be identified;

[0012] The RGB image and the NIR image are processed by a convolution layer and a maximum pooling layer respectively to obtain an RGB feature map and a NIR feature map;

[0013] Performing feature extraction and bimodal image fusion on the RGB feature map and the NIR feature map multiple times to obtain a final fused feature map;

[0014] Determining the object to be identified according to the final fusion feature map;

[0015] Wherein, the feature extraction is performed through a bottleneck module, and the dual-modal image fusion is performed through a dual-modal fusion module.

[0016] Furthermore, the dual-modal image fusion is performed by a dual-modal fusion module, and the specific steps include:

[0017] The RGB extraction features and NIR extraction features extracted by the bottleneck module are spliced ​​according to the channels to obtain the spliced ​​RGB-NIR aggregate features;

[0018] Performing a pooling operation on the aggregated features of RGB-NIR to obtain a global embedding feature in the channel dimension, and obtaining a channel-level attention vector based on the global embedding feature;

[0019] Multiplying the channel-level attention vector and the RGB extraction feature and performing element-by-element addition to obtain a first RGB feature map;

[0020] Multiplying the channel-level attention vector and the NIR extracted feature and performing element-by-element addition to obtain a first NIR feature map;

[0021] Perform channel dimension splicing on the first RGB feature map and the first NIR feature map to form a high-level input feature map, and then correct the high-level input feature map to obtain a corrected high-level input feature map;

[0022] Multiplying the corrected high-level input feature map with the RGB extraction feature and performing element-by-element addition to obtain a second RGB feature map;

[0023] Multiplying the corrected high-level input feature map and the NIR extracted feature map and performing element-by-element addition to obtain a second NIR feature map;

[0024] Finally, the second RGB feature map and the second NIR feature map are added element by element to obtain a fused feature map.

[0025] Furthermore, the multiple times of performing feature extraction and dual-modal image fusion on the RGB feature map and the NIR feature map to obtain a final fused feature map specifically includes:

[0026] Using a first bottleneck module to perform feature extraction on the RGB feature map and the NIR feature map to obtain a first RGB extraction feature and a first NIR extraction feature;

[0027] Using a second bottleneck module to extract the first RGB extraction feature and the first NIR extraction feature respectively, to obtain a second RGB extraction feature and a second NIR extraction feature;

[0028] Using a bimodal fusion module to perform a first bimodal image fusion on the second RGB extraction feature and the second NIR extraction feature to obtain a first fusion feature map;

[0029] Using a third bottleneck module to perform feature extraction on the first fusion feature map to obtain a third RGB extraction feature and a third NIR extraction feature;

[0030] Using a bimodal fusion module, performing a second bimodal image fusion on the third RGB extraction feature and the third NIR extraction feature to obtain a second fusion feature map;

[0031] Using a fourth bottleneck module to perform feature extraction on the second fusion feature map to obtain a fourth RGB extraction feature and a fourth NIR extraction feature;

[0032] The fourth RGB extraction feature and the fourth NIR extraction feature are subjected to a third bimodal image fusion using a bimodal fusion module to obtain a final fusion feature map.

[0033] Furthermore, the use of the first bottleneck module to extract features from the RGB feature map and the NIR feature map specifically includes: using shared convolution kernel parameters to extract features from the first bottleneck module of RGB and the first bottleneck module of NIR, and the first bottleneck module includes the first bottleneck module of RGB and the first bottleneck module of NIR.

[0034] Furthermore, the bottleneck module adopts depthwise separable convolution DSConv, and the convolution kernel sizes of the depthwise separable convolution DSConv are 1*1 and 3*3.

[0035] Furthermore, obtaining a channel-level attention vector based on a global embedding feature specifically includes: generating a channel-level attention vector through a 1×1 convolution and a Sigmoid function using the global embedding feature.

[0036] Furthermore, the channel dimension splicing of the first RGB feature map and the first NIR feature map to form a high-level input feature map, and then correcting the high-level input feature map to obtain the corrected high-level input feature map specifically includes:

[0037] The first RGB feature map and the first NIR feature map are concatenated in channel dimension, and a 1×1 convolution is used to implement a channel dimension reduction operation;

[0038] The Softmax function is used as a correction coefficient for the high-level input feature map to filter out unimportant features.

[0039] Furthermore, determining the object to be identified according to the final fusion feature map specifically includes:

[0040] Inputting the final fusion feature into the output head of the network model to obtain a predicted output result;

[0041] Constructing a loss function to perform loss calculation on the predicted output result to obtain the final screening result;

[0042] The object to be identified is determined according to the final screening result.

[0043] Furthermore, the formula of the loss function is as follows:

[0044] Loss = L RGBBox +L RGBCls +λ×(L NIRBox +L NIRCls ),

[0045] Among them, L RGBCls and LNIRCls are the classification losses of the prediction results in RGB images and NIR images, respectively, and L RGBBox and L NIRBox are the position offset losses of the prediction results in the RGB image and the NIR image, respectively, and λ is the regularization balance term used to balance the losses between the prediction results of the RGB image and the NIR image.

[0046] Furthermore, the convolution kernel size of the convolution layer is 7*7, and the convolution kernel size of the maximum pooling layer is 3*3.

[0047] Furthermore, the first bottleneck module, the second bottleneck module, the third bottleneck module and the fourth bottleneck module are stacked 3 times, 4 times, 6 times and 3 times respectively.

[0048] A second aspect of the present invention provides a recognition system based on RGB-NIR multimodal image fusion, comprising:

[0049] An acquisition module acquires the RGB image and NIR image of the object to be identified;

[0050] A processing module, processing the RGB image and the NIR image through a convolution layer and a maximum pooling layer in sequence, respectively, to obtain an RGB feature map and a NIR feature map;

[0051] A fusion module performs feature extraction and bimodal image fusion on the RGB feature map and the NIR feature map multiple times to obtain a final fused feature map;

[0052] A recognition module, which determines the object to be recognized according to the final fusion feature map;

[0053] Wherein, the feature extraction is performed through a bottleneck module, and the dual-modal image fusion is performed through a dual-modal fusion module.

[0054] The recognition method and system based on RGB-NIR multimodal image fusion provided by the present invention have the following beneficial effects: the present invention obtains the final fusion feature map by performing feature extraction and dual-modal image fusion on the RGB feature map and the NIR feature map for multiple times; wherein the feature extraction is performed by a bottleneck module, and each bottleneck performs feature extraction from the bottom layer to the high layer; the dual-modal image fusion is performed by a dual-modal fusion module (TFWF module), and the TFWF module is based on an idea of ​​optimizing and fusing RGB-NIR features in two steps to improve the model recognition performance, which not only inherits the real-time reasoning speed of the RGB network structure, but also has the powerful feature representation performance of multimodal image fusion, and makes full use of the complementary characteristics of the two images to help the model fit better. Finally, the object to be identified is determined according to the final fusion feature map. This method solves the problem of accurately identifying targets between objects of similar colors and in night scenes, and can also achieve real-time reasoning speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flowchart of a recognition method based on RGB-NIR multimodal image fusion according to an embodiment of the present invention;

[0056] Figure 2 It is a specific embodiment of the recognition method based on RGB-NIR multimodal image fusion of the present invention;

[0057] Figure 3 The specific network structure of the DSconv module;

[0058] Figure 4 The specific network structure of the Prediction Head module;

[0059] Figure 5 It is a fusion schematic diagram of a dual-modal fusion module (TFWF module) according to a specific embodiment of the present invention;

[0060] Figure 6 It is a schematic diagram of a flow chart of a recognition system based on RGB-NIR multimodal image fusion according to an embodiment of the present invention;

[0061] Figure 7 This is a picture of the detection effect of an embodiment of the present invention;

[0062] Figure 8 This is the detection effect picture of the comparative example YOLO v5;

[0063] Fig. 9 This is the detection effect picture of the comparative example YOLO v7;

[0064] Fig.10This is a comparison picture of the detection effect of YOLO v8. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0066] The first aspect of the present invention provides a recognition method based on RGB-NIR multimodal image fusion, such as Figure 1 As shown, the following steps are included:

[0067] S101, obtaining an RGB image and a NIR image of an object to be identified;

[0068] S102, respectively, processes the RGB image and the NIR image through the convolution layer and the maximum pooling layer in sequence to obtain an RGB feature map and a NIR feature map; wherein the convolution kernel size of the convolution layer is 7*7, and the convolution kernel size of the maximum pooling layer is 3*3.

[0069] S103, performing feature extraction and bimodal image fusion on the RGB feature map and the NIR feature map multiple times to obtain a final fused feature map;

[0070] S104, determining the object to be identified according to the final fusion feature map;

[0071] Among them, feature extraction is performed through a bottleneck module, and bimodal image fusion is performed through a bimodal fusion module. The bottleneck module adopts a depthwise separable convolution DSConv, and the convolution kernel size of the depthwise separable convolution DSConv is 1*1 and 3*3. The bimodal fusion module is a TFWF module. The TFWF module takes the RGB feature extraction data and the NIR feature extraction data as input, and learns the fused shared feature weights (specified as weight 1 adjustment and weight 2 adjustment in the present invention) through two steps.

[0072] Furthermore, feature extraction and bimodal image fusion are performed on the RGB feature map and the NIR feature map multiple times to obtain the final fused feature map, which specifically includes:

[0073] The first bottleneck module is used to extract features from the RGB feature map and the NIR feature map to obtain first RGB extraction features and first NIR extraction features; wherein, the first bottleneck module is used to extract features from the RGB feature map and the NIR feature map, specifically including: using shared convolution kernel parameters to extract features from the first bottleneck module of RGB and the first bottleneck module of NIR, and the first bottleneck module includes the first bottleneck module of RGB and the first bottleneck module of NIR.

[0074] The first RGB extraction feature and the first NIR extraction feature are respectively extracted using the second bottleneck module to obtain a second RGB extraction feature and a second NIR extraction feature;

[0075] Using the bimodal fusion module, a first bimodal image fusion is performed on the second RGB extracted features and the second NIR extracted features to obtain a first fused feature map;

[0076] Using the third bottleneck module to perform feature extraction on the first fusion feature map, to obtain a third RGB extraction feature and a third NIR extraction feature;

[0077] Using the bimodal fusion module, a second bimodal image fusion is performed on the third RGB extracted features and the third NIR extracted features to obtain a second fused feature map;

[0078] Using the fourth bottleneck module to perform feature extraction on the second fused feature map, to obtain a fourth RGB extraction feature and a fourth NIR extraction feature;

[0079] The bimodal fusion module is used to perform a third bimodal image fusion on the fourth RGB extracted features and the fourth NIR extracted features to obtain the final fusion feature map.

[0080] Further, the stacking times of the first bottleneck module, the second bottleneck module, the third bottleneck module and the fourth bottleneck module are 3 times, 4 times, 6 times and 3 times respectively.

[0081] Combine the following Figure 2 As shown, a specific embodiment of the present invention is given for the recognition method based on RGB-NIR multimodal image fusion, as follows:

[0082] Inspired by the effectiveness of the residual network, the backbone network involved in this method is as follows Figure 2As shown in the figure. The network is composed of a parallel dual branch, which is divided into an RGB feature extraction branch and a NIR feature extraction branch. First, the RGB-NIR modal image passes through a convolution layer with a convolution kernel size of 7 and a maximum pooling layer with a convolution kernel size of 3. Then, the RGB feature map and the NIR feature map output by the pooling layer pass through bottleneck modules with stacking times of 3, 4, 6, and 3 respectively. Each bottleneck module consists of a depth-separable convolution DSConv with convolution kernel sizes of 1 and 3. The purpose of using DSConv instead of ordinary convolution kernels is to reduce the number of model parameters and improve the inference speed of the model while ensuring the feature extraction capability of the model. The RGB feature data and NIR feature data output by the second, third, and fourth bottleneck modules are input into the TFWF designed by this method for dual-modal image fusion, and the fused output feature map is used as the input of the next bottleneck module. The modal image fusion is to make full use of the complementary characteristics of the two images to help the model fit better. Each bottleneck extracts features from the bottom to the top. In addition, convolutional neural networks have the characteristics of feature locality and translation invariance. As the number of layers of the network increases, the high-level features become more abstract and diverse, and the differences between different positions and channels are also greater. Therefore, the probability that the pixel offset of different modal images changes the output results will continue to increase with the depth of the network layer. Therefore, we only use convolution kernels with shared weight parameters for feature extraction in the lower layers of the network to maintain the stability of extraction of the same target by different modalities. Preferably, this method selects the convolution kernel parameter sharing strategy for RGB-NIR in the first bottleneck module. Figure 2 In the example, Prediction Head (the output head of the network outputs the prediction result), NMS (non-maximum suppression) (screens out the final result). In RGB (3, 640, 640), 3 represents the channel, and 640 represents the width and height of the image. In NIR, it is a single channel, and the image width and height are 640. Figure 3 It is the specific network structure of the DSconv module, where PointwiseConv is point-by-point convolution and DepthwiseConv is depth-by-depth convolution; Figure 4 It is the specific network structure of the Prediction Head module (Box is the positioning information, cls is the category information).

[0083] Furthermore, the dual-modal image fusion is performed by a dual-modal fusion module, and the specific steps include:

[0084] The RGB extraction features and NIR extraction features extracted by the bottleneck module are spliced ​​according to the channels to obtain the spliced ​​RGB-NIR aggregate features;

[0085] A pooling operation is performed on the aggregated features of RGB-NIR to obtain a global embedding feature in the channel dimension, and a channel-level attention vector is obtained based on the global embedding feature; wherein, obtaining the channel-level attention vector based on the global embedding feature specifically includes: generating a channel-level attention vector through a 1×1 convolution and a Sigmoid function on the global embedding feature;

[0086] Multiply the channel-level attention vector with the RGB extracted features and add them element-by-element to obtain the first RGB feature map;

[0087] Multiply the channel-level attention vector and the NIR extracted features and add them element-by-element to obtain the first NIR feature map;

[0088] The first RGB feature map and the first NIR feature map are spliced ​​in channel dimension to form a high-level input feature map, and then the high-level input feature map is corrected to obtain a corrected high-level input feature map, specifically including: the first RGB feature map and the first NIR feature map are spliced ​​in channel dimension, and 1×1 convolution is used to realize channel dimension reduction operation; Softmax function is used as a correction coefficient for the high-level input feature map to filter unimportant features;

[0089] Multiply the corrected high-level input feature map with the RGB extracted feature and add them element by element to obtain a second RGB feature map;

[0090] Multiply the corrected high-level input feature map with the NIR extracted feature and add them element by element to obtain a second NIR feature map;

[0091] Finally, the second RGB feature map and the second NIR feature map are added element by element to obtain a fused feature map.

[0092] Combine the following Figure 5As shown in the figure, the specific fusion process of the TFWF module is described as follows: First, for the first step (weight 1 adjustment), the RGB extracted features and NIR extracted features extracted by the bottleneck module are spliced ​​according to the channel to obtain the spliced ​​RGB-NIR aggregate features, and then the spliced ​​aggregate features are subjected to the global average pooling (GAP) operation to obtain the global embedding features in the channel dimension. On this basis, weights are generated first, and the channel-level attention vector is obtained by using 1×1 convolution and Sigmoid function. Each element of this channel-level attention vector corresponds to the weight of a channel, indicating the importance or significance of the channel; then these weights are applied to the initial RGB-NIR features. Specifically, the weight of each channel is multiplied element by element with the feature map of the channel, so that important channels will be given greater weights, while unimportant channels will be relatively suppressed. Finally, the weighted feature map is fused with the initial feature map by element-by-element addition, in order to enhance the model's attention to important features while retaining the original feature information and avoid losing key information. The first RGB feature map and the first NIR feature map are obtained through the first step. The first step is inspired by residual connections to avoid losing some important feature information that the model does not pay enough attention to.

[0093] For the second step (weight 2 adjustment), the first RGB feature map and the first NIR feature map independently obtained from the first step are spliced ​​in the channel dimension to form a high-level input feature map. A 1×1 convolution is used to implement the channel dimension reduction operation on the high-level input feature map to ensure that it is the same as the feature map of the first step at the channel level. Subsequently, the Softmax function is used as the correction coefficient of the high-level input feature map to filter out unimportant feature information, thereby improving the feature response between modalities. Finally, the corrected feature map is multiplied by the initial RGB-NIR feature and element-by-element addition is performed to obtain the second RGB feature map and the second NIR feature map after weight adjustment. Finally, the two modal images (the second RGB feature map and the second NIR feature map) are added pixel by pixel to obtain the final modality fusion output feature map (fusion feature map).

[0094] Figure 5 In the figure, c is concatenation by channel dimension, global average pooling GAP: (Global AveragePooling), Fout represents the output of this module, the input of this module is Frgb and FNIR (Frgb represents the extracted RGB features, FNIR represents the extracted NIR features), and Conv is convolution.

[0095] Furthermore, according to the final fusion feature map, the object to be identified is determined, which specifically includes:

[0096] Input the final fusion features into the output head of the network model to obtain the predicted output result;

[0097] Construct a loss function to calculate the loss of the predicted output results and obtain the final screening results;

[0098] The object to be identified is determined through the final screening results.

[0099] The specific embodiments of the above steps are as follows:

[0100] The purpose of fusing multimodal images is to improve the network's understanding of target features and achieve better target recognition results. However, due to the parallax of the two cameras, the background of the same target area in the multimodal image may appear different. Therefore, if during the model training process, only the difference between the model's predicted value and the label in the RGB image is used as the loss term, the NIR feature information that has been fused in the backbone feature extraction network will be understood as interference information, thereby reducing the detection effectiveness of the model. In order to reduce the influence of mutual exclusion between multimodal images, this method will regress and classify the bounding box of the predicted NIR image, and take into account the position offset loss and classification loss calculated for the NIR image. Specifically, the position offset loss and classification loss obtained for RGB-NIR are weightedly combined to construct a loss function for RGB-NIR multimodal image fusion to help the network comprehensively understand the fused features. The formula of the loss function is as follows:

[0101] Loss = L RGBBox +L RGBCls +λ×(L NIRBox +L NIRCls ),

[0102] Among them, L RGBCls and L NIRCls are the classification losses of the prediction results in RGB images and NIR images, respectively, and L RGBBox and L NIRBox are the position offset losses of the prediction results in the RGB image and the NIR image, respectively, and λ is the regularization balance term used to balance the losses between the prediction results of the RGB image and the NIR image.

[0103] The second aspect of the present invention provides a recognition system based on RGB-NIR multimodal image fusion, such as Figure 6 As shown, including:

[0104] An acquisition module 201 acquires an RGB image and a NIR image of an object to be identified;

[0105] The processing module 202 processes the RGB image and the NIR image through a convolution layer and a maximum pooling layer in sequence to obtain an RGB feature map and a NIR feature map;

[0106] The fusion module 203 performs feature extraction and dual-modal image fusion on the RGB feature map and the NIR feature map multiple times to obtain a final fused feature map;

[0107] The recognition module 204 determines the object to be recognized according to the final fusion feature map;

[0108] Among them, feature extraction is performed through the bottleneck module, and bimodal image fusion is performed through the bimodal fusion module.

[0109] The experimental verification and result analysis of the present invention are as follows:

[0110] The verification process of this experiment uses precision, recall, and mean class precision (mAP) as metrics to verify the performance of the deep learning model. mAP (mean Average Precision), the mean of the mean precision, is the average of AP (Average Precision), which is the main evaluation indicator of the target detection algorithm. The higher the mAP value, the better the detection effect of the target detection model on the given data set. The relevant verification indicator formula is expressed as follows:

[0111]

[0112] Among them, TP represents the number of images correctly predicted as positive samples by the model, FP represents the number of images incorrectly predicted as positive samples by the model, FN represents the number of images incorrectly predicted as negative samples by the model, K represents the total number of categories, APi represents the accuracy value obtained for the i-th category, P(r) represents the Pr curve, which is the precision and recall rate curve. This represents the integral, that is, the area formed by the PR curve.

[0113] The quantitative analysis and predictive visualization analysis comparison of this method and the existing methods are shown in Table 1. FPS (frame per second) indicates the number of image frames processed per second. The higher the FPS, the better the real-time performance of the model. mAP@0.5 (mean Average Precision (IoU = 0.5)): when IoU is set to 0.5, the AP of all images of each category is calculated, and then the average of all categories is calculated. mAP@0.5 is an indicator for evaluating the performance of target detection models. The predictive visualization analysis comparison results are shown in Table 1. Figure 7-10 ,in, Figure 7 This is a picture of the detection effect of an embodiment of the present invention. Figure 8 This is the detection effect picture of the comparative example YOLO v5; Fig. 9 This is the detection effect picture of the comparative example YOLO v7; Fig.10This is a comparison picture of the detection effect of YOLO v8.

[0114] Table 1 Comparative test results

[0115] method Accuracy / % Recall / % mAP@0.5 FPS YOLO v5 85.5 85.4 88.7 120 YOLO v7 86.1 82.8 88.5 251 YOLOv8 90.6 85.1 88.9 232 This method 93.4 87.0 91.9 335

[0116] It can be seen from Table 1 that the various indicators detected by the recognition method based on RGB-NIR multimodal image fusion of the present invention are higher than those of the comparative example, indicating that the present method achieves the best quantitative results and has a high prediction accuracy. In addition, the detection effect of the present method is as follows Figure 7 As shown in the figure, the meaning of person0.32 detection is as follows: person represents the predicted category, 0.32 represents the confidence, which is a number between 0 and 1. The larger the confidence, the greater the confidence that the model believes that the target belongs to person, which means that the model prediction is more accurate. The same is true for car0.57. The comparison method ( Figure 8-10 ) both had missed detections and false detections. Through visual comparison, it can be seen that the method of the present invention does not have serious missed detections or false detections, and the prediction accuracy is high. The method can perform vehicle and pedestrian recognition well.

[0117] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, systems, etc. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A recognition method based on RGB-NIR multimodal image fusion, It is characterized in that The following steps are involved: Obtain RGB image and NIR image of the object to be identified; The RGB image and the NIR image are processed by a convolution layer and a maximum pooling layer respectively to obtain an RGB feature map and a NIR feature map; Performing feature extraction and bimodal image fusion on the RGB feature map and the NIR feature map multiple times to obtain a final fused feature map; Determining the object to be identified according to the final fusion feature map; Wherein, the feature extraction is performed through a bottleneck module, and the dual-modal image fusion is performed through a dual-modal fusion module.

2. The recognition method based on RGB-NIR multimodal image fusion according to claim 1, It is characterized in that The dual-modal image fusion is performed by a dual-modal fusion module, and the specific steps include: The RGB extraction features and NIR extraction features extracted by the bottleneck module are spliced ​​according to the channels to obtain the spliced ​​RGB-NIR aggregate features; Performing a pooling operation on the aggregated features of RGB-NIR to obtain a global embedding feature in the channel dimension, and obtaining a channel-level attention vector based on the global embedding feature; Multiplying the channel-level attention vector and the RGB extraction feature and performing element-by-element addition to obtain a first RGB feature map; Multiplying the channel-level attention vector and the NIR extracted feature and performing element-by-element addition to obtain a first NIR feature map; Perform channel dimension splicing on the first RGB feature map and the first NIR feature map to form a high-level input feature map, and then correct the high-level input feature map to obtain a corrected high-level input feature map; Multiplying the corrected high-level input feature map with the RGB extraction feature and performing element-by-element addition to obtain a second RGB feature map; Multiplying the corrected high-level input feature map and the NIR extracted feature map and performing element-by-element addition to obtain a second NIR feature map; Finally, the second RGB feature map and the second NIR feature map are added element by element to obtain a fused feature map.

3. The recognition method based on RGB-NIR multimodal image fusion according to claim 1, It is characterized in that The multiple times of performing feature extraction and dual-modal image fusion on the RGB feature map and the NIR feature map to obtain a final fused feature map specifically includes: Using a first bottleneck module to perform feature extraction on the RGB feature map and the NIR feature map to obtain a first RGB extraction feature and a first NIR extraction feature; Using a second bottleneck module to extract the first RGB extraction feature and the first NIR extraction feature respectively, to obtain a second RGB extraction feature and a second NIR extraction feature; Using a bimodal fusion module to perform a first bimodal image fusion on the second RGB extraction feature and the second NIR extraction feature to obtain a first fusion feature map; Using a third bottleneck module to perform feature extraction on the first fusion feature map to obtain a third RGB extraction feature and a third NIR extraction feature; Using a bimodal fusion module, performing a second bimodal image fusion on the third RGB extraction feature and the third NIR extraction feature to obtain a second fusion feature map; Using a fourth bottleneck module to perform feature extraction on the second fused feature map to obtain a fourth RGB extraction feature and a fourth NIR extraction feature; The fourth RGB extraction feature and the fourth NIR extraction feature are subjected to a third bimodal image fusion using a bimodal fusion module to obtain a final fusion feature map.

4. The recognition method based on RGB-NIR multimodal image fusion according to claim 3, It is characterized in that The method of using the first bottleneck module to extract features from the RGB feature map and the NIR feature map specifically includes: using shared convolution kernel parameters to extract features from the first bottleneck module of RGB and the first bottleneck module of NIR, wherein the first bottleneck module includes the first bottleneck module of RGB and the first bottleneck module of NIR.

5. The recognition method based on RGB-NIR multimodal image fusion according to claim 1, It is characterized in that The bottleneck module adopts depthwise separable convolution DSConv, and the convolution kernel sizes of the depthwise separable convolution DSConv are 1*1 and 3*3.

6. The recognition method based on RGB-NIR multimodal image fusion according to claim 2, It is characterized in that The obtaining of the channel-level attention vector based on the global embedding feature specifically includes: the global embedding feature generates a channel-level attention vector through 1×1 convolution and Sigmoid function.

7. The recognition method based on RGB-NIR multimodal image fusion according to claim 2, It is characterized in that The step of performing channel dimension splicing on the first RGB feature map and the first NIR feature map to form a high-level input feature map, and then correcting the high-level input feature map to obtain the corrected high-level input feature map specifically includes: The first RGB feature map and the first NIR feature map are concatenated in channel dimension, and a 1×1 convolution is used to implement a channel dimension reduction operation; The Softmax function is used as a correction coefficient for the high-level input feature map to filter out unimportant features.

8. The recognition method based on RGB-NIR multimodal image fusion according to claim 1, It is characterized in that Determining the object to be identified according to the final fusion feature map specifically includes: Inputting the final fusion feature into the output head of the network model to obtain a predicted output result; Constructing a loss function to perform loss calculation on the predicted output result to obtain the final screening result; The object to be identified is determined according to the final screening result.

9. The recognition method based on RGB-NIR multimodal image fusion according to claim 8, It is characterized in that The formula of the loss function is as follows: Loss=L RGBBox +L RGBCls +λ×(L NIRBox +L NIRCls ), Among them, L RGBCls and L NIRCls are the classification losses of the prediction results in RGB images and NIR images, respectively, and L RGBBox and L NIRBox are the position offset losses of the prediction results in the RGB image and the NIR image, respectively, and λ is the regularization balance term used to balance the losses between the prediction results of the RGB image and the NIR image.

10. The recognition method based on RGB-NIR multimodal image fusion according to claim 1, It is characterized in that The convolution kernel size of the convolution layer is 7*7, and the convolution kernel size of the maximum pooling layer is 3*3.

11. The recognition method based on RGB-NIR multimodal image fusion according to claim 3, It is characterized in that The first bottleneck module, the second bottleneck module, the third bottleneck module and the fourth bottleneck module are stacked 3 times, 4 times, 6 times and 3 times respectively.

12. A recognition system based on RGB-NIR multimodal image fusion, It is characterized in that include: An acquisition module acquires the RGB image and NIR image of the object to be identified; A processing module, processing the RGB image and the NIR image through a convolution layer and a maximum pooling layer in sequence, respectively, to obtain an RGB feature map and a NIR feature map; A fusion module performs feature extraction and bimodal image fusion on the RGB feature map and the NIR feature map multiple times to obtain a final fused feature map; A recognition module, which determines the object to be recognized according to the final fusion feature map; Wherein, the feature extraction is performed through a bottleneck module, and the dual-modal image fusion is performed through a dual-modal fusion module.

Citation Information

Patent Citations

  • Image salient target detection method based on modal unique and common feature extraction

    CN114663371A

  • Multi-modal feature target detection method based on dynamic convolution and attention mechanism

    CN116452937A

  • Multi-modal image segmentation method based on cross attention

    CN117911426A

  • Feature fusion and dense connection-based method for infrared plane object detection

    US20210174149A1

  • Apparatuses and methods involving multi-modal imaging of a sample

    US20220125280A1

Cited By

  • Traditional Chinese medicinal material identification method based on RGB image and near infrared spectrum

    CN121121369A

  • Multi-mode fusion carrier identification method and system for unmanned forklift tail end operation

    CN121354057A

  • Multi-modal fusion carrier identification method and system for unmanned fork truck end operation

    CN121354057B

  • Cross-modal target tracking method based on adversarial learning and feature consistent adapter

    CN121392450A

  • Cross-modal target tracking method based on adversarial learning and feature consistent adapter

    CN121392450B