An adaptive infrared and visible light dual-mode fusion detection method

Abstract

An adaptive infrared-visible light dual-mode fusion detection method belongs to the field of target detection technology. The weight matrix trained by a convolutional neural network is used as a fusion strategy for feature fusion. As the network trains, the fusion feature ratio of each part of the infrared-visible light image is adaptively and accurately adjusted. A relevant weight calculation network structure is adopted. A relevant weight calculation module composed of a multi-layer convolutional network is used to perform weight calculations on different areas of the two infrared-visible light feature maps to obtain a weight matrix of the infrared target feature map and a weight matrix of the corresponding visible light target feature map. The weight matrix is ​​then multiplied with the corresponding position of the original infrared-visible light feature map and then superimposed. The output fusion feature map increases the loss function of the training weight matrix, thereby improving the multi-task joint loss function. The present invention is applicable to the field of target detection. When fusing multi-mode information, it can better integrate the different advantage intervals of multi-mode signal sources, thereby improving the accuracy of target detection and environmental adaptability.

Description

Technical Field

[0001] The present invention relates to an adaptive infrared-visible light dual-mode fusion detection method, belonging to the technical field of target detection. Background Art

[0002] As a key branch of computer vision, object detection methods address the problem of identifying and localizing objects of interest in images or videos. Object detection technology has become integral to everyday life, with widespread applications in areas such as autonomous driving, security monitoring, smart homes, industrial automation, and medical diagnosis, significantly boosting productivity. Deep learning-based object detection methods can be categorized into two types, depending on whether or not candidate object boxes are generated: two-stage and one-stage. Two-stage methods split the object detection process into two steps: first, generating region proposals (regions of interest), and then classifying the generated proposals and fine-tuning the positions of the proposal boxes. Examples include the Region-based Convolutional Neural Network (R-CNN), Fast-RCNN, and Faster-RCNN. One-stage methods do not directly generate region proposals. Instead, they map the input image to object box locations and use regression analysis to determine the size and position of the object boxes. Examples include the Single Shot Multibox Detector (SSD) and You Only Look Once (YOLO) series.

[0003] Because the optical characteristics of targets under detection vary in complex situations, visible light images of targets under detection are easily affected by environmental factors such as weather and time of day, resulting in poor image quality. This makes traditional target detection systems based on single-modal visible light images susceptible to image quality limitations, leading to false detections and missed detections. To address this issue, many researchers have proposed using multimodal images as training data. Multimodal images, such as infrared and visible light images, have complementary advantages. Infrared images have the advantage of relying on the heat source generated by the target object and are unaffected by lighting conditions, but they cannot capture detailed target information. Visible light images have the advantage of clearly capturing the target's texture features and detailed information, but are easily affected by lighting conditions. Therefore, research on multimodal target detection has become a current research hotspot. Krotosky et al. used HOG as a feature extraction factor to extract features from input bimodal images and perform cascade fusion, then used SVM for classification to obtain detection results. Fayez Lahoud proposed a real-time feature fusion method that separates images into different layers for fusion. Xu Ningwen et al. used convolutional neural networks to perform feature-level fusion of infrared and visible light images on a small sample set. However, these methods manually adjust the weights of features by adjusting the distribution of features, making them unsuitable for changing environments. Furthermore, they all use global feature adjustment, which is insufficient for integrating multimodal features. Further research is needed to improve the complementary advantages of multimodal feature information. Summary of the Invention

[0004] In response to the problem that existing target detection methods cannot adapt to changing environments and do not fully integrate multimodal features, the main purpose of the present invention is to disclose an adaptive infrared and visible light dual-mode fusion detection method, which uses a weight matrix trained by a convolutional neural network as a fusion strategy for feature fusion, and adaptively and accurately adjusts the fusion feature ratio of each part of the infrared and visible light image as the network trains; adopts a related weight calculation network structure; uses a related weight calculation module composed of a multi-layer convolutional network to perform weight calculation on different areas of the two transmitted infrared and visible light feature maps, thereby obtaining a weight matrix of the infrared target feature map and a weight matrix of the corresponding visible light target feature map; then the weight matrix is ​​multiplied by the corresponding position of the original infrared and visible light feature map and then superimposed, and the output fusion feature map increases the loss function of the training weight matrix, thereby improving the multi-task joint loss function; so as to achieve the purpose of better integrating the different advantage intervals of the multimodal source when fusing multimodal information, thereby improving the accuracy of target detection and environmental adaptability.

[0005] The object of the present invention is achieved through the following technical solutions:

[0006] The present invention discloses an adaptive infrared-visible light dual-mode fusion detection method, which specifically includes the following steps:

[0007] Step 1: Use two backbone feature extraction networks to extract visible light image features and infrared image features respectively.

[0008] The registered infrared and visible light images are input and two backbone feature extraction networks are used to extract visible light image features and infrared image features respectively. The visible light image contains three channels of information, and the infrared image contains one channel of information, for a total of four channels of information.

[0009] Step 2: Calculate the feature weight matrix through the relevant weight calculation network to obtain the fusion feature map, and perform feature fusion adaptively;

[0010] Through a related weight calculation network composed of a multi-layer convolutional network, the weight calculation is performed on different areas of the visible light feature map and infrared feature map obtained in step 1 to obtain the weight feature map W of the infrared target feature map IR And the weighted feature map W of the feature map of the corresponding visible light target RGB After obtaining the weighted feature map, it is multiplied with the corresponding position of the original infrared and visible light feature map and then superimposed to output the fusion feature map f m (w,h). The whole process is represented by:

[0011]

[0012]

[0013] where f fus is the transformation of the convolutional network; W IR is the weighted feature map of the infrared target feature map; W RGB is the weighted feature map of the feature map of the corresponding visible light target; and is the input infrared visible light.

[0014] Fusion feature map f m The size of (w,h) and the input infrared and visible light characteristics and The core of the feature fusion network is the correlation weight calculation network, which is composed of a multi-layer convolutional neural network. The size of the input infrared and visible light feature map is m×w×h, where m represents the number of channels, and w and h represent the width and height of the feature map. The correlation weight calculation network first calculates the input infrared feature map. and the input visible light features Perform corresponding channel cascade operations:

[0015]

[0016] Where * represents the convolution operation, K i is the convolution kernel after the channel, Z concatis the feature map obtained after the channel cascade operation, and its size is ((m1+m2)×w×h). Get Z concat Then, we input multiple convolution layers and perform batch normalization after each convolution operation. Finally, we use the softmax function to compress and normalize the channel dimension:

[0017]

[0018]

[0019] The weighted feature map W of the infrared target feature map is obtained IR And the weighted feature map W of the feature map of the corresponding visible light target RGB , their sizes are (w×h), for each element in the matrix [ω IR ,ω RGB ] have ω IR +ω RGB =1.

[0020] Step 3: Use multi-task joint loss function for feature classification and bounding box regression;

[0021] After obtaining the fused feature map based on step 2, a region proposal box is generated to preliminarily divide the feature map area. The feature classification module is used to determine the proposal box containing the target, and the bounding box regression is used to adjust the position of the proposal box to make it close to the real target box position.

[0022] The feature classification module uses the softmax function as a classifier to calculate the probability value of the target contained in the initial detection box. The detection boxes are divided into two categories: containing the target and not containing the target, so as to preliminarily obtain the candidate regions containing the target. Classification loss L cls (p,i)=-logP i .

[0023] The center coordinates, width, and height of the bounding box containing the target are four-dimensional vectors A = (A x ,A y ,A w ,A h ), the bounding box regression module learns a mapping F to maximize the real box G = (G x ,G y ,G w ,G h )’s regression box R=(R x ,R y ,R w ,R h ), the mapping relationship is as follows:

[0024] F(Ax ,A y ,A w ,A h )=(R x ,R y ,R w ,R h )

[0025] (R x ,R y ,R w ,R h )≈(G x ,G y ,G w ,G h )

[0026] Train the regression module to learn the parameters W * T , input the initial target bounding box parameter φ(A), and get the predicted value d of the regression box * (A):

[0027] d * (A)=W * T ·φ(A)

[0028] The final learned parameter W * T Value for:

[0029]

[0030] The argmin function represents the variable value when the objective function value is the smallest, and λ is the proportional coefficient used to allocate the proportion of each branch. Represents the true value.

[0031] The loss function of the regression module is smooth L1 , the loss is L reg :

[0032]

[0033] After training, the regression module outputs the translation and transformation scale of each anchor box relative to the ground truth box, and corrects the position of the initial target box. Through binary classification of features and bounding box regression operations, all region proposal boxes containing the target are obtained to complete the target detection task. The subsequent network will realize target recognition. These region proposal boxes of different scales and the original infrared and visible light fusion features are input into the pooling layer, and the region proposal boxes of different sizes are adjusted and output as W p *H p Fixed size and fixed length output.

[0034] Step 4: Use the fully connected network to calculate the specific category of the target in the proposed box in the fusion feature map, and output the prediction confidence. At the same time, use the bounding box regression again to adjust the position of the proposed box;

[0035] The fully connected network consists of an object classification module and a bounding box regression module based on adaptive weight allocation. The fused feature maps from different layers are aligned via the region of interest and extracted through interpolation to form a uniformly sized fused feature map. This fused feature map then passes through two fully connected layers and is then split into two branches. Each branch also passes through one fully connected layer before being fed into the object classification module and the bounding box regression module. The activation function for each fully connected layer is the ReLU function. The height and width of each feature map are H and W, B is the batch size, and C is the number of categories.

[0036] As a method for gathering regional features, ROI alignment uses bilinear interpolation to replace quantization operations, thereby converting the step-by-step quantization feature aggregation process into a continuous operation. While traversing the candidate box, it keeps the boundary coordinates unchanged while continuing to subdivide the candidate region into K×K units. In each unit, bilinear interpolation is used to calculate the values ​​of the four fixed coordinate positions, and finally a maximum pooling operation is performed. The error back propagation formula for ROI alignment is:

[0037]

[0038] Where d(,) represents the coordinate difference between the two points, Δh and Δw are the differences between the horizontal and vertical coordinates of the feature map pixel and the floating-point number during forward propagation, which become the bilinear interpolation coefficients of the original gradient.

[0039] In order to realize the training of the dual-mode target fusion detector based on adaptive weight allocation, and also complete the training of the weight map of the adaptive weight fusion module, the multi-task joint loss function is defined as follows:

[0040]

[0041] Where: σ is the hyperparameter related to the deep learning network;

[0042] ω is the network hyperparameter for the relevant weight calculation;

[0043] i represents the anchor box index number;

[0044] L cls represents the classification loss term;

[0045] L reg represents the regression loss term;

[0046] N cls Represents the batch size of training;

[0047] Nreg Represents the size of the feature map;

[0048] λ represents the correlation coefficient, which is used to balance the weights of the classification branch and the regression branch;

[0049] η represents the weight matrix training coefficient, which is used to balance the overall loss.

[0050] t i Represents the bounding box prediction value;

[0051] Represents the value of the real box corresponding to the anchor box containing the target;

[0052] p i Represents the target confidence output by the feature classification module;

[0053] Represents the prediction confidence of the true box;

[0054] and are classification loss and regression loss respectively;

[0055] L ω (p,p i * |ω) represents the loss function of the weight matrix, When the hyperparameter of the relevant weight calculation network is ω, the error obeys the Gaussian distribution and the probability of the output being the true value is:

[0056]

[0057] In the fusion network training, the mini-batch gradient descent method is used to minimize the loss, thereby obtaining the optimal parameters of the network (σ * ,ω * ):

[0058]

[0059] The weight feature matrix W of the infrared target feature map is IR And the weight feature matrix W of the feature map of the corresponding visible light target RGB To achieve optimal distribution, the entire process is achieved through automatic adjustment.

[0060] Step 5. Use the classification loss function and regression loss function to calculate the classification loss and regression loss of the region proposal network. Use the mini-batch gradient descent method to minimize the loss and train the object classification and bounding box regression.

[0061] Step 6: The region proposal network trained in step 5 is retrained using the multi-task joint loss function to train the parameters of the entire target detection network to obtain the target box and the corresponding category confidence.

[0062] Step 7: Use the target detector trained in step 6 to perform fusion detection on the complementary information collected by different sensors, combine the advantages of different signal sources, improve the accuracy of target detection and environmental adaptability, and effectively solve the problems of missed detection and false detection of targets in complex environments.

[0063] Beneficial effects:

[0064] 1. The present invention discloses an adaptive infrared-visible light dual-mode fusion detection method, which uses an adaptive feature fusion module and a subsequent multi-task classification and regression module to calculate the optimal fusion coefficients of different feature maps at different positions to fuse features, thereby greatly improving target detection accuracy while increasing spatiotemporal adaptability.

[0065] 2. This invention discloses an adaptive infrared-visible dual-mode fusion detection method. Its weight calculation network leverages the learning and adaptive adjustment capabilities of deep learning networks to optimize weights based on prediction results, thereby enhancing the all-weather adaptability of the dual-mode fusion detection network. Furthermore, a weight map replaces a single weight, better adapting to the uneven distribution of clarity across target categories in a pair of infrared-visible dual-mode images, improving the environmental adaptability of target detection.

[0066] 3. The present invention discloses an adaptive infrared-visible light dual-mode fusion detection method, which calculates the weights of different areas of two infrared-visible light feature maps through a related weight calculation module composed of a multi-layer convolutional network to obtain the weight matrix of the infrared target feature map and the weight matrix of the corresponding visible light target feature map; the weight matrix is ​​then multiplied with the corresponding position of the original infrared-visible light feature map and then superimposed, and the output fusion feature map increases the loss function of the training weight matrix, improves the multi-task joint loss function, and enhances the target detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a flow chart of an adaptive infrared-visible light dual-mode fusion detection method disclosed in the present invention;

[0068] Figure 2 This is the basic framework of the dual-mode target fusion detection network with adaptive weight allocation in this embodiment;

[0069] Figure 3 are two different residual units in this embodiment;

[0070] Figure 4 Detailed structure of the backbone feature extraction network in this embodiment;

[0071] Figure 5 The overall structure of the feature fusion network with adaptive weight allocation in this embodiment;

[0072] Figure 6 is the relevant weight calculation network in this embodiment;

[0073] Figure 7 This is a structural diagram of the infrared-visible light fusion feature classification and bounding box regression module in this embodiment;

[0074] Figure 8 This is a schematic diagram of the principle of generating the initial detection frame in this embodiment;

[0075] Figure 9 This is a schematic diagram of the multi-task classification and regression module based on adaptive weight allocation according to this embodiment. DETAILED DESCRIPTION

[0076] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. The technical problems solved by the technical solution of the present invention and the beneficial effects thereof are also described. It should be noted that the embodiments described are only intended to facilitate understanding of the present invention and do not serve to limit the present invention in any way.

[0077] Previous target detection algorithms based solely on single-mode images (infrared or visible light) were unable to adapt to changes in weather, lighting, and time. Therefore, to better identify all-weather targets in complex spatiotemporal environments (such as pedestrian monitoring, autonomous driving, and military detection), different sensors are used to collect complementary information for fusion detection. This leverages the advantages of different signal sources and effectively solves the problems of missed and false target detection in complex environments. This embodiment leverages the different advantages of infrared and visible light camera detection information, using infrared and visible light images of the target as the source of fusion information.

[0078] This embodiment discloses an adaptive infrared-visible light dual-mode fusion detection method, such as Figure 2 As shown in the figure, the system includes backbone feature extraction, adaptive feature fusion, multi-task classification and regression, and target recognition. Feature extraction involves a dual backbone feature extraction network that extracts features from infrared and visible light images separately. The resulting two-way features are then adaptively weighted and fused through a fusion network. These features are then fed into a classifier and regressor for feature classification and bounding box regression. Finally, target recognition classifies multiple vehicle targets to produce the final detection results.

[0079] This embodiment discloses an adaptive infrared-visible light dual-mode fusion detection method for identifying a vehicle, comprising the following steps:

[0080] Step 1: Use two backbone feature extraction networks to extract visible light image features and infrared image features respectively.

[0081] The backbone feature extraction network consists of two vertically symmetrical multi-layer convolutional neural networks. The visible light image input has three channels, while the infrared image input has a single channel. The two networks are essentially identical, so the visible light image feature extraction network was chosen for analysis. The network has five layers. The first layer is relatively simple and performs preprocessing on the input image. The next four layers are largely identical, consisting of residual blocks, but with varying numbers of residual blocks and network depth. The residual blocks use a short-circuit connection structure, which effectively mitigates the vanishing gradient problem in deep learning network training.

[0082] The backbone feature has a five-layer structure. The first layer of the backbone feature extraction network has two modules: convolution layer and pooling layer. The convolution layer includes batch normalization, and the loss function used is ReLU, with a kernel size of 7×7 and a stride of 2. The pooling layer uses the maximum pooling method, with a kernel size of 3×3 and a stride of 2. The second layer consists of three residual blocks, which are composed of two different residual units, namely CB (Conv Block) and IB (Identity Block), as shown in Figure 2. Figure 3 As shown. CB is used to match the dimensional difference between the network output and input, and IB is used when the output and input dimensions match, which is used to deepen the depth of the network. The number of residual blocks in the third, fourth, and fifth layers is 4, 6, and 3, respectively. The detailed framework is as follows Figure 4 As shown in Figure 1, the number of channels, height, and width of the visible light image input to the network are represented by a three-dimensional feature vector (C, H, W). The initial vector is set to (3, H, W). After passing through the five-layer network structure, the height, height, and width are (64, H / 4, W / 4), (256, H / 4, W / 4), (512, H / 8, W / 8), (1024, H / 16, W / 16), and (2048, H / 32, W / 32).

[0083] Step 2: Calculate the feature weight matrix through the relevant weight calculation network and perform feature fusion adaptively;

[0084] The input of the feature fusion network with adaptive weight allocation is the two infrared and visible light features extracted by the backbone feature extraction network improved by the attention mechanism ( and ), the output is the feature map f corresponding to the fusion according to the weight map (weight matrix) m The core idea of ​​the feature fusion network with adaptive weight allocation is to calculate the weights of different regions of the two infrared and visible light feature maps transmitted through a related weight calculation module composed of a multi-layer convolutional network, thereby obtaining the weight feature map W of the infrared target feature map. IR And the weighted feature map W of the feature map of the corresponding visible light targetRGB After obtaining the weight map (weight matrix), multiply it with the corresponding position of the original infrared and visible light feature map and then superimpose it to output the fusion feature map f m (w,h). Let the transformation made by the convolutional network be f fus The whole process is represented by equations (1) and (2), and the overall framework is as follows: Figure 5 As shown:

[0085] like Figure 5 As shown, the fusion feature map f m The size of (w,h) and the input infrared and visible light characteristics and The core of the feature fusion network is the relevant weight calculation network, which is composed of a multi-layer convolutional neural network. Its specific structure is as follows Figure 6 As shown. The size of the input infrared and visible light feature map is m×w×h, where m represents the number of channels, and w and h represent the width and height of the feature map. The relevant weight calculation network first calculates the input infrared feature map. and the input visible light features Perform the corresponding channel concatenation operation (concat), as shown in formula (3), where * represents the convolution operation, K i is the convolution kernel after the channel, Z concat It is the feature map obtained after the concat operation, and its size is ((m1+m2)×w×h).

[0086]

[0087] Get Z concat Then, the multi-layer convolution layer is input, and batch normalization is performed after each convolution operation (the convolution layer includes batch integration operation (Batch Normalization)). The loss function used is ReLU, the kernel size is 3×3, and the step size is 2. Finally, the softmax function is used in the channel dimension for compression and normalization as shown in formula (4), and the weight feature map W of the infrared target feature map is obtained. IR And the weighted feature map W of the feature map of the corresponding visible light target RGB , their sizes are (w×h), for each element in the matrix [ω IR ,ω RGB ] have ω IR +ω RGB =1.

[0088]

[0089] The design philosophy of the correlation weight calculation network is to leverage the learning and adaptive adjustment capabilities of deep learning networks to optimize weights based on prediction results, thereby enhancing the all-weather adaptability of the dual-mode fusion detection network. Furthermore, replacing single weights with a weight map (also known as a weight matrix) better accommodates the uneven distribution of clarity across target categories within a pair of infrared and visible light dual-mode images, where some target categories in a pair have richer infrared information, while others have more visible light information.

[0090] Step 3: Use multi-task joint loss function for feature classification and bounding box regression;

[0091] The fusion features are obtained from the fusion layer mentioned above. First, the anchor box is generated to divide the feature map area. The classification module is used to determine the anchor box containing the target. The bounding box regression module is used to adjust the anchor box position to be close to the true target box position. The region proposal box is generated to contain the target to be identified, completing the target detection function. After pooling in the pooling layer, the fixed-length output lays the foundation for subsequent target recognition. The overall structure is as follows: Figure 7 shown.

[0092] Use the multi-scale anchor frame to perform a window sliding operation on the infrared and visible light dual-mode feature map extracted by the backbone feature extraction network to generate the initial detection frame. Figure 8 shown.

[0093] The feature classification module uses the softmax function as a classifier to calculate the probability value P of the vehicle target contained in the initial detection box. i (value range is [0,1]) to classify the detection frames, as shown in formula (5), and divide them into two categories: containing the target and not containing the target, so as to preliminarily obtain the candidate region containing the target. Classification loss L cls (p,i) is shown in formula (6).

[0094]

[0095] L cls (p,i)=-logP i (6)

[0096] Define the center point coordinates and width and height of the bounding box containing the target as a four-dimensional vector A = (A x ,A y ,A w ,A h ), the bounding box regression module learns a mapping F to maximize the real box G = (G x ,G y ,G w ,G h )’s regression box R=(R x,R y ,R w ,R h ), the mapping relationship is shown in formulas (7) and (8).

[0097] F(A x ,A y ,A w ,A h )=(R x ,R y ,R w ,R h ) (7)

[0098] (R x ,R y ,R w ,R h )≈(G x ,G y ,G w ,G h ) (8)

[0099] Train the regression module to learn the parameters W * T , input the initial target bounding box parameter φ(A), and get the predicted value d of the regression box * (A), as shown in formula (9).

[0100] d * (A)=W * T ·φ(A) (9)

[0101] The final learned parameter W * T Value As shown in formula (10).

[0102]

[0103] The argmin function represents the variable value that takes the minimum value of the objective function. Represents the true value.

[0104] The loss function of the regression module is smooth L1 , the loss is L reg . As shown in formula (11).

[0105]

[0106] After training, the regression module will output the translation and transformation scale of each anchor box relative to the ground truth box to correct the position of the initial target box. Through the binary classification of features and bounding box regression operations, all region proposal boxes containing vehicle targets are obtained. At this point, the target detection task has been completed, and the subsequent network will realize target recognition. These region proposal boxes of different scales and the original infrared and visible light fusion features are input into the pooling layer, and the region proposal boxes of different sizes are adjusted and output as W p *H p Fixed size and fixed length output.

[0107] Step 4: Use a fully connected network for target recognition and classification

[0108] like Figure 9 As shown in Figure 1, the fully connected network can be divided into a target classification module and a bounding box regression module based on adaptive weight allocation. The fused feature maps of different layers are interpolated and extracted through the region of interest alignment layer (ROI Align) to form a unified 7×7 fused feature map. They then pass through two fully connected layers (FC layers) and are then divided into two branches. Each branch also passes through a fully connected layer and then inputs the target classification module and bounding box regression module respectively. The activation function of each fully connected layer is the ReLU function. The parameter design of the target classification module and bounding box regression network based on adaptive weight allocation is shown in Table 1. The table shows the dimensions of the feature maps, where the height and width of each feature map are H and W, B is the batch size, and C is the number of categories.

[0109] Table 1 Parameter design of target classification module and bounding box regression network based on adaptive weight allocation

[0110]

[0111] As a method for clustering regional features, ROI Align uses bilinear interpolation to replace quantization, thereby converting the step-by-step quantization feature aggregation process into a continuous operation. While traversing the candidate box, it maintains the boundary coordinates unchanged while further subdividing the candidate region into K×K units. In each unit, bilinear interpolation is used to calculate the values ​​of the four fixed coordinate positions, and finally a maximum pooling operation is performed. The error back propagation formula for ROI Align is shown in Equation (12), where d(,) represents the coordinate difference between two points, and Δh and Δw are the differences between the horizontal and vertical coordinates of the feature map pixel and the floating-point number during forward propagation, which become the bilinear interpolation coefficients of the original gradient.

[0112]

[0113] In order to realize the training of the dual-mode vehicle target fusion detector based on adaptive weight allocation, the weight map (weight matrix) of the adaptive weight fusion module is also trained. Therefore, the multi-task joint loss function is defined as shown in Equation (13).

[0114]

[0115] Where: σ is the hyperparameter related to the deep learning network;

[0116] ω is the network hyperparameter for the relevant weight calculation;

[0117] i represents the anchor box index number;

[0118] L cls represents the classification loss term;

[0119] L reg represents the regression loss term;

[0120] N cls Represents the batch size of training;

[0121] N reg Represents the size of the feature map;

[0122] λ represents the correlation coefficient, which is used to balance the weights of the classification branch and the regression branch;

[0123] η represents the weight matrix training coefficient, which is used to balance the overall loss.

[0124] t i Represents the bounding box prediction value;

[0125] Represents the value of the real box corresponding to the anchor box containing the target;

[0126] p i Represents the target confidence output by the feature classification module;

[0127] Represents the prediction confidence of the true box;

[0128] and are classification loss and regression loss respectively;

[0129] represents the loss function of the weight matrix, as shown in Equation (7).

[0130]

[0131] Formula (7) shows that when the hyperparameter of the relevant weight calculation network is ω, the error obeys the Gaussian distribution and the probability that the output is the true value is shown in (8).

[0132]

[0133] In the fusion network training, the MBGD (mini-batch gradient descent) method is used to minimize the loss, thereby obtaining the optimal parameters of the network (σ * ,ω * ) As shown in formula (9), the weight feature matrix W of the infrared target feature map is IR And the weight feature matrix W of the feature map of the corresponding visible light target RGB To achieve optimal distribution, the entire process is achieved through automatic adjustment.

[0134]

[0135] Step 5: Training and prediction of detection network

[0136] The training process of the adaptive dual-mode fusion detector consists of three steps: training the target classification and bounding box regression modules, obtaining the proposed region, and training the entire target detector. By reasonably dividing the training steps and setting the relevant parameters, the network can converge as quickly as possible.

[0137] (1) Training target classification and bounding box regression modules

[0138] The multi-task joint loss function (6) is used for training, and the training process is the process of continuously decreasing the function loss.

[0139] (2) Obtaining the recommended area

[0140] After the target classification and bounding box regression network training is completed, the feature map is input into the trained network, and the region proposal box containing the target is obtained. This information is saved for subsequent network training. The process at this time is similar to that of target detection.

[0141] (3) Train the entire infrared and visible light vehicle target detector

[0142] The previously saved vehicle target region proposals are fed into the fully connected network. A softmax function is used to determine the specific category, and a smooth loss layer is used to accurately regress the bounding box. This completes the training of the final softmax layer and the bounding box regression layer, and the training of the entire detector network is also complete.

[0143] After the detector network is trained, its image detection process follows: an image is input, passed through the backbone feature extraction network to obtain a feature map, and then a candidate bounding box containing the object is generated. The output is fixed-length through the pooling layer, and then passed through the fully connected layer to generate a target bounding box with the target category confidence and a fine-tuned target position. The candidate box generation process often produces a large number of overlapping bounding boxes. Non-maximum suppression is used to eliminate bounding boxes with low local confidence and retain those with high confidence. This improves the accuracy and environmental adaptability of object detection.

[0144] To verify the effectiveness of the proposed adaptive infrared-visible dual-mode fusion detection method, we compared the detection performance of single-mode target detection methods (Faster-RCNN and YOLOv4) and the adaptive infrared-visible dual-mode fusion detection method on a complex urban road vehicle dataset (containing five target categories: truck, sedan, van, bus, and SUV) at different times and locations. The performance of the methods was measured using the performance parameter multi-class average precision (mAP), with higher mAP values ​​indicating better performance. Table 2 shows that the mAP values ​​of our method are both higher than those of the two traditional single-mode detection methods, confirming the advantages of our method in target detection accuracy and environmental adaptability over traditional single-mode detection methods.

[0145] Table 2 Quantitative evaluation results on the test set at different times and locations

[0146]

[0147] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An adaptive infrared and visible light dual-mode fusion detection method, characterized by: The following steps are included: Step 1: Use two backbone feature extraction networks to extract visible light image features and infrared image features respectively; Step 2: Calculate the feature weight matrix through the relevant weight calculation network to obtain the fusion feature map, and perform feature fusion adaptively; The implementation method of step 2 is: Through a related weight calculation network composed of a multi-layer convolutional network, the weight calculation is performed on different areas of the visible light feature map and infrared feature map obtained in step 1 to obtain the weight feature map W of the infrared target feature map IR And the weighted feature map W of the feature map of the corresponding visible light target RGB ; After obtaining the weighted feature map, multiply it with the corresponding position of the original infrared and visible light feature map and then superimpose it to output the fusion feature map f m (w,h); the whole process means: where f fus is the transformation of the convolutional network; W IR is the weighted feature map of the infrared target feature map; W RGB is the weighted feature map of the feature map of the corresponding visible light target; and is the input infrared visible light; Fusion feature map f m The size of (w,h) and the input infrared and visible light characteristics and The core of the feature fusion network is the correlation weight calculation network, which is composed of a multi-layer convolutional neural network. The size of the input infrared and visible light feature map is m×w×h, where m represents the number of channels, and w and h represent the width and height of the feature map. The correlation weight calculation network first calculates the input infrared feature map. and the input visible light features Perform corresponding channel cascade operations: Where * represents the convolution operation, K i is the convolution kernel after the channel, Z concat is the feature map obtained after the channel cascade operation, whose size is ((m1+m2)×w×h); get Z concat Then, multiple convolution layers are input, and batch normalization is performed after each convolution operation; finally, the softmax function is used to compress and normalize the channel dimension: The weighted feature map W of the infrared target feature map is obtained IR And the weighted feature map W of the feature map of the corresponding visible light target RGB , their sizes are (w×h), for each element in the matrix [ω IR ,ω RGB ] have ω IR +ω RGB =1; Step 3: Use multi-task joint loss function for feature classification and bounding box regression; Step 4: Use the fully connected network to calculate the specific category of the target in the proposed box in the fusion feature map, and output the prediction confidence. At the same time, use the bounding box regression again to adjust the position of the proposed box; Step 5. Use the classification loss function and regression loss function to calculate the classification loss and regression loss of the region proposal network; use the mini-batch gradient descent method to minimize the loss and train the object classification and bounding box regression modules; Step 6: The region proposal network trained in step 5 is retrained using the multi-task joint loss function to train the parameters of the entire target detection network to obtain the target box and the corresponding category confidence.

2. The adaptive infrared-visible light dual-mode fusion detection method according to claim 1, characterized in that: It also includes step 7, using the target detector trained in step 6 to perform fusion detection on the complementary information collected by different sensors, combining the advantages of different signal sources, improving the accuracy of target detection and environmental adaptability, and effectively solving the problems of missed detection and false detection of targets in complex environments.

3. The adaptive infrared-visible light dual-mode fusion detection method according to claim 1, characterized in that: The implementation method of step one is: Input the registered infrared and visible light images, and use two backbone feature extraction networks to extract visible light image features and infrared image features respectively; among them, the visible light image contains three-channel information, and the infrared image contains single-channel information, so a total of four channels of information are extracted.

4. The adaptive infrared-visible dual-mode fusion detection method according to claim 3, characterized in that: The implementation method of step 3 is: After obtaining the fused feature map based on step 2, generate a region proposal box to preliminarily divide the feature map area, use the special classification module to determine the proposal box containing the target, and use the bounding box regression to adjust the position of the proposal box so that it is close to the position of the real target box; The feature classification module uses the softmax function as a classifier to calculate the probability value of the target contained in the initial detection box. The detection frame is divided into two categories: containing the target and not containing the target, so as to preliminarily obtain the candidate area containing the target; the classification loss L cls (p,i)=-logP i ; The center coordinates, width, and height of the bounding box containing the target are four-dimensional vectors A = (A x ,A y ,A w ,A h ), the bounding box regression module learns a mapping F to maximize the real box G = (G x ,G y ,G w ,G h )’s regression box R=(R x ,R y ,R w ,R h ), the mapping relationship is as follows: F(A x ,A y ,A w ,A h )=(R x ,R y ,R w ,R h ) (R x ,R y ,R w ,R h )≈(G x ,G y ,G w ,G h ) Train the regression module to learn the parameters W * T , input the initial target bounding box parameter φ(A), and get the predicted value d of the regression box * (A): The final learned parameters Value for: The argmin function represents the variable value when the objective function value is the smallest, and λ is the proportional coefficient used to allocate the proportion of each branch. represents the true value; The loss function of the regression module is smooth L1 , the loss is L reg : After training, the regression module outputs the translation and transformation scale of each anchor frame and the real frame, and corrects the position of the initial target frame. Through the binary classification and bounding box regression operation of the features, all the region proposal frames containing the target are obtained to complete the target detection task. The subsequent network will realize the target recognition. These region proposal frames of different scales and the original infrared and visible light fusion features are input into the pooling layer, and the different sizes of the region proposal frames are adjusted and output as W. p *H p Fixed size and fixed length output.

5. The adaptive infrared-visible light dual-mode fusion detection method according to claim 1, characterized in that: The implementation method of step 4 is: The fully connected network is divided into a target classification module and a bounding box regression module based on adaptive weight allocation. The fused feature maps of different layers are aligned through the region of interest and interpolated to form a fused feature map of uniform size. The fused feature maps are then passed through two fully connected layers and then divided into two branches. Each branch also passes through a fully connected layer before being input into the target classification module and the bounding box regression module. The activation function of each fully connected layer is the ReLU function. The height and width of each feature map are H and W, B is the batch size, and C is the number of categories. As a method of gathering regional features, ROI alignment uses bilinear interpolation to replace quantization operations, thereby converting the step-by-step quantization feature aggregation process into a continuous operation. It keeps the boundary coordinates unchanged while traversing the candidate box, and continues to subdivide the candidate area into K×K units. In each unit, bilinear interpolation is used to calculate the values ​​of the four fixed coordinate positions, and finally a maximum pooling operation is performed. The error back propagation formula of ROI alignment is: Where d(,) represents the coordinate difference between the two points, Δh and Δw are the differences between the horizontal and vertical coordinates of the feature map pixel and the floating point number during forward propagation, which become the bilinear interpolation coefficients of the original gradient; In order to realize the training of the dual-mode target fusion detector based on adaptive weight allocation, and also complete the training of the weight map of the adaptive weight fusion module, the multi-task joint loss function is defined as follows: Where: σ is the hyperparameter related to the deep learning network; ω is the network hyperparameter for the relevant weight calculation; i represents the anchor box index number; L cls represents the classification loss term; L reg represents the regression loss term; N cls Represents the batch size of training; N reg Represents the size of the feature map; λ represents the correlation coefficient, which is used to balance the weights of the classification branch and the regression branch; η represents the weight matrix training coefficient, which is used to balance the overall loss; t i Represents the bounding box prediction value; Represents the value of the real box corresponding to the anchor box containing the target; p i Represents the target confidence output by the feature classification module; Represents the prediction confidence of the real box; and are classification loss and regression loss respectively; represents the loss function of the weight matrix, When the hyperparameter of the relevant weight calculation network is ω, the error obeys the Gaussian distribution and the probability of the output being the true value is: In the fusion network training, the mini-batch gradient descent method is used to minimize the loss, thereby obtaining the optimal parameters of the network (σ * ,ω * ): The weight feature matrix W of the infrared target feature map is IR And the weight feature matrix W of the feature map of the corresponding visible light target RGB To achieve optimal distribution, the entire process is achieved through automatic adjustment.

Images