A multi-modal target detection method for complex scenes

Abstract

The present application belongs to the field of computer vision and pattern recognition, and discloses a kind of multi-modal target detection method for complex scene, for solving the problem that single modal target detection method cannot accurately detect the target of complex scene.The multi-modal target detection method of the present application first constructs a double-input double-flow backbone network based on YOLOv5 model to obtain multi-scale features of infrared and visible light images, then builds a cross-modal feature fusion module based on attention mechanism and intermediate fusion strategy to fuse multi-modal features and solve the modal balance problem, then inputs the photographed, registered and divided training set image pairs into the detection network built to obtain the weight file of the best result, finally loads the weight file and test set image pairs into the model to obtain the infrared and visible light image pairs with detection frame and confidence.The method of the present application can detect the target in complex scene, and has higher accuracy, fewer parameters and good real-time performance.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and pattern recognition, and particularly relates to a multi-modal target detection method for complex scenes. BACKGROUND

[0002] Target detection is one of the main technologies of computer vision and pattern recognition, and is an important technical means of intelligentization and informatization, and plays an important role in remote sensing target recognition, automatic driving and industrial automation. However, the image information under a single modality has many deficiencies, and is easily limited by environmental factors such as low light intensity and occlusion, resulting in unstable and unreliable detection results. Taking visible light as an example, due to the characteristics of its imaging method, the features captured under low light intensity lack sufficient details, and the contour boundary between the foreground and the background is blurred, making it difficult for the detection method to accurately locate and identify the target. Therefore, more and more researchers pay attention to multi-modal target detection technology, and use multiple sensors to obtain multi-modal information to improve the recognition accuracy of the target detection model.

[0003] Hwang et al. published a multi-modal dataset in CVPR in 2015, which provides aligned images of two modalities of visible light and infrared for pedestrian detection, named Kai st, which opens the door to the field of multi-modal target detection. Zhang et al. proposed a method named CIAN based on this dataset, which converges two cross-modal intermediate feature maps into a unified feature map under the guidance of cross-modal interactive attention, and further fuses multi-modal feature information. Zheng et al. proposed GFD-SSD based on SDD detection model, which uses two different novel gating fusion units to learn the cross-modal feature maps generated from the intermediate layers of two backbone networks. Zhang et al. proposed an adaptive multi-modal feature fusion method, which combines intra-modal and inter-modal attention mechanisms, allowing the network to learn multi-modal features and dynamically weight and fuse them. Cao et al. proposed an MCFF module based on channel attention mechanism, which dynamically collects features from RGB and thermal modalities according to the light condition to obtain better fusion effect.

[0004] Based on the above research content, it can be known that multi-modal images can improve the detection accuracy of detection. However, due to the great difference between cross-image modalities, it may cause feature modality imbalance problem, which brings difficulties to the deployment and application of multi-modal detection method, so it is necessary to fully combine the complementarity of cross-modal to generate robust features. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art, provide a multi-modal target detection method for complex scenes, which can realize detection on image pairs with complex scenes, suppress feature modal imbalance problems, has high detection accuracy and better real-time performance.

[0006] The technical solution of the present application to solve the above technical problems is:

[0007] A multi-modal target detection method for complex scenes, comprising the following steps:

[0008] (S1), using infrared and visible light two different modal cameras to take pictures of the detection target body in complex scenes to collect image pairs, and putting the photographed images into image set A;

[0009] (S2), putting the image pairs in image set A into an image registration algorithm for registration to obtain a registered image set B;

[0010] (S3), putting each pair of images in image set B into a labeling software for labeling of target class and position information to obtain image set C, and then dividing any selected image pairs in image set C into training image set D and test image set E according to 8:2;

[0011] (S4), building a double-flow target detection network based on YOLOv5 framework, and building a cross-modal feature fusion module based on attention mechanism;

[0012] (S5), inputting image set D into the double-flow target detection network based on YOLOv5 framework for training and optimization to obtain the optimal weight file after training;

[0013] (S6), loading the trained weight file into the double-flow detection network, and putting image set E into the network for testing to obtain image pairs with detection frames and confidence.

[0014] Preferably, in step (S1), the complex scene includes but is not limited to low light intensity, haze, occlusion, camouflage and forest.

[0015] Preferably, in step (S2), the image registration algorithm is used to align the misaligned image pairs, so that the target positions in the cross-modal image pairs remain consistent.

[0016] Preferably, in step (S3), the position and class of the target in the image set Before of one of the registered modalities are labeled. The labeled image set After is randomly allocated to obtain a training set and a test set, and finally the label file is converted into a txt format file suitable for YOLO training.

[0017] Preferably, in step (S4), the construction of the dual-stream detection network comprises the following steps:

[0018] (S4-1), two parallel backbone networks are built using the CSPDarkNet network for extracting multi-scale feature information of infrared images and visible light images respectively, and the network down-sampling channel numbers are 64, 128, 256, 512 and 1024. After the fourth stage at the deep level of 512 channels in the two parallel backbone networks, the features of this layer are spliced along the channel dimension to fuse the deep semantic and feature information. In order to reduce the number of model parameters, the 1024 channels after splicing are compressed to 512 again using a 1×1 size convolution operation, and further down-sampling is performed using convolution;

[0019] (S4-2), a cross-modal feature fusion module is built based on attention mechanism and intermediate fusion strategy, which not only preserves the original features of the modal, but also compensates for the difference features, fully combines the complementarity of cross-modal to produce robust features to solve the problem of feature imbalance between cross-modal, in order to improve the fusion efficiency, the module is inserted into different stages of the dual-stream backbone network several times;

[0020] (S4-3), the feature map of the fused cross-modal difference feature is added to the dual-stream backbone network to enhance the feature map in the backbone network, and the enhanced feature maps of different scales are linearly superimposed and input into the detection layer;

[0021] (S4-4), in the detection layer, the multi-scale features are aggregated using the feature pyramid layer to fully fuse the context information of the target and improve the detection accuracy and efficiency;

[0022] (S4-5), before network training, the K-means clustering algorithm is used to obtain anchor box priors from the data set for the generation of bounding box prediction.

[0023] Preferably, in step (S4-2), the backbone network is composed of two parallel dual-stream CSPDarkNet, one of which outputs the visible light image feature map F R , and the other outputs the infrared image feature map F I; first, the difference feature map between modalities is obtained by subtracting the two feature maps; second, the infrared and visible light feature maps are respectively averaged along the horizontal and vertical coordinates to obtain two new feature maps, which are then spliced and sent to a shared 1x1 convolution kernel transformation function for information interaction, after which the spliced feature maps are separated and processed through two different 1x1 convolution kernels to obtain new outputs, and the attention weights are obtained by using sigmoid as the activation function on the outputs, and the complementary feature map is obtained by multiplying the weights and the difference feature map, and finally the complementary feature map and the visible light feature map F R and the infrared feature map F I are added to obtain the enhanced feature map, and the expression of the whole process is as follows:

[0024]

[0025] where σ represents the sigmoid function, f 1×1 represents 1x1 convolution operation, GAP represents average pooling along two spatial ranges (H, 1) or (1, W), and represents dot product operation.

[0026] Preferably, in step (S5), the loss function of network training contains three parts, i.e. classification loss L cls , target loss L obj and regression positioning loss L box , and the total loss is equal to the sum of the three losses:

[0027] L=L cls +L obj +L box (2)

[0028] The classification loss and the target loss are both defined based on binary cross entropy, and the expressions are as follows:

[0029]

[0030] where y i is the label value, p is the predicted probability, and N is the total number of categories; for the regression positioning loss, an improved CIoU loss is used for calculation, and the loss function is defined as follows:

[0031]

[0032] where IoU is the value of intersection over union, e angle is the angle offset coefficient term of the center point positions of the two bounding boxes b and b gt , which is used to punish the excessive directional angle offset, ρ 2 (.) is the Euclidean distance calculation of the center point of the bounding box, and c 2 represents the center point of the bounding box bgt The diagonal distance of the minimum bounding rectangle, v is the shape penalty term constructed by the squared difference between the width and height of the predicted box and the ground truth box, and α is the balance coefficient.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] 1. The multimodal target detection method for complex scenes in this invention combines the backbone network construction idea of ​​partial image fusion with a channel concatenation method to achieve deep feature fusion. Two parallel backbone networks are built using the CSPDarkNet network to extract multi-scale feature information from infrared and visible light images, respectively. At the deep level of the two parallel backbone networks (512 channels), the features extracted by the dual-stream backbones are concatenated along the channel dimension to fuse and interact with deeper semantic and feature information. A 1×1 convolution operation is used to compress the concatenated 1024 channels back to 512, and further downsampling is used to expand the number of channels to 1024. This scheme can efficiently reduce parameters while obtaining good detection results.

[0035] 2. The multimodal target detection method of the present invention for complex scenarios combines the characteristics of the differential amplifier circuit where the common-mode signal is suppressed and the differential-mode signal is amplified. Based on the attention mechanism, a differential fusion module is proposed to interact with cross-modal feature information. This not only preserves the original features of the modality, but also compensates for the difference features to enhance the original feature information, so as to solve the problem of cross-modal feature imbalance.

[0036] 3. The classification and target loss of the multimodal target detection method in complex scenes of the present invention are based on the commonly used binary cross-entropy loss function, while the localization loss is reconstructed by combining the characteristics of CIoU loss. An angle offset coefficient is introduced into the distance penalty term to further limit the position offset of the predicted box and make it closer to the position of the real bounding box. At the same time, the shape penalty term is improved to replace the CIoU length and width consistency parameter, and a more stringent squared difference is used as the penalty term, so that the true size of the predicted box is more consistent with the true position bounding box. Attached Figure Description

[0037] Figure 1 This is a flowchart of the multimodal target detection method for complex scenarios according to the present invention.

[0038] Figure 2 This is a schematic diagram illustrating the specific process of the network structure of the multimodal target detection method for complex scenarios according to the present invention.

[0039] Figure 3 This is a schematic diagram of the cross-modal feature fusion module of the multimodal target detection method for complex scenes according to the present invention. Detailed Implementation

[0040] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0041] See Figure 1 The multimodal target detection method for complex scenes of the present invention includes the following steps:

[0042] (S1) Use two different modes of cameras, infrared and visible light, to take pictures of the target subject in a complex scene and collect image pairs, and put the captured images into image set A. The complex scene includes, but is not limited to, low light intensity, haze, obstruction, camouflage and forest.

[0043] (S2) Since two cameras with different modes are used for shooting, there is a certain angle difference between the lenses. The same target position in the images of different modes is misaligned. It is necessary to use an image registration algorithm to correct it. The image pairs in image set A are put into the image registration algorithm for registration to obtain the registered image set B.

[0044] (S3) Input each pair of images in image set B into the labeling software labelme to label the target category and location information to obtain image set C. Then convert the label file into a txt format file suitable for YOLO training. Finally, randomly select image pairs in image set C to divide the training image set D and test image E in an 8:2 ratio.

[0045] (S4) A dual-stream object detection network was built based on the YOLOv5 framework, and a cross-modal feature fusion module was built based on the attention mechanism;

[0046] (S5) Load the image set D into the dual-stream object detection network based on the YOLOv5 framework for training and optimization. Use the SGD algorithm as the optimizer. Stop updating the weights and bias parameters when the set number of training rounds is reached. Finally, retain the optimal weight file after training.

[0047] (S6) Load the trained weight file into the dual-stream detection network and put the image set E into the network for testing to obtain image pairs with detection boxes and confidence scores.

[0048] See Figure 1For images with complex backgrounds, incorporating multimodal approaches into object detection tasks can enhance information richness and further improve detection accuracy. For object detection with multimodal inputs, feature mode imbalance is crucial; misalignment and insufficient integration between different modalities can lead to uneven feature contributions and representations. RGB and thermal modal features differ in pedestrian morphology, texture, and properties. The guiding principle for fusion module design is to preserve and enhance robust features, suppress redundant and interfering features, and complement differential features in the cross-modal feature space. We combine state-of-the-art attention mechanisms and differential amplifier circuit characteristics to propose a lightweight and efficient cross-modal feature fusion method for multimodal object detection. Our solution involves decomposing cross-modal features, preserving specific features within a modality such as color and thermal features, and complementing and enhancing feature differences between modalities. The channel compression rate is adjusted within the attention mechanism, thus enabling efficient cross-modal feature fusion with minimal parameters. Furthermore, to obtain multi-scale fusion features, the fusion module is inserted into feature layers of different scales in the dual-stream backbone network to achieve complementary enhancement of cross-modal information. Our invention, after the fourth stage (512 channels deep within two parallel backbone networks), concatenates the features of this layer along the channel dimension to fuse deep semantic and feature information. Then, a 1×1 convolution operation is used to compress the concatenated 1024 channels back to 512. Compared to the typical fully symmetrical dual-stream backbone network construction method, our method significantly reduces the number of network parameters while improving detection performance. Our method also introduces more penalty terms in the loss function to limit the position offset and size of the predicted bounding boxes generated by the algorithm, making them more consistent with the ground truth boxes in the labels, further improving the accuracy of object detection. Through the above settings, the multimodal object detection method of this invention has the characteristics of high accuracy and few parameters.

[0049] See Figure 2 In step (S4), the construction of the dual-stream detection network includes the following steps:

[0050] (S4-1) Two parallel backbone networks were built using the CSPDarkNet network to extract multi-scale feature information from infrared and visible light images, respectively. The downsampling channels of the networks were 64, 128, 256, 512, and 1024, respectively. At the deepest level of the two parallel backbone networks, i.e., at the 512-channel level, the features of this layer were concatenated along the channel dimension to fuse deep semantic and feature information. In order to reduce the number of model parameters, a 1×1 convolution operation was used to compress the concatenated 1024 channels back to 512, and further downsampling was performed using convolution.

[0051] (S4-2) Combining the amplification characteristics of the differential amplifier circuit, a cross-modal feature fusion module is built based on the attention mechanism and intermediate fusion strategy. This module not only retains the original features of the modality, but also compensates for the differences in features. It fully combines the complementarity of cross-modality to generate robust features to solve the feature imbalance problem between cross-modality. In order to improve the fusion efficiency, the module is inserted into different stages of the dual-stream backbone network multiple times.

[0052] (S4-3) The feature maps that fuse cross-modal difference features are added into the dual-stream backbone network to enhance the feature maps in the backbone network. At the same time, the enhanced feature maps of different scales are linearly superimposed and input into the detection layer.

[0053] (S4-4) In the detection layer, a feature pyramid layer is used to aggregate multi-scale features, fully integrate the contextual information of the target, and improve detection accuracy and efficiency.

[0054] (S4-5) Before network training, K-means clustering algorithm is used to obtain anchor box priors from the dataset for predicting bounding box generation.

[0055] See Figure 1 In step (S5), the loss function for network training consists of three parts, namely classification loss L cls Target loss L obj and regression localization loss L box The total loss equals the sum of these three losses:

[0056] L = L cls +L obj +L box (1)

[0057] Both classification loss and target loss are based on the definition of binary cross-entropy, and their expressions are as follows:

[0058]

[0059] Where y i Let p be the label value, N be the predicted probability, and N be the total number of categories. For regression localization loss, an improved CIoU loss is used for calculation, and the loss function is defined as follows:

[0060]

[0061] Where IoU is the crossover-union ratio, e angle For b and b gt The angular offset coefficient term at the center positions of the two bounding boxes is used to penalize excessive angular offset, ρ 2 (.) represents the Euclidean distance between the bounding box center point and the boundary box center point, used for penalty. 2 Represents b and bgt The diagonal distance of the minimum bounding rectangle, v is the shape penalty term constructed by the squared difference between the width and height of the predicted box and the ground truth box, and α is the balance coefficient.

[0062] See Figure 3 The following section describes the method for building a cross-modal feature fusion module:

[0063] In step (S4-2), the backbone network consists of two parallel dual-stream CSPDarkNets, one of which outputs a visible light image feature map F. R Another CSPDarkNet output is an infrared image feature map F. I First, subtraction is performed on the two feature maps to obtain the difference feature map between the modes. Second, average pooling is performed on the infrared and visible light feature maps along the horizontal and vertical coordinates respectively to obtain two new feature maps. These are then concatenated and sent to a shared 1×1 convolutional kernel transformation function for information exchange. The concatenated feature maps are then separated and transformed by two different 1×1 convolutional kernels to obtain a new output. A sigmoid function is used as the activation function on this output to obtain attention weights. Multiplying these weights by the difference feature map yields the complementary feature map. Finally, the complementary feature map is combined with the visible light feature map F. R and infrared feature map F I The enhanced feature map can be obtained by adding them together. The expression for the whole process is as follows:

[0064]

[0065] Where σ represents the sigmoid function, f 1×1 ∠ represents a 1×1 convolution operation, GAP represents average pooling along two spatial ranges (H, 1) or (1, W), and ⊙ represents a dot product operation.

[0066] The above are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above content. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A multi-modal object detection method for complex scenes, characterized in that, It comprises the following steps: (S1), using infrared and visible light two different modal cameras to take pictures of the target subject in a complex scene, and put the photographed images into image set A; (S2), using image registration algorithm to register the image pairs in image set A to obtain registered image set B; (S3), put each pair of images in image set B into the labeling software to label the target class and position information to obtain image set C, and then divide any selected image pair in image set C into training image set D and test image set E according to 8:2; (S4), based on YOLOv5 framework, a double-flow target detection network is built, and a cross-modal feature fusion module is built based on attention mechanism, wherein the construction of the double-flow detection network comprises the following steps: (S4-1), using CSPDarkNet network to build two parallel backbone networks for extracting multi-scale feature information of infrared images and visible light images, respectively, and the network down-sampling channel number is 64, 128, 256, 512 and 1024, at the deep 512 channel number of the two parallel backbone networks, i.e. after the fourth stage, the features of this layer are spliced along the channel dimension to fuse the deep semantic and feature information, in order to reduce the model parameter amount, the 1024 channel number after splicing is compressed to 512 by using 1×1 size convolution operation, and the channel number is expanded to 1024 by further down-sampling convolution; (S4-2), a cross-modal feature fusion module is built based on attention mechanism and intermediate fusion strategy and is inserted into different stages of the dual-stream backbone network multiple times; the backbone network is composed of two parallel dual-stream CSPDarkNets, one of which outputs visible light image feature map F R and the other outputs infrared image feature map F I ; first, subtraction operation is performed on the two feature maps to obtain inter-modal difference feature map; second, average pooling operation is performed on the infrared and visible light feature maps along the horizontal and vertical coordinates respectively to obtain two new feature maps, which are then spliced and sent to a shared 1x1 convolution kernel transformation function for information interaction; then the spliced feature map is separated and new outputs are obtained after two different 1x1 convolution kernel transformations, on which sigmoid is used as the activation function to obtain attention weights, and the weights are multiplied with the difference feature map to obtain complementary feature map, finally the complementary feature map is added to the visible light feature map F R and the infrared feature map F I to obtain the enhanced feature map, and the expression of the whole process is as follows: where σ denotes a sigmoid function, f 1×1 denotes performing a 1x1 convolution operation, GAP denotes average pooling along two spatial ranges (H, 1) or (1, W), and represents a dot product operation; (S4-3), the feature map of the fused cross-modal difference feature is added to the double-flow backbone network to enhance the feature map in the backbone network, and the enhanced feature maps of different scales are linearly superimposed and input into the detection layer; (S4-4), in the detection layer, the multi-scale features are aggregated using the feature pyramid layer to fully fuse the context information of the target and improve the detection accuracy and efficiency; (S4-5), before network training, the K-means clustering algorithm is used to obtain anchor box priors from the data set for the generation of prediction bounding boxes; (S5), input the image set D into the double-flow target detection network based on YOLOv5 framework for training and optimization to obtain the optimal weight file after training; (S6), load the trained weight file into the double-flow detection network, and put the image set E into the network for testing to obtain the image pair with detection frame and confidence.

2. The method for multi-modal target detection under complex scenes according to claim 1, characterized in that, In step (S1), the complex scene includes but is not limited to low light intensity, haze, occlusion and camouflage. 3.The method for multi-modal object detection in complex scenes of claim 1, wherein, In step (S2), the image registration algorithm is used to align the misaligned image pairs, so that the target position in the cross-modal image pairs remains consistent.

4. The method for multi-modal object detection in complex scenes of claim 1, wherein, In step (S3), the target in the image set Before of one of the registered modalities is labeled for position and class, the labeled image set After is randomly assigned to obtain the training set and test set, and finally the label file is converted into a txt format file suitable for YOLO training.

5. The method for multi-modal object detection in complex scenes of claim 1, wherein, In step (S5), the loss function of network training contains 3 parts, i.e. classification loss L cls , target loss L obj and regression positioning loss L box , and the total loss is equal to the sum of the three losses: L = L cls + L obj + L box (2) The classification loss and target loss are both defined based on binary cross entropy, and the expressions are as follows: where y i is the label value, p is the predicted probability, and N is the total number of classes; for regression positioning loss, the improved CIoU loss is used for calculation, and the loss function is defined as follows: Where IoU is the crossover-union ratio, e angle For the predicted box b and the true box b gt The angular offset coefficient term at the center points of the two bounding boxes is used to penalize excessive angular offset of the predicted box, ρ 2 (.) represents the Euclidean distance to the center point of the bounding box. 2 Represents b and b gt The diagonal distance of the minimum bounding rectangle, v is the shape penalty term constructed by the squared difference between the width and height of the predicted box and the ground truth box, and α is the balance coefficient.

Images