Personnel detection method based on cross-modal fusion model of multi-spectral target detection

Abstract

The application discloses a personnel detection method based on a cross-modal fusion model of multi-spectral target detection, first collects personnel image data in a complex scene, and classifies and labels the personnel image data to construct a data set; the data set is preprocessed; a double-flow detection model is designed, wherein a double-flow feature extraction network and a CFT module are included; a Faster R-CNN is used as a basic detector to detect basic features, and another branch is introduced as an infrared feature detection branch, so that a double-flow convolutional neural network is constructed; the CFT module is introduced, so that the network can pay more attention to fine-grained features with high correlation and importance in the two branches, and the influence of redundant information is reduced; compared with the prior art, the double-flow detection model and the CFT module used as a channel for information fusion between branches effectively improve the accuracy and generalization ability of the overall detection model, and are suitable for personnel monitoring tasks in various complex scenes, and provide a feasible and efficient technical solution for personnel detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-spectral image processing and target detection, and particularly relates to a personnel detection method based on a multi-spectral target detection cross-modal fusion model. TECHNICAL BACKGROUND

[0002] The multi-spectral target detection task is a complex task involving multiple disciplines and fields such as sensor technology, data acquisition and preprocessing, target detection, cross-modal fusion, and domain adaptation. Yukihiro et al. studied the combination of the prediction results of multiple models, especially in the target detection task under low light conditions, and used domain adaptation technology to improve the robustness and generalization ability of the model. Their work was published in the paper "YOLO in the Dark-Domain Adaptation Method for Merging Multiple Models. Kai Kang et al. introduced "generalizable pedestrian detection" to solve the problem of poor generalization ability of target detection algorithms on different environments and datasets. Their work has been published in the paper "Generalizable Pedestrian Detection: The Elephant In The Room". Yanwei Pang et al. proposed in 2020 to use weather conditions to adaptively adjust the input image to enhance the robustness and accuracy of the target detection algorithm in adverse weather conditions. Their work has been published in the paper "Image-Adaptive YOLO for Object Detection in Adverse Weather Conditions". Wang, Y. et al. proposed an adaptive fusion method based on convolutional neural networks for multi-spectral pedestrian detection, which realized the adaptive fusion of different modal information through dynamic weight adjustment. Their work has been published in the paper "Adaptive fusion for multi-spectral pedestrian detection using convolutional neural networks".

[0003] Most traditional multi-spectral target detection cross-modal models use convolution as the fusion channel between multiple modalities. Using convolution as the fusion channel cannot obtain global information between multiple modalities, and the robustness cannot be guaranteed. Since convolution only focuses on local information and the detection accuracy will also decrease in the case of insufficient receptive field. SUMMARY

[0004] Invention purposes: In view of the problems pointed out in the background art, the present application provides a personnel detection method based on a multi-spectral target detection cross-modal fusion model, which detects RGB images and thermal images through a double-flow model and fuses the features of the double branches through a fusion channel CFT based on a Transformer module, thereby improving the personnel detection accuracy and robustness.

[0005] Technical solutions: The present application provides a personnel detection method based on a multi-spectral target detection cross-modal fusion model, comprising the following steps:

[0006] Step 1: Collect personnel images under various complex conditions and classify, manually label them to make a self-built dataset; pre-process the original pictures of the self-built dataset, and divide the processed pictures into a training set and a validation set according to a certain proportion;

[0007] Step 2: Build a double-flow detection network, build a CFT (Cross-Modality Fusion Transformer) module as a fusion module of the two branches and design a loss function; the double-flow detection network uses a residual neural network ResNet as the backbone network, and also includes an RGB branch and a Thermal branch; the CFT module is used to fuse the features of the RGB branch and the Thermal branch, and find the potential relationship between the two branches;

[0008] Step 3: Input the pictures of the public dataset into the backbone network of the double-flow detection network for hierarchical feature extraction, then input them into the double-flow detection network for detailed feature extraction and feature fusion, input the extraction results into an MLP (Multilayer Perceptron) to obtain classification results and prediction results, and obtain the best pre-training weight through multiple iterations and test the effect in the test set of the public dataset;

[0009] Step 4: Process the training weight obtained in the public dataset to adapt to the self-built dataset, and obtain the best training weight through multiple iterations based on the pre-training weight;

[0010] Step 5: Use the trained weight to make a prediction and visualize the prediction result.

[0011] Further, in step 1, the various complex scenes include rainy days, foggy days, and low-light conditions, the pre-processing of the images includes: enhancing the image data through any one or several of flipping, panning, and cropping, and using a Gaussian filter to denoise the image to make it clearer and more readable, finally using LabelImg to manually label the image, and exporting the image and label information.

[0012] Further, the step 1 further comprises the following steps:

[0013] Step 1.1: read the absolute addresses of all pictures into the memory and put them into a list;

[0014] Step 1.2: divide the list into a training set picture address list and a verification set picture address list according to the principle of 8:2;

[0015] Step 1.3: move the pictures represented by the training picture addresses to the train folder, and move the pictures represented by the verification set picture address list to the verify folder, and complete the division of the data set.

[0016] Further, the RGB branch is based on the Faster RCNN model, and the last five maximum pooling layers and fully connected layers in the Faster RCNN model are removed, and the last four 14x14conv2d, 512 of the feature extraction layer are replaced with a DConv2d.

[0017] Further, the Thermal branch is based on the VGG model, and the linear layer and the Dropout layer that play a classification role are deleted, only the feature extraction layer in the model is used, and the step and padding of the convolution in the model are modified, so that the image size does not change after passing through each convolution layer, and then the task of changing the image size is handed over to the pooling layer. The specific padding size is as follows:

[0018] W out =(W in -F+2P) / S+1

[0019] H out =(H in -F+2P) / S+1

[0020] Where, W out is the output image width, W in is the input image width. H in is the input image height, H out is the output image height, F is the convolution kernel size, S is the step, and P is the padding size.

[0021] Further, the CFT module is composed of an inter-layer normalization module LayerNorm, a multi-head attention module Multi-HeadAttention, and a multi-layer perception MLP. The RGB branch feature map and the thermal branch feature map are spliced and then added with position encoding, and are sequentially subjected to the inter-layer normalization module LayerNorm, the multi-head attention module Multi-HeadAttention, the inter-layer normalization module LayerNorm, and the multi-layer perception MLP for feature fusion, to obtain the RGB branch feature map and the thermal branch feature map after fusion by the CFT module. The multi-head attention module is a stack of four multi-head attentions.

[0022] Further, the dual-flow detection network includes a plurality of CFT modules, and the RGB branch and the thermal branch are each composed of the same number of stages. After feature extraction of a picture by a certain stage of the RGB branch and the thermal branch, respective feature maps are formed. The feature maps are first subjected to first fusion to obtain the most significant features, and then the most significant features are subjected to second fusion by the CFT module. Subsequently, the feature maps after the second fusion are subjected to an addition operation. Finally, the feature maps after the second fusion are mapped to the two branches, and are subjected to an addition operation by the feature maps of the respective branches to enter the next stage. The operation of the next stage is consistent with that of the previous stage.

[0023] Further, the loss function designed in step 2 is as follows:

[0024] The loss function of the dual-flow detection network is designed. Since the dual-flow model is a cross-modal fusion model, the loss function includes a bounding box loss L box , a classification loss L cls , and a confidence loss L conf . The total loss L total is calculated according to the following formula.

[0025] L total =L box +L cls +L conf =L box +L cls +L noobj +L obj

[0026]

[0027] L cls =-∑y i *log(p i )

[0028]

[0029]

[0030] wherein, L box The generalized IOU loss is adopted, whether the jth prediction box (obj) of the ith grid is a positive sample, L cls The cross-entropy loss function is adopted, y represents the true value, p represents the probability value predicted by the model, and L noobj , L obj The mean square error is adopted.

[0031] Further, the public data set selects FLIRADAS and LLVIP, reads the pictures and labels in the public data set, and inputs the built double-flow detection network model.

[0032] Further, the step 4 specifically operates as follows.

[0033] Step 4.1: Modify the total number of categories in the weight file obtained in step 3 to adapt to the total number of categories of the self-built data set.

[0034] Step 4.2: Replace the original MLP layer in the weight file obtained in step 3 with a self-defined MLP layer adapted to the self-built data set.

[0035] Beneficial effects:

[0036] 1. The present application adopts a double-flow mode in the image feature extraction network. In the RGB branch, the convolution in the last 4 layers of the feature extraction layer of the Faster RCNN model is replaced with a dilated convolution, which not only ensures the accuracy of the RGB branch but also reduces the parameter amount of the branch. In the Thermal branch, the feature extraction layer of the VGG model is adopted, and the ordinary convolution in the first module of the VGG model is replaced with a depth separable convolution, and the channel number of the ordinary convolution in the second module is reduced, so that the branch can more effectively extract thermal images.

[0037] 2. The present application adopts a CFT (Cross-Modality Fusion Transformer) module to fuse the features of the double branches on the basis of the double-flow feature extraction network, which not only enhances the robustness of the model but also improves the detection accuracy. The CFT is a plug-and-play module, which can dynamically adjust the number of CFT modules to make the model adapt to more complex application scenarios.

[0038] 3. The present application first trains with an object detection data set irrelevant to personnel detection, and then trains with a public personnel detection data set to verify the generalization effect of the double-flow model.

[0039] 4. The present application further makes the model lightweight by modifying the original VGG to ensure that the feature extraction effect remains unchanged. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the process of the present invention.

[0041] Figure 2 This is a schematic diagram of the RGB branch in a dual-stream model.

[0042] Figure 3 This is a schematic diagram of the Thermal branch of the two-stream model.

[0043] Figure 4 This is a simplified diagram of the CFT module.

[0044] Figure 5 This is a diagram of the overall structure of the dual-flow model.

[0045] Figure 6 This is an illustration of GIoU. Detailed Implementation

[0046] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] This invention proposes a person detection method based on a cross-modal fusion model of multispectral target detection. The overall process is as follows: Figure 1 As shown, the specific steps include:

[0048] Step 1: Use web crawler scripts to collect images of people in various complex situations, classify them, and manually label them to create a self-built dataset.

[0049] This project utilizes web crawlers to acquire images and heatmaps of people in various complex scenes, such as rainy days, foggy days, and low-light conditions. The images are then preprocessed, including enhancing the image data through one or more methods such as flipping, translation, and cropping, and using a Gaussian filter to denoise the images, making them clearer and more readable. Finally, the images are manually labeled using LabelImg, and the images and label information are exported.

[0050] Step 2: Preprocess the collected original images of personnel, and divide the processed images into training and validation sets according to a certain ratio. This includes the following steps;

[0051] Step 2.1: Read the absolute addresses of all images into memory and put them into a list.

[0052] Step 2.2: Divide the list into a training set image address list and a validation set image address list according to the 8:2 ratio.

[0053] Step 2.3: Move the pictures represented by the training picture address to the train folder, and move the pictures represented by the verification set picture address list to the verify folder, complete the division of the data set.

[0054] Step 3: Build a double-flow detection network, and build a CFT (Cross-Modality Fusion Transformer) module as a fusion module of the two branches. Specifically, the following steps are included:

[0055] Step 3.1: Build an RGB branch of the double-flow detection network model, which is used for feature extraction of RGB images.

[0056] The RGB branch in the double-flow model is based on the FasterRCNN model. On this basis, the last five layers of maximum pooling and fully connected layers in Faster RCNN are removed. And the last four 14x14conv2d, 512 of the feature extraction layer are replaced with DConv2d (Dilated / Atrous Convolution). Specifically as shown in the following figure. The convolution kernel of all convolution layers in the figure is 3x3 with a step of 1 and padding of 0. The core size of the maximum pooling layer is 3x3. The numbers marked on each convolution layer in the figure, for example, 224x224x64, represent that the image size is 224x224, and the number of convolution kernels, that is, the output channel number, is 64. A Relu activation function is used after each convolution layer. A maximum pooling layer is used after every two convolution layers to increase the receptive field. Figure 2

[0057] Step 3.2: Build a Thermal branch of the double-flow detection network model, which is used for feature extraction of thermal images.

[0058] On the basis of the VGG model, the linear layer and the Dropout layer that play a classification role are deleted, and only the feature extraction layer in the model is used. And the step and padding of the convolution are modified, so that the image size does not change after passing through each convolution layer. Then the task of changing the image size is handed over to the pooling layer. The specific padding size can be calculated according to the following formula.

[0059] W out =(W in -F+2P) / S+1

[0060] H out =(H in -F+2P) / S+1

[0061] Where W out is the output image width, W in is the input image width. H in ​H is the height of the input image out H is the height of the output image, F is the size of the convolution kernel, S is the stride, and P is the padding size.

[0062] For example: suppose the size of the input image is W x H, the size of the convolution kernel is F x F, the size of the padding is P, and the stride is S. In order to keep the size of the output image the same as the input image, appropriate padding and stride can be selected to satisfy the following conditions:

[0063] (W-F+2P) / S+1=W out

[0064] (H–F+2P) / S+1=H out

[0065] By solving the equation set, we can get:

[0066] 2P=(F-1)×S

[0067] S=1

[0068] This means that when the size of the convolution kernel is equal to the stride, the size of the padding is (F-1) / 2, and the stride is 1, the image size can be kept unchanged. In practical applications, adjustments can be made according to specific circumstances. The convolution operation may produce floating-point results, in order to get integer results, usually need to round or truncate the output image.

[0069] Step 3.3: Build a CFT module to fuse the features of the RGB branch and the Thermal branch, and find the potential relationship between the two branches.

[0070] The CFT module is mainly composed of a layer normalization module (LayerNorm), a multi-head attention module (Multi-HeadAttention), and a multi-layer perception (MLP), as shown in Figure 4 . Among them, F Ri represents the RGB branch feature map, F Ti is the Thermal branch feature map, F Ri ' represents the RGB branch feature map after fusion by the CFT module, F Ti ' is the Thermal branch feature map after fusion by the CFT module, and the multi-head attention module is a stack of four multi-head attentions. The two feature maps are spliced and then added with position encoding, which can perform feature fusion.

[0071] After the above steps, the dual-flow detection model is built, as shown in Figure 5 . Among them, h i and ψ i represent the convolution modules of the RGB branch and the Thermal branch, P iRepresentative feature input. In this embodiment, the dual-stream detection network includes a plurality of CFT modules, the RGB branch and the Thermal branch are composed of the same number of stages, and the picture forms a respective feature map after feature extraction by a certain stage of the RGB branch and the Thermal branch. First, the feature maps are fused for the first time to obtain the most significant features, then the most significant features are fused for the second time through the CFT module, then the feature maps fused for the second time are added, and finally the feature maps fused for the second time are mapped to the two branches. After the feature maps of each branch are added, the next stage is entered, and the operation of the next stage is consistent with that of the previous stage.

[0072] Step 3.4: Design the loss function of the dual-stream detection model. Since the dual-stream model is a cross-modal fusion model, the loss function includes the boundary box loss (L box ), the classification loss (L cls ) and the confidence loss (L conf ) and the total loss L total is calculated according to the following formula (1).

[0073] L total = L box + L cls + L conf

[0074] = L box + L cls + L noobj + L obj

[0075] Wherein, L box uses the generalized IOU loss, represents whether the jth prediction box obj of the ith grid is a positive sample.

[0076] For the description of GIoU, please refer to Figure 6 :

[0077]

[0078] Wherein, L cls uses the cross-entropy loss function.

[0079] L cls = -∑y i *log(p i )

[0080] Wherein y represents the true value, and p represents the probability value predicted by the model.

[0081] L noobj , L obj use the mean square error:

[0082]

[0083] where y and represent the true and predicted values, respectively.

[0084] Step 4: Input the images of the public dataset into the dual-stream network for feature extraction and feature fusion, then input the extracted results into the MLP (Multilayer Perceptron) to obtain the classification results and prediction results. After multiple iterations, the best pre-training weight is obtained and placed in the test set of the public dataset to test the effect.

[0085] In this embodiment, first select the public datasets FLIRADAS and LLVIP, read the images and labels in the public dataset and input them into the built dual-stream network model. After multiple rounds of training and verification, the best training weight file is obtained. The weight file is used for test set to detect the generalization of the weight file.

[0086] Step 5: Process the training weight obtained in the public dataset to adapt to the self-built dataset. Then on the basis of the training weight, multiple iterations are performed to obtain the best training weight. Specifically, the following steps are included:

[0087] Step 5.1: Modify the total number of categories in the weight file obtained in step 4 to adapt to the total number of categories of the self-built dataset.

[0088] Step 5.2: Replace the original MLP layer in the weight file obtained in step 4 with a custom MLP layer that adapts to the self-built dataset.

[0089] Step 6: Use the best training weight to make predictions and visualize the prediction results. Including the following steps:

[0090] Step 6.1: Load the image and video data to be predicted into the model to obtain the prediction result tensor. At the same time, for video data, it can be decomposed into multiple frames of images and input into the model for prediction one by one to obtain the corresponding prediction results.

[0091] The prediction results include category information and the coordinates of the top-left corner and bottom-right corner of the bounding box. Using these two key coordinates, the bounding box can be drawn.

[0092] Step 6.2: Display the results using visualization tools.

[0093] The following table is a performance comparison table of different models on different datasets.

[0094]

[0095]

[0096] The performance indicators mAP50, mAP75 and mAP are referenced from MS-COCO, and represent the accuracy when the confidence is 50%, 75% and the average accuracy respectively. It can be seen from the above table that the double-stream model adopted by the present application has at least 3% improvement compared with the single-channel model.

[0097] The above embodiments are only for illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and implement it, and cannot limit the protection scope of the present application. Any equivalent transformation or modification made according to the spirit and essence of the present application shall be covered within the protection scope of the present application.

Claims

1. A personnel detection method based on a multi-spectral target detection cross-modal fusion model, characterized in that, Comprising the following steps: Step 1: Collect personnel images in various complex situations and classify, manually label, and make a self-built dataset; preprocess the original pictures of the self-built dataset, and divide the processed pictures into a training set and a validation set according to a certain proportion; Step 2: Build a dual-flow detection network, build a CFT module as a fusion module of the two branches and design a loss function; the dual-flow detection network uses a residual neural network ResNet as the backbone network, including an RGB branch and a Thermal branch, and the CFT module is used to fuse the features of the RGB branch and the Thermal branch and find the potential relationship between the two branches; The RGB branch is based on the Faster RCNN model, and the last five maximum pooling layers and fully connected layers in the Faster RCNN model are removed, and the last four 14x14conv2d,512 of the feature extraction layer are replaced with a DConv2d; The Thermal branch is based on the VGG model, and the linear layer and the Dropout layer that play a classification role are deleted, only the feature extraction layer in the model is used, and the convolution step and the padding are modified, so that the image size does not change after passing through each convolution layer, and then the task of changing the image size is handed over to the pooling layer, and the specific padding size is as follows: W out = (W in -F+2P) / S+1 H out = (H in -F + 2P) / S + 1 wherein W out is the output image width, W in is the input image width, H in is the input image height, H out is the output image height, F is the convolution kernel size, S is the stride, and P is the padding size. The loss function of the double-flow detection network is designed. Since the double-flow model is a cross-modal fusion model, the loss function includes a boundary box loss L box , a classification loss L cls and a confidence loss L conf . The total loss L total is calculated according to the following formula: L total = L box + L cls + L conf = L box + L cls + L noobj + L obj L cls = -∑ y i * log (p i ) wherein L box The generalized IOU loss is adopted, whether the jth prediction box (obj) representing the ith grid is a positive sample, L cls The cross-entropy loss function is adopted, y represents the true value, p represents the probability value predicted by the model, and L noobj , L obj The mean square error is adopted; Step 3: Input the pictures in the public dataset into the backbone network of the dual-flow detection network for hierarchical feature extraction, then input them into the dual-flow detection network for detailed feature extraction and feature fusion, input the extraction results into the MLP to obtain the classification results and the prediction results, and obtain the best pre-training weight through multiple iterations and test the effect in the test set of the public dataset; Step 4: Process the training weight obtained in the public dataset to adapt to the self-built dataset, and obtain the best training weight through multiple iterations based on the pre-training weight; Step 5: Use the trained weight to make predictions and visualize the prediction results.

2. The personnel detection method based on the cross-modal fusion model of multi-spectral target detection according to claim 1, characterized in that, In step 1, various complex scenes include rainy days, foggy days, and low-light conditions, and image preprocessing includes: enhancing image data through any one or several of flipping, translation, and cropping, and using a Gaussian filter to denoise the image to make it clearer and more readable. Finally, the picture is manually labeled using LabelImg, and the picture and label information are exported.

3. The method of claim 2, wherein the method further comprises: The step 1 further comprises the following steps: Step 1.1: Read all picture absolute addresses into memory and put them into a list; Step 1.2: Divide the list into a training set picture address list and a validation set picture address list according to the principle of 8:2; Step 1.3: Move the pictures represented by the training picture addresses to the train folder, and move the pictures represented by the validation set picture address list to the verify folder, and complete the division of the dataset.

4. The personnel detection method based on the cross-modal fusion model of multi-spectral target detection according to claim 1, characterized in that, The CFT module is composed of an inter-layer normalization module Layer Norm, a multi-head attention module Multi-HeadAttention, and a multi-layer perception MLP. The RGB branch feature map and the thermal branch feature map are spliced and then added with position encoding, and then sequentially pass through the inter-layer normalization module Layer Norm, the multi-head attention module Multi-HeadAttention, the inter-layer normalization module Layer Norm, and the multi-layer perception MLP for feature fusion to obtain the RGB branch feature map and the thermal branch feature map after the CFT module fusion. The multi-head attention module is a stack of four multi-head attentions.

5. The method of claim 1 or 4, wherein the method is a multi-spectral target detection based cross-modality fusion model for personnel detection. The double-flow detection network includes a plurality of CFT modules, and the RGB branch and the thermal branch are each composed of the same number of stages. After the picture passes through the feature extraction of a certain stage of the RGB branch and the thermal branch, the respective feature maps are formed. First, the feature maps are fused for the first time to obtain the most significant features. Then, the most significant features are fused for the second time through the CFT module. Subsequently, the feature maps after the second fusion are added. Finally, the feature maps after the second fusion are mapped to the two branches. After the addition operation of the feature maps of each branch, the next stage is entered. The operation of the next stage is consistent with that of the previous stage.

6. The method of claim 1, wherein the method is a multi-spectral target detection based cross-modal fusion model for personnel detection. The public dataset selects FLIR ADAS and LLVIP, reads the pictures and labels in the public dataset, and inputs the built double-flow detection network model.

7. The method of claim 1, wherein the method is a multi-spectral target detection based cross-modal fusion model for personnel detection. The specific operation in step 4 is as follows: Step 4.1: Modify the total number of categories in the weight file obtained in step 3 to adapt to the total number of categories of the self-built dataset. Step 4.2: Replace the original MLP layer in the weight file obtained in step 3 with a self-defined MLP layer that adapts to the self-built dataset.

Images