RGB-t multispectral pedestrian detection method based on target perception fusion strategy

Abstract

The application discloses an RGB-T (Thermal) multispectral pedestrian detection method based on a target perception fusion strategy, which aims to accurately determine the position of a pedestrian from a pair of RGB and thermal infrared images. Given a pair of RGB and thermal infrared images to be detected, the method first extracts multispectral features from the RGB and thermal infrared images; then uses a target perception fusion strategy to fuse the extracted multispectral features; finally, the fused features are sent to a detection head for pedestrian detection, and the pedestrian position frame and confidence are output. The application can effectively fuse the deep features of multispectral images, enhance the feature expression of the pedestrian area and suppress irrelevant background noise features during the fusion process, and can realize accurate pedestrian detection. In addition, the target perception fusion strategy is a feature fusion strategy that can be used in single-stage and double-stage detectors, which can effectively improve the accuracy of YOLO and Faster R-CNN detectors in the multispectral pedestrian detection task.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of RGB-T multispectral image pedestrian detection method, especially it is related to the pedestrian target position acquisition method based on multispectral image, applied to unmanned driving, road condition perception and intelligent monitoring etc. BACKGROUND

[0002] Pedestrian detection can be used to determine whether the input image or video frame contains pedestrians. In the field of intelligent transportation, pedestrian detection can be used to determine the pedestrian situation on the road, providing a reference for unmanned driving decision-making; In the field of intelligent security, pedestrian detection can also be used to reflect the personnel situation of the security scene, and prompt the possible risk of security personnel.

[0003] Current pedestrian detection methods are mainly based on RGB images, although they can perform excellent detection performance in scenes with good lighting conditions, but their performance in scenes with poor lighting conditions is not satisfactory, which is caused by the low signal-to-noise ratio of RGB images in low light conditions. Thermal infrared (Thermal) images are sensitive to human body heat radiation, and they are not affected by lighting conditions, and can provide clear human shape information in all-weather. However, thermal infrared images can only provide shape information and cannot provide color and texture information, which makes pedestrian detection methods based on thermal infrared images vulnerable to objects similar in structure to pedestrians. Therefore, multispectral pedestrian detection methods have emerged, which combine the advantages of RGB and thermal infrared images, with the goal of achieving all-weather pedestrian detection.

[0004] Multispectral pedestrian detection methods have received widespread attention from researchers due to their robust pedestrian detection performance. Multispectral Deep Neural Networks for Pedestrian Detection studies the influence of fusing multispectral features at different stages of neural networks on detection results. It designs three fusion methods. One is to directly splice RGB and thermal infrared images into four-channel images and input them into the neural network to output the detection results, this method is called early fusion; One is to input RGB and thermal infrared images into neural networks respectively, fuse the intermediate layer features of the neural networks, and complete detection based on the fused features, this method is called mid-fusion; One is to input RGB and thermal infrared images into two separate neural networks, and fuse their detection results, this method is called late fusion. This study found that the detection effect of mid-fusion is better than that of early fusion and late fusion. Based on this finding, subsequent studies mainly adopt the mid-fusion strategy.

[0005] Although fusing multispectral features can provide complementary information for single spectrum, simply concatenating multispectral features cannot significantly improve the detection performance. To solve this problem, recent research works proposed different feature fusion strategies. These fusion strategies can be divided into two categories according to whether segmentation branch is needed in feature fusion: with segmentation branch and without segmentation branch.

[0006] For feature fusion strategies without segmentation branch. Weakly Aligned Cross-modal Learning for Multispectral Pedestrian Detection focuses on solving the misalignment problem in multispectral data. It uses two separate neural networks to extract multispectral features and predict the displacement relationship between multispectral features to achieve weakly aligned multispectral pedestrian detection.

[0007] Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems is aware of the diversity of multispectral data and solves the problem of information imbalance in cross-modal data.

[0008] Spatio-contextual Deep Network-based Multimodal Pedestrian Detection for Autonomous Driving focuses on solving the problem of spatial and contextual information aggregation in the process of multispectral feature fusion. It uses graph attention network to fuse multispectral features and uses conditional random field to process the spatial information of fused features, and then uses channel attention mechanism and recurrent neural network to process the contextual information of fused features.

[0009] BAANet: Learning Bi-directional Adaptive Attention Gates for Multispectral Pedestrian Detection finds that it is difficult for RGB images to capture clear pedestrian information in low-light scenes, and it is difficult for thermal infrared images to distinguish pedestrians from other objects in scenes with high ambient temperature but good lighting because the thermal radiation of pedestrians is similar to that of other objects in the environment. It proposes a bidirectional attention gate mechanism and a scene lighting classification network that can adaptively use beneficial spectral information under different lighting conditions.

[0010] Learning a Dynamic Cross-modal Network for Multispectral Pedestrian Detection dynamically combines local and non-local information when fusing multispectral features to achieve better detection performance.

[0011] Multimodal Object Detection via Probabilistic Ensembling combines the detection results of multiple detectors using ensemble learning.

[0012] For the feature fusion strategy using segmentation branch during fusion. Multispectral Pedestrian Detection via Simultaneous Detection and Segmentation introduces a new segmentation branch for the original detector and uses a multi-task loss function to supervise the segmentation and detection network. This method achieves good detection performance.

[0013] Guided Attentive Feature Fusion for Multispectral Pedestrian Detection uses the segmentation branch to predict the segmentation mask, and guides the attention of cross-spectral and single-spectrum based on the mask.

[0014] Locality Guided Cross-modal Feature Aggregation and Pixel-level Fusion for Multispectral Pedestrian Detection uses the segmentation branch to predict the segmentation mask, and based on the mask, it selects local features from complementary spectral features to enhance the feature expression at specific locations.

[0015] Recent research has found that methods using segmentation branches during fusion are generally superior to methods not using segmentation branches. Therefore, the present invention uses a strategy that uses a segmentation branch during fusion. Unlike all the above methods, the present invention aims to fuse multispectral features and optimize the fused features to enhance the feature expression of the pedestrian region and suppress the feature expression of the background region, achieving more accurate multispectral pedestrian detection results. SUMMARY

[0016] In view of the problems in the prior art, the present invention proposes an RGB-T multispectral pedestrian detection method based on a target perception fusion strategy, and the overall process is as follows:

[0017] An RGB-T multispectral pedestrian detection method based on a target perception fusion strategy, the method being: given a pair of registered visible light (RGB) and thermal infrared (T) images, first extracting multispectral features, then fusing the extracted multispectral features in the feature space based on a target perception fusion strategy, and finally inputting the fused features into a detection head commonly used in the target detection field to output the position frame and confidence of the pedestrian, completing the detection process.

[0018] In the technical solution, further, the fusion of the extracted multispectral features in the feature space based on the target perception fusion strategy includes two steps:

[0019] S1: Multispectral feature aggregation. The multispectral features are processed using a feature channel hierarchical mechanism to output preliminary fused features. The multispectral features are processed in two ways using a feature channel hierarchical mechanism, and the specific steps are: one way is to use a convolution layer to compress the channel dimension of the multispectral features, and the other way is to use a convolution and residual module to compress the channel dimension of the multispectral features. Then the two processed features are spliced in the channel dimension and output preliminary fused features through a convolution layer; the advantage of multispectral feature aggregation is that it provides more gradient flow for spliced multispectral features, which can better optimize the multispectral features.

[0020] S2: Multispectral feature optimization. The preliminary fused features are optimized in two ways, and the specific steps are: one way is to use three convolution layers to process the preliminary fused features into a single-channel two-dimensional confidence map, then calculate the correlation vector between the confidence map and the preliminary fused channel feature map, and then use two fully connected layers to process the correlation vector. The other way is to multiply the preliminary fused features by the correlation vector processed by the fully connected layer to obtain optimized features and output; the advantage of multispectral feature optimization is that the weight of each channel feature map is adaptively adjusted according to the correlation between the feature map and the two-dimensional confidence map.

[0021] A pedestrian detection model for performing the above method, in the training process of the detection model, in order to ensure the accuracy of the model, a correlation maximum loss function is designed in the multispectral feature optimization process. The correlation maximum loss function includes two parts: 1) a segmentation loss function to supervise the consistency between the predicted two-dimensional confidence map and the true confidence map; 2) a maximum information entropy loss function to supervise the maximization of the correlation degree of each position in the correlation vector.

[0022] The training process of the detection model includes the following steps:

[0023] 1) Use an RGB camera and a thermal infrared light camera to collect registered multispectral images in an application scenario;

[0024] 2) using a convolutional neural network to extract deep features from the RGB image and the thermal infrared image, respectively, denoted as F rgb and F thermal , recording the multispectral features output by the convolutional neural network at different levels, and concatenating the extracted multispectral features in the channel direction to obtain features concatenated in the channel direction;

[0025] 3) using a target perception fusion module to fuse the features concatenated in the channel direction in the feature space, the specific steps being as follows:

[0026] 4) performing the following processing at each level:

[0027] First, the concatenated multispectral features are processed using a feature channel layering mechanism to output preliminary fused features F x :

[0028]

[0029] where [F rgb , F thermal ] represents the RGB and thermal infrared image features concatenated in the channel direction; represents the i-th convolutional layer in the multispectral feature aggregation module, θ i represents the learnable parameters of the i-th convolutional layer; represents a residual convolutional module.

[0030] Next, the preliminary fused features F x are optimized in two ways, and the optimized features F y are output:

[0031]

[0032] where,

[0033]

[0034] represents a pixel-by-pixel multiplication operation; s is a correlation vector; represents a multilayer perceptron composed of two fully connected layers; represents the correlation operation between the predicted two-dimensional confidence map m and the fused features F x in the i-th channel feature map; σ(·) represents a sigmoid operation; θ seg represents the learnable parameters of the segmentation branch.

[0035] According to the predicted two-dimensional confidence map m, the real two-dimensional confidence map and the correlation vector s, the correlation maximum loss function

[0036]

[0037] wherein the real two-dimensional confidence map The acquisition method of the true value is that the area in the label box corresponding to the given picture is set as 1, and other areas are set as 0. is a segmentation loss function; is a maximum information entropy loss function; alpha represents a balance coefficient, which is used to control the weight relationship between the two loss functions;

[0038]

[0039]

[0040] represents a binary cross-entropy loss function; represents a Dice loss function; epsilon is a small constant, which is used to prevent division by 0 and division by 0. i and represents a two-dimensional confidence map m and a real two-dimensional confidence map the value at the i-th position; s i represents the value of the correlation vector s at the i-th position.

[0041] 5) Repeat step 4) at different levels of the convolutional neural network, and record the features F output by the target perception fusion module at different levels y and loss value

[0042] 6) Send the features at different levels into a feature pyramid network to fuse multi-scale features, and record.

[0043] 7) Input the multi-scale features obtained in step 6) into the detection head of the detector to output the prediction box and the corresponding confidence score.

[0044] 8) Calculate the loss value between the prediction result and the real target box, and record.

[0045] 9) Weight the loss values calculated in steps 5) and 8).

[0046] 10) According to the loss value obtained in step 9), update the learnable parameters of the convolutional neural network using the stochastic gradient descent method.

[0047] 11) Repeat steps 1)-10) until the number of repetitions reaches the pre-set training number, complete the training process, and obtain the detection model.

[0048] The advantages of the present application are:

[0049] The RGB and thermal infrared image information are fully utilized to complement each other, the defects that the current detector is vulnerable to light conditions are made up, and robust all-weather pedestrian detection is realized; the multispectral feature fusion process is supervised, instead of only using the classification and regression loss of the final detection head to supervise the predicted frame; the feature output by the target-aware fusion strategy emphasizes the feature expression at the target position and weakens the feature expression on the irrelevant background. The feature map can more significantly distinguish the target object from the irrelevant background noise, and then help the detector to achieve better detection performance; the target-aware fusion strategy provided in the application is a convenient and universal multispectral feature fusion strategy, which can be used in Faster R-CNN and YOLO, and improves the detection performance of them in the multispectral pedestrian detection task. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 Definition of feature and channel feature map in neural network.

[0051] Figure 2 Flow chart of model training stage of the embodiment of the application.

[0052] Figure 3 Flow chart of pedestrian detection of the model of the embodiment of the application.

[0053] Figure 4 Flow chart of multispectral image feature aggregation module of the embodiment of the application.

[0054] Figure 5 Flow chart of multispectral feature optimization module of the model training stage of the embodiment of the application.

[0055] Figure 6 Flow chart of multispectral feature optimization module of the model inference stage of the embodiment of the application.

[0056] Figure 7 Original input image, real annotation and detection result of the embodiment of the application.(a) RGB-T input picture, the left picture is an RGB image, and the right picture is a thermal infrared image.(b) Real annotation and detection result. The green frame represents the real annotation, and the red frame represents the detection result. DETAILED DESCRIPTION

[0057] The technical scheme of the application is further illustrated below in combination with specific embodiments and drawings.

[0058] EMBODIMENT

[0059] The application provides an RGB-T multispectral pedestrian detection method based on a target-aware fusion strategy, and the steps are as follows:

[0060] Given a pair of registered visible light RGB and thermal infrared T images, first, multispectral features are extracted, then the extracted multispectral features are fused in the feature space based on a target perception fusion strategy, and finally the fused features are sent into a detection head to output the position frame and confidence of the pedestrian, completing the detection process.

[0061] The application also provides a pedestrian detection model for performing the above method, and the training process of the detection model and the specific process of pedestrian detection based on the model will be described below.

[0062] As Figure 1 , define the feature and channel feature map: after the input picture is sent into the neural network, the output result is defined as a feature, and its dimension is h x w x c, wherein h, w and c are the height, width and channel number of the feature respectively. Each h x w matrix in the channel direction is called a channel feature map.

[0063] A multispectral pedestrian detection model based on a target perception fusion strategy, the training process is as shown in Figure 2 .

[0064] 1) First, use an RGB camera and a thermal infrared light camera to collect registered multispectral (RGB and thermal infrared) images in an application scenario.

[0065] 2) Use a convolutional neural network to extract deep features from the RGB image and the thermal infrared light image respectively, represented as F rgb and F thermal , record the multispectral features output by the convolutional neural network at different levels, and splice the extracted multispectral features in the channel direction to obtain the spliced features in the channel direction.

[0066] 3) Use a target perception fusion module to fuse the spliced features in the feature space, and the specific steps are as follows:

[0067] 4) Process as follows at each level:

[0068] The target perception fusion module includes a multispectral feature aggregation module and a multispectral feature optimization module. The multispectral feature aggregation module uses a feature channel layering mechanism to process the spliced multispectral features to output preliminary fused features F x . The multispectral features extracted by the neural network are processed using a feature channel layering mechanism, and the multispectral features are processed in two ways: one way uses a convolutional layer to compress the channel dimension of the multispectral features, and the other way uses a convolutional layer and a residual module to compress the channel dimension of the multispectral features. Then the two processed features are spliced in the channel dimension and output preliminary fused features through a convolutional layer:

[0069]

[0070] where [F rgb , F thermal ] represents the RGB and thermal infrared image features spliced in the channel direction; represents the i-th convolutional layer in the multispectral feature aggregation module, θ i represents the learnable parameters of the i-th convolutional layer; represents a residual convolutional module. The process is shown in Figure 4 .

[0071] The multispectral feature optimization module optimizes the preliminarily fused feature F x in two ways and outputs the optimized feature F y . The specific steps are as follows: one way is to use three layers of convolutional layers to process the preliminarily fused feature into a single-channel two-dimensional confidence map, then calculate the correlation vector between the confidence map and the preliminarily fused channel feature map, and then use two fully connected layers to process the correlation vector. The other way is to multiply the preliminarily fused feature and the correlation vector processed by the fully connected layers to obtain the optimized feature and output:

[0072]

[0073] where,

[0074]

[0075]

[0076]

[0077] represents a pixel-by-pixel multiplication operation; s is the correlation vector: represents a multilayer perceptron composed of two fully connected layers; represents the correlation operation between the predicted two-dimensional confidence map m and the fused feature F x i-th channel feature map; σ(·) represents the sigmoid operation; θ seg represents the learnable parameters of the segmentation branch.

[0078] According to the predicted two-dimensional confidence map m, the real two-dimensional confidence map and the correlation vector s, the correlation maximum loss function

[0079]

[0080] where the real two-dimensional confidence map The true value is obtained in the following way: the area within the annotation box corresponding to the given picture is set to 1, and the other areas are set to 0; is the segmentation loss function; To maximize the information entropy loss function; a represents the balance coefficient, which is used to control the weight relationship between the two loss functions.

[0081]

[0082]

[0083] represents the binary cross-entropy loss function; represents the Dice loss function; ε is a small constant to prevent division by 0 and division by 0. m i and represents the two-dimensional confidence map m and the real two-dimensional confidence map the value at the i-th position; s i represents the value of the correlation vector s at the i-th position. The optimization process is as shown. Figure 5

[0084] 5) Repeat step 4) at different levels of the convolutional neural network, and record the features F output by the target perception fusion module at different levels y and loss values

[0085] 6) Send the features at different levels into the feature pyramid network to fuse multi-scale features, and record.

[0086] 7) Input the multi-scale features obtained in step 6) into the detection head of the detector, output the predicted frame and the corresponding confidence score.

[0087] 8) Calculate the loss value between the predicted result and the real target frame, and record.

[0088] 9) Weight the loss values calculated in steps 5) and 8).

[0089] 10) According to the loss value calculated in step 9), update the learnable parameters of the convolutional neural network using the stochastic gradient descent method.

[0090] 11) Repeat steps 1)-10) until the number of repetitions reaches the pre-set training number, and complete the training process.

[0091] As Figure 3 shows the specific process of performing pedestrian detection based on the above model, the steps are as follows:

[0092] 1) First, use the RGB camera and the thermal infrared light camera to collect the registered multi-spectral (RGB and thermal infrared) images in the application scene.

[0093] ​2) Use a convolutional neural network to extract deep features from RGB images and thermal infrared images respectively, denoted as F rgb and F thermal , record the multispectral features output by the convolutional neural network at different levels, and splice the extracted multispectral features in the channel direction.

[0094] 3) Use a target perception fusion module to fuse the extracted multispectral features in the feature space, the specific steps are as follows:

[0095] 4) The following processing is performed at each level:

[0096] The multispectral feature aggregation module uses a feature channel hierarchical mechanism to process the spliced multispectral features and outputs the preliminary fused features F x :

[0097]

[0098] Where [F rgb , F thermal ] represents the RGB and thermal infrared image features spliced in the channel direction; represents the i-th convolutional layer in the multispectral feature aggregation module, θ i represents the learnable parameters of the i-th convolutional layer; represents a residual convolutional layer, the process is shown in Figure 4 .

[0099] The multispectral feature optimization module optimizes the preliminary fused features F x in two ways, and outputs the optimized features F y :

[0100]

[0101] Where,

[0102]

[0103]

[0104]

[0105] represents a pixel-by-pixel multiplication operation; s is a correlation vector; represents a multilayer perceptron composed of two fully connected layers; represents the correlation operation of the predicted two-dimensional confidence map m and the fused feature F x i-th channel feature map; σ(·) represents the sigmoid operation, the process is shown in Figure 6 .

[0106] 5) Repeat step 4) at different layers of the convolutional neural network, and record the feature F output by the target perception fusion module at different layers. y .

[0107] 6) Feed features from different levels into the feature pyramid network to fuse multi-scale features and record them;

[0108] 7) Input the output multi-scale features into the detector head, output the predicted bounding box and the corresponding confidence score, and complete the detection process.

[0109] like Figure 7 The results of pedestrian detection based on the method of this invention are provided by Figure 7 It can be seen that the detection results of the method of the present invention are completely consistent with the actual annotations, and the method can realize all-weather pedestrian detection.

Claims

1. A method for pedestrian detection based on an RGB-T multispectral fusion strategy, characterized in that, The steps are as follows: Given a pair of registered visible light RGB and thermal infrared T images, multispectral features are first extracted. Then, based on a target perception fusion strategy, the extracted multispectral features are fused in the feature space. Finally, the fused features are fed into the detection head to output the pedestrian's bounding box and confidence score, thus completing the detection process. The target-aware fusion strategy fuses the extracted multispectral features in the feature space, including two steps: S1: Multispectral Feature Aggregation: The multispectral features are processed using a feature channel hierarchical mechanism to output preliminarily fused features; S2: Multispectral feature optimization is divided into two paths to optimize the initially fused features. Specifically, one path uses three convolutional layers to process the initially fused features into a single-channel two-dimensional confidence map, then calculates the correlation vector between the two-dimensional confidence map and the initially fused channel feature map, and then uses two fully connected layers to process the correlation vector; the other path multiplies the initially fused features with the correlation vector processed by the fully connected layers to obtain the optimized features and output them.

2. The RGB-T multispectral pedestrian detection method based on a target perception fusion strategy according to claim 1, characterized in that, In step S1, the feature channel layering mechanism processes the multispectral features in two ways. Specifically, one way uses a convolutional layer to compress the channel dimension of the multispectral features, and the other way uses a convolutional and residual module to compress the channel dimension of the multispectral features. Then, the features processed by the two ways are concatenated in the channel dimension and output as a preliminary fused feature after passing through a convolutional layer.

3. A detection model for performing the method of claim 1 or 2, characterized in that, The training process of the detection model includes the following steps: 1) Use an RGB camera and a thermal infrared camera to acquire multispectral images for registration in the application scenario; 2) Use a convolutional neural network to extract deep features from the RGB image and the thermal infrared image respectively, as shown below: and Record the multispectral features output by the convolutional neural network at different levels, and then concatenate the extracted multispectral features in the channel direction to obtain the features concatenated in the channel direction. 3) The target-aware fusion module is used to fuse the features stitched together in the channel direction in the feature space. The specific steps are as follows: 4) Perform the following processing at each level: First, a feature channel hierarchical mechanism is used to process the spliced ​​multispectral feature outputs for preliminary feature fusion. : ； in, This represents the RGB and thermal infrared image features stitched together along the channel direction; This represents the i-th convolutional layer in the multispectral feature aggregation module. This represents the learnable parameters of the i-th convolutional layer; This represents a residual convolutional module; Next, the features of the initial fusion were optimized in two ways. and output the optimized features. : ； in, ； This indicates a pixel-by-pixel multiplication operation; This is a correlation vector; This represents the sigmoid operation; This represents a multilayer perceptron consisting of two fully connected layers. Two-dimensional confidence plot representing the prediction Features of integration Correlation calculation of the feature map of the i-th channel; The learnable parameters representing the split branches; Based on the predicted two-dimensional confidence plot True two-dimensional confidence plot and correlation vector Calculate the correlation maximum loss function ； Among them, the true two-dimensional confidence graph The truth value is obtained by setting all regions within the bounding box corresponding to the given image to 1, and other regions to 0. For the segmentation loss function; To maximize the information entropy loss function; This represents the balance coefficient, which is used to control the weighting relationship between the two loss functions; ； ； Represents the binary cross-entropy loss function; Dice loss function It is a small constant to prevent division by zero and division by zero cases; and Represents a two-dimensional confidence plot And the true two-dimensional confidence graph The value at the i-th position; Represents the correlation vector The value at the i-th position; 5) Repeat step 4) at different layers of the convolutional neural network and record the features output by the target perception fusion module at different layers. and loss value ; 6) Feed features from different levels into the feature pyramid network to fuse multi-scale features and record them; 7) Input the multi-scale features obtained in step 6) into the detector head, and output the predicted bounding box and the corresponding confidence score; 8) Calculate and record the loss between the predicted result and the true target box; 9) The loss value calculated in weighted steps 5) and 8); 10) Based on the loss value obtained in step 9), update the learnable parameters of the convolutional neural network using stochastic gradient descent. 11) Repeat steps 1)-10) until the number of repetitions reaches the preset number of training iterations to complete the training process and obtain the detection model.

Images