A multispectral pedestrian detection method based on multi-stage feature fusion information reuse

Abstract

The present invention discloses a multi-spectral pedestrian detection method based on multi-stage feature fusion information reuse, which belongs to the technical field of multimodal information processing. The method comprises: extracting intermediate layer features from paired infrared and visible light images through a dual-stream VGG16 network to perform early feature stacking fusion, obtaining early fusion features, and generating pedestrian suggestions based on the fusion features. Each generated pedestrian suggestion is first mapped back to three multimodal features for multi-feature pooling, and then high-level pooling features are weighted fused. At the same time, the pooling features are combined using high- and low-level feature pooling strategies to complete the union of high- and low-level features. Finally, the pooling features are sent to the fully connected layer to complete the detection task. Experimental results show that on the KAIST multi-spectral pedestrian detection dataset, a pedestrian detection accuracy of 76.24% is obtained, and the average pedestrian missed detection rate is reduced to 27.63%.

Description

Technical Field

[0001] The invention relates to a multi-spectral pedestrian detection method based on multi-stage feature fusion information multiplexing, and belongs to the technical field of multi-modal information processing. Background Art

[0002] With the rapid rise of artificial intelligence, driverless technology and intelligent video surveillance have become important components of the smart city field. Driverless technology can integrate pedestrian detection technology to sense the external environment and automatically avoid pedestrians, thereby ensuring the safety of pedestrians. Intelligent video surveillance can also use pedestrian detection technology to obtain the location of pedestrian targets in the scene, and achieve functions such as preventing illegal and criminal acts and preventing accidents. Therefore, pedestrian detection technology has a positive significance for regulating traffic and maintaining public order, and is one of the key technologies for building smart cities.

[0003] Unmanned driving technology and intelligent video surveillance are all-weather applications. However, traditional pedestrian detection that only uses visible light as the input source is difficult to meet the requirements of all-weather applications because visible light images have poor performance under poor lighting conditions and inclement weather such as smoke, fog, and dust. Pedestrian targets are easily lost at night and in bad weather. Infrared image technology is based on the principle of infrared thermal imaging. It can distinguish pedestrian targets from backgrounds by the difference in thermal radiation. Therefore, it has the advantages of strong anti-interference and is not easily affected by bad environments. It can also better display human figures at night, effectively making up for the defect that visible light images are easily affected by light. However, compared with visible light images, infrared images have lower resolution and lower texture clarity. Under good lighting conditions, infrared images can provide less visual information than visible light images. Therefore, the advantages and disadvantages of visible light images and infrared images make them complementary in all-weather conditions. Therefore, multispectral pedestrian detection technology that uses infrared and visible light images as input sources can effectively overcome the impact of environmental factors such as light and bad weather on unmanned driving and intelligent video surveillance.

[0004] However, the existing pedestrian detection methods that fuse visible light images and infrared images only select a certain layer of feature fusion, such as Figure 1A The low-level feature fusion shown, Figure 1B The intermediate layer feature fusion shown, Figure 1C These fusion methods limit the input of the fusion mechanism to the features of the selected layer, but ignore the fact that the remaining layers can still provide effective information for the pedestrian detection task, and only use the fused features to complete the subsequent detection task, which to some extent ignores the local information on the single modality, resulting in poor detection performance. Summary of the invention

[0005] In order to solve the problem that the detection performance of the existing pedestrian detection method that fuses visible light images and infrared images is heavily dependent on the quality of the fusion features, and once the fusion mechanism performance is unstable, the detection result will also be greatly affected, the present invention provides a multispectral pedestrian detection method based on multi-stage feature fusion information multiplexing, the method comprising:

[0006] The paired infrared and visible light images are respectively extracted from the intermediate layer features through the two-stream VGG16 network for early feature stacking and fusion to obtain early fusion features, and pedestrian suggestions are generated based on the early fusion features; each generated pedestrian suggestion is first mapped back to the three multimodal features for multi-feature pooling to obtain high-level pooling features, and then high-level pooling features are weighted fused, and high- and low-level feature pooling strategies are used to combine high- and low-level pooling features to complete the combination of high- and low-level features and obtain high- and low-level pooling joint features; finally, the high- and low-level pooling joint features are sent to the fully connected layer to obtain the classification score and positioning box of the pedestrian target;

[0007] The pedestrian suggestion refers to the preliminary positioning information of the pedestrian; each pair of infrared and visible light images is an infrared image and a visible light image taken at the same time and the same scene and after being registered.

[0008] Optionally, the method comprises:

[0009] S1, obtaining pairs of infrared images and visible light images; each pair of images is an infrared image and a visible light image taken at the same time and in the same scene and registered;

[0010] S2, extract low-level features, middle-level features, and high-level features from each pair of infrared images and visible light images through a two-stream VGG16 network;

[0011] S3, perform feature fusion on the extracted infrared low-layer feature Conv3-T and the visible low-layer feature Conv3-V to obtain the low-layer fusion feature Conv3-F, and perform a region of interest pooling operation on the low-layer fusion feature Conv3-F to obtain the low-layer pooling feature Pool-conv3; perform early feature stacking fusion on the extracted infrared intermediate layer feature Conv4-T and the visible intermediate layer feature Conv4-V to obtain the early fusion feature Conv4-F, and perform a convolution operation on the early fusion feature Conv4-F to obtain the third modality high-level feature Conv5-F, and perform a region of interest pooling operation on the third modality high-level feature Conv5-F to obtain the third modality high-level pooling feature Pool-F; perform region of interest pooling operations on the extracted infrared high-level feature Conv5-T and the visible high-level feature Conv5-V to obtain the infrared high-level pooling feature Pool-T and the visible high-level pooling feature Pool-V respectively;

[0012] S4, performing a post-pooling feature weighted fusion operation on the obtained infrared high-level pooling feature Pool-T and the visible high-level pooling feature Pool-V to obtain a post-weighted fusion pooling feature Pool-W, and performing a cascade operation on the post-weighted fusion pooling feature Pool-W, the third modality high-level pooling feature Pool-F and the low-level pooling feature Pool-conv3 to obtain a high- and low-level pooling joint feature, and finally sending the high- and low-level pooling joint feature to the fully connected layer to obtain the classification score and positioning box of the pedestrian target.

[0013] The present application also provides a multi-spectral pedestrian detection network framework based on multi-stage feature fusion information reuse, the network framework includes: a feature extraction module, an intermediate layer feature fusion module, a region proposal generation module, a regional attention module, a late pooling feature fusion module, a high-low layer feature cascade module and a fully connected layer; wherein, the infrared intermediate layer feature Conv4-T and the visible intermediate layer feature Conv4-V are extracted by the feature extraction module and sent to the intermediate layer feature fusion module to obtain the early fusion feature Conv4-F, and then the early fusion feature Conv4-F is convolved to obtain the third modal high-level feature Conv5-F, which is input into the region proposal generation module to obtain the pedestrian suggestion, and at the same time, the infrared high-level feature Conv5-T and the visible high-level feature Conv5-V extracted by the feature extraction module are connected to the region attention module to obtain the pedestrian suggestion. Optimize the features, and then map the pedestrian proposal back to the low-level fusion features Conv3-F, infrared high-level features Conv5-T, visible high-level features Conv5-V extracted by the feature extraction module, and the features Conv5-F obtained by the intermediate layer feature fusion module, and pool them to obtain the low-level pooling features Pool-conv3, infrared high-level pooling features Pool-T, visible high-level pooling features Pool-V, and third-modality high-level pooling features Pool-F, and then input Pool-T and Pool-V into the late pooling feature fusion module to obtain the late weighted fusion pooling features Pool-W, and then input Pool-W, Pool-conv3, and Pool-F into the high- and low-level feature cascade module to obtain the high- and low-level pooling joint features, and input them into the fully connected layer to obtain the classification score and positioning box of the pedestrian target.

[0014] Optionally, the feature extraction module and the fully connected layer use a VGG16 network pre-trained on ImageNet as a basic network.

[0015] Optionally, the network framework is trained and tested using the following steps to obtain a network framework with optimal parameters:

[0016] Step 1, construct training set Train-02 and test set Test-20;

[0017] Step 2, training the network framework using the image training set Train-02;

[0018] Step 2.1, sending the paired infrared and visible light images to the feature extraction module in turn to extract features until the infrared high-level features Conv5-T and the visible high-level features Conv5-V are generated respectively; the feature extraction module includes two networks with exactly the same network structure, which are used to extract low-level, middle-level and high-level features of the visible light stream SV and the infrared stream ST respectively;

[0019] Step 2.2, select the last layer features of the fourth convolution from the visible light stream SV and the infrared stream ST with feature sizes of C×H×W: Conv4-T and Conv4-V are input into the intermediate layer feature fusion module to complete the early feature fusion operation;

[0020] (1) Perform feature cascading in the channel dimension to obtain stacked features of size 2C×H×W;

[0021] (2) Use 1*1 convolution to reduce the dimension and exchange information of the stacked features to obtain the early fusion feature Conv4-F with a size of C×H×W;

[0022] Step 2.3: Send the early fusion feature Conv4-F into the fusion stream SF to generate the third modality high-level feature Conv5-F, where the fusion stream is the fifth convolution operation of the VGG16 network, including a maximum pooling layer, three groups of 3*3 convolution layers and ReLU activation layers;

[0023] Step 2.4: Connect the region proposal generation module after the third modality high-level feature Conv5-F to generate pedestrian proposals; connect the region attention module after the infrared high-level feature Conv5-T and the visible high-level feature Conv5-V;

[0024] Step 2.5: Use the region of interest to map each generated pedestrian proposal back to Conv5-T, Conv5-V, Conv5-F and pool them to obtain pooled features Pool-T, Pool-V, Pool-F of size 7×7;

[0025] Step 2.6: Input Pool-T and Pool-V into the late pooling feature fusion module for late pooling feature fusion:

[0026] (1): Pool-T and Pool-V are stacked, and the stacked features are compressed using global average pooling to obtain global information;

[0027] (2) The features obtained in the previous step are compressed and stacked using three fully connected layers to model the modality correlation and separate two features at the same time.

[0028] (3): Use the Sigmoid activation function to obtain the weights w1 and w2 of the two modes for the two features obtained in the previous step. Multiply w1 and w2 with Pool-T and Pool-V respectively to obtain the corresponding two weighted features.

[0029] (4): The two weighted features obtained in the previous step are concatenated and dimensionally reduced to obtain the late fusion feature Pool-W;

[0030] Step 2.7: Select the infrared low-level feature Conv3-T and visible low-level feature Conv3-V from ST and SV in step 2.1, cascade Conv3-T and Conv3-V and reduce the dimension to obtain the low-level fusion feature Conv3-F, obtain the low-level pooling feature Pool-conv3 on Conv3-F through step 2.5, use the high-low-level feature cascade module to perform feature cascade and dimension reduction on Pool-conv3, Pool-W, Pool-T and Pool-V to obtain the features used for detection;

[0031] Step 2.8: Send the features used for detection obtained in step 2.7 to the fully connected layer to obtain the classification score and positioning box of the pedestrian target;

[0032] Step 2.9: Repeat steps 2.1 to 2.8 above and iterate the training until the network framework with the optimal parameters is obtained;

[0033] Step (4): Test the network framework under the optimal parameters.

[0034] Optionally, during the training process of the network framework, step 1 includes:

[0035] Step 1.1, sample the training part video sequences of infrared and visible light videos in the multispectral pedestrian detection dataset every 2 frames to obtain the paired infrared and visible light image training set Train-02. Each pair of images with the same sequence number in the image training set Train-02 is an infrared image and a visible light image taken at the same time and in the same scene and after registration;

[0036] The test part of the video sequence in the multispectral pedestrian detection dataset is sampled every 20 frames to obtain the paired infrared and visible light image test set Test-20. Each pair of images with the same sequence number in the image test set Test-20 is an infrared image and a visible light image taken at the same time and in the same scene and after registration.

[0037] Step 1.2: Flip all images in the image training set Train-02 to expand the training data.

[0038] Optionally, the region proposal generation module in the network framework adopts the region proposal generation network RPN of the Faster RCNN detection network, and uses two 1*1 convolutions to respectively predict the position and classification score of the pedestrian.

[0039] Optionally, the loss function of the RPN is the classification loss L cls and position loss L regs The weighted sum of:

[0040]

[0041] Where α is the weight set, which is set to 1.

[0042] Optionally, the classification loss L cls For: L cls (p i ,g i )=-log[g i p i +(1-g i )(1-p i )], is the cross entropy loss, that is, the logarithmic loss is calculated for each sample and then divided by the total number of samples N; where p i is the probability that the i-th prediction box is predicted as a pedestrian target, g i = {1,0} represents the label values ​​of foreground and background respectively; in the region proposal generation network, N regs Indicates the number of selected negative samples, set to 256;

[0043] Position loss L regs for Among them, function R is the Smooth L1 loss expression x represents

[0044]

[0045] t i ={t x ,t y ,t w ,t h}, represents the offset of the prediction of the i-th sample, With t i The dimensions are the same, indicating the actual offset of the i-th sample relative to the true value; x, y, w, h represent the center point coordinates, width, and height of the detection box.

[0046] The beneficial effects of the present invention are:

[0047] By adding a regional attention network after the two unimodal features of visible light and infrared light respectively, the features of the corresponding modalities are guided to pay more attention to the pedestrian part, thereby enriching the features of the two unimodal modes and improving the pedestrian detection performance. Furthermore, by adding the late pooling feature fusion on the basis of the early feature fusion, the errors and valid information in the early fusion features are corrected and supplemented, which solves the problem of existing detection methods that only a single feature layer is selected for the feature fusion of infrared and visible light, resulting in waste of feature information and causing pedestrian missed detection and false detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1A This is a simplified flow chart of the existing multi-spectral pedestrian detection method based on underlying feature fusion;

[0050] Figure 1B This is a simplified flow chart of the existing multi-spectral pedestrian detection method based on intermediate layer feature fusion;

[0051] Figure 1C This is a simplified flow chart of the existing multi-spectral pedestrian detection method based on high-level feature fusion;

[0052] Figure 2 A network block diagram corresponding to a multispectral pedestrian detection method based on multi-stage feature fusion information multiplexing provided in one embodiment of the present application.

[0053] Figure 3 A schematic structural diagram of a late-stage pooling feature fusion module in a network block diagram corresponding to a multi-spectral pedestrian detection method based on multi-stage feature fusion information reuse provided in an embodiment of the present application.

[0054] Figure 4 This is a simulation diagram of the detection results of the single-stage fusion strategy in the existing method on the visible light images in the KAIST dataset.

[0055] Figure 5 This is a simulation diagram of the detection results of the single-stage fusion strategy in the existing method on the infrared image of the KAIST dataset.

[0056] Figure 6 This is a simulation diagram of the detection results of the multi-spectral pedestrian detection method based on multi-stage feature fusion information multiplexing provided in one embodiment of the present application on the visible light image in the KAIST dataset.

[0057] Figure 7 This is a simulation diagram of the detection results of the multi-spectral pedestrian detection method based on multi-stage feature fusion information multiplexing provided in one embodiment of the present application on the infrared image of the KAIST dataset. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0059] As shown in Figure 1, some current multispectral pedestrian detection algorithms usually only select a certain layer of feature fusion in the multispectral information fusion stage, such as Figure 1A The low-level feature fusion shown, Figure 1B The intermediate layer feature fusion shown, Figure 1C The high-level feature fusion shown. These fusion methods limit the input of the fusion mechanism to the features of the selected layer, but ignore the fact that the remaining layers can still provide effective information for the pedestrian detection task, and only use the fused features to complete subsequent detection tasks, which ignores the local information on the single modality to a certain extent. Especially the two-stage target detector, which is equivalent to using only the early fused features in the classification stage and not using the features of the single modality. The quality of the detection performance will be more dependent on the quality of the fused features. Once the performance of the fusion mechanism is unstable, the detection results will also be greatly affected. Therefore, the present invention provides a multi-spectral pedestrian detection technology based on multi-stage feature fusion information reuse, which is specifically introduced as follows:

[0060] Embodiment 1:

[0061] This embodiment provides a multispectral pedestrian detection method based on multi-stage feature fusion information multiplexing, the method comprising:

[0062] The paired infrared and visible light images are respectively extracted from the intermediate layer features through the two-stream VGG16 network for early feature stacking and fusion to obtain early fusion features, and pedestrian suggestions are generated based on the early fusion features; each generated pedestrian suggestion is first mapped back to the three multimodal features for multi-feature pooling to obtain high-level pooling features, and then high-level pooling features are weighted fused, and high- and low-level feature pooling strategies are used to combine high- and low-level pooling features to complete the combination of high- and low-level features and obtain high- and low-level pooling joint features; finally, the high- and low-level pooling joint features are sent to the fully connected layer to obtain the classification score and positioning box of the pedestrian target;

[0063] The pedestrian suggestion refers to the preliminary positioning information of the pedestrian; each pair of infrared and visible light images is an infrared image and a visible light image taken at the same time and the same scene and after being registered.

[0064] Specifically, the method includes:

[0065] S1, obtaining pairs of infrared images and visible light images; each pair of images is an infrared image and a visible light image taken at the same time and in the same scene and registered;

[0066] S2, extract low-level features, middle-level features, and high-level features from each pair of infrared images and visible light images through a two-stream VGG16 network;

[0067] S3, perform feature fusion on the extracted infrared low-layer feature Conv3-T and the visible low-layer feature Conv3-V to obtain the low-layer fusion feature Conv3-F, and perform a region of interest pooling operation on the low-layer fusion feature Conv3-F to obtain the low-layer pooling feature Pool-conv3; perform early feature stacking fusion on the extracted infrared intermediate layer feature Conv4-T and the visible intermediate layer feature Conv4-V to obtain the early fusion feature Conv4-F, and perform a convolution operation on the early fusion feature Conv4-F to obtain the third modality high-level feature Conv5-F, and perform a region of interest pooling operation on the third modality high-level feature Conv5-F to obtain the third modality high-level pooling feature Pool-F; perform region of interest pooling operations on the extracted infrared high-level feature Conv5-T and the visible high-level feature Conv5-V to obtain the infrared high-level pooling feature Pool-T and the visible high-level pooling feature Pool-V respectively;

[0068] S4, performing a post-pooling feature weighted fusion operation on the obtained infrared high-level pooling feature Pool-T and the visible high-level pooling feature Pool-V to obtain a post-weighted fusion pooling feature Pool-W, and performing a cascade operation on the post-weighted fusion pooling feature Pool-W, the third modality high-level pooling feature Pool-F and the low-level pooling feature Pool-conv3 to obtain a high- and low-level pooling joint feature, and finally sending the high- and low-level pooling joint feature to the fully connected layer to obtain the classification score and positioning box of the pedestrian target.

[0069] Embodiment 2:

[0070] This embodiment provides a multi-spectral pedestrian detection method based on multi-stage feature fusion information reuse. Figure 2The method includes establishing a multispectral pedestrian detection network framework based on multi-stage feature fusion information reuse, and using a public data set to train it to obtain a network framework under optimal parameters, and then using the trained network framework to achieve pedestrian detection, the network framework includes: a feature extraction module, an intermediate layer feature fusion module, a region proposal generation module, a region attention module, a late pooling feature fusion module, a high-low layer feature cascade module and a fully connected layer; wherein the infrared intermediate layer feature Conv4-T and the visible intermediate layer feature Conv4-V are extracted by the feature extraction module and sent to the intermediate layer feature fusion module to obtain the early fusion feature Conv4-F, and then the early fusion feature Conv4-F is convolved to obtain the third modality high-level feature Conv5-F, which is input into the region proposal generation module to obtain pedestrian suggestions, and at the same time, the infrared high-level features Conv5-T and Conv4-V extracted by the feature extraction module are extracted by the feature extraction module. The visible high-level feature Conv5-V is then connected to the regional attention module to optimize the features, and then the pedestrian proposal is mapped back to the low-level fusion feature Conv3-F extracted by the feature extraction module, the infrared high-level feature Conv5-T, the visible high-level feature Conv5-V, and the feature Conv5-F obtained by the intermediate layer feature fusion module, and they are pooled to obtain the low-level pooling feature Pool-conv3, the infrared high-level pooling feature Pool-T, the visible high-level pooling feature Pool-V, and the third modality high-level pooling feature Pool-F, respectively. Then Pool-T and Pool-V are input into the late pooling feature fusion module to obtain the late weighted fusion pooling feature Pool-W, and then Pool-W, Pool-conv3, and Pool-F are input into the high- and low-level feature cascade module to obtain the high- and low-level pooled joint features, and input into the fully connected layer to obtain the classification score and positioning box of the pedestrian target.

[0071] The training process of the network framework includes:

[0072] Step (1): Preprocess the dataset;

[0073] Specifically, they include:

[0074] Step 1.1: Sample the training video sequences of infrared and visible light videos in the KAIST multispectral pedestrian detection dataset every 2 frames to obtain the paired infrared and visible light image training set Train-02.

[0075] The test part of the video sequence in the multispectral pedestrian detection dataset is sampled every 20 frames to obtain the paired infrared and visible light image test set Test-20.

[0076] Ensure that the two images with the same serial number in the two training sets (test sets) are infrared and visible light images taken at the same time and in the same scene and registered; Step 1.2: Flip all images in Train-02 to expand the training data;

[0077] This application uses a public dataset to train the established multispectral pedestrian detection network framework based on multi-stage feature fusion information reuse. The KAIST multispectral pedestrian detection dataset is a pedestrian detection dataset produced by KAIST University in South Korea in 2015. It provides infrared video sequences and visible light video sequences in conventional traffic scenes, and performs strict image registration through methods such as camera calibration. It is also the only large-scale dataset that provides paired aligned visible light and infrared images so far.

[0078] The KAIST dataset can be found in Hwang Soonmin's "Multispectral pedestrian detection: Benchmark dataset and baseline", which was published in IEEE Conference on Computer Vision and Pattern Recognition in 2015, pages 1037-1045.

[0079] Step (2): Train the network model with a learning rate of 0.001, an impulse of 0.9, a weight decay term of 0.0005, and a batch size of 2. The specific operations are as follows:

[0080] Step 2.1: See Figure 2 , the paired infrared and visible light images and the corresponding pedestrian annotations in the training set processed in step (1) are sequentially sent to the feature extraction module to extract features until high-level features Conv5-T and Conv5-V are generated.

[0081] The feature extraction module includes two networks with exactly the same structure, namely Stream-Visible (SV) and Stream-Thermal (ST), both of which are based on the VGG16 network without the fully connected layer, and include five convolution blocks Conv1-Conv5. Each convolution block consists of a 3*3 convolution layer, a ReLU activation layer, and a maximum pooling layer. The two networks do not share parameters.

[0082] Step 2.2: See Figure 2 , select the last layer of features from the fourth convolution of ST and SV with feature sizes of C×H×W: Conv4-T and Conv4-V are input into the intermediate layer feature fusion module to complete the early feature fusion operation:

[0083] (1) Perform feature cascading in the channel dimension to obtain stacked features of size 2C×H×W;

[0084] (2) Use 1*1 convolution to reduce the dimension and exchange information of the stacked features to obtain the fused feature Conv4-F with a size of C×H×W;

[0085] Step 2.3: See Figure 2 , Conv4-F is sent to the fusion stream (Stream-Fusion, SF) to generate high-level features Conv5-F, where the fusion stream is the fifth convolution operation of the VGG16 network;

[0086] Step 2.4: See Figure 2 , after Conv5-F, the Region Proposal Network (RPN) is connected to generate pedestrian proposals (Proposals). After Conv5-T and Conv5-V, the Region Attention Network (RAN) is connected to optimize the performance of each single-modal branch. The region proposal generation module uses the region proposal generation network of the Faster RCNN detection network, and uses two 1*1 convolutions to predict the location and classification score of pedestrians respectively. RAN has the same structure as the RPN convolution part, but RPN finally outputs pedestrian proposals through a pedestrian proposal layer, and the two share network parameters.

[0087] Faster RCNN can refer to Ren Shaoqing's "Faster R-CNN: Towards real-time object detection with region proposal networks", which was published in "IEEE Transactions on Pattern Analysis and Machine Intelligence" on pages 1137-1149 in 2016.

[0088] The loss function of RAN is the same as that of RPN, which is the classification loss L cls and position loss L regs The weighted sum of is:

[0089]

[0090] Where α is the weight set, which is set to 1.

[0091] The classification loss L of the two cls For: L cls (p i ,g i)=-log[g i p i +(1-g i )(1-p i )], is the cross entropy loss, that is, the logarithmic loss is calculated for each sample and then divided by the total number of samples N, where p i is the probability that the i-th sample is predicted to be a positive sample, g i = {1,0} represents the label values ​​of foreground and background respectively. In the region proposal generation network, N regs Set to 256.

[0092] Position loss L regs for Among them, the function R is the Smooth L1 loss, which is: t i ={t x ,t y ,t w ,t h}, represents the offset of the prediction of the i-th sample, With t i The dimensions are the same, indicating the actual offset of the i-th sample relative to the true value; x, y, w, h represent the center coordinates, width, and height of the box.

[0093] Step 2.5: See Figure 2 , use the region of interest alignment pooling operation (ROI Align) to map each generated Proposals back to Conv5-T, Conv5-V, Conv5-F and pool them to obtain pooled features Pool-T, Pool-V, Pool-F of size 7×7;

[0094] Step 3.6: See Figure 2 , Pool-T and Pool-V are input into the late pooling feature fusion module for late pooling feature fusion. See the late pooling feature fusion network structure for details. Figure 3 , the specific operations are as follows:

[0095] (1): Pool-T and Pool-V are denoted as P V , P T , perform feature stacking, and then use global average pooling: f gp (·), compress the cascaded features and compress the 7*7*1024-dimensional features into F g , so that it has a global receptive field. The operation of this step is expressed by the formula:

[0096] (2): For F gThree fully connected layers are used to model modality correlation. The first FC layer transforms the feature F g The dimension is reduced to 256 to reduce the number of model parameters and the amount of calculation. Then, it is activated by the ReLU layer to enhance the nonlinearity of the features, so as to better fit the complex correlation between the modalities. Then, the features are separated by two parallel FC layers, and the two separated features are upgraded back to the original dimension. The operation of this step is expressed by the formula: F1 = f ad (δ(f sq (F g ))), F2=f ad (δ(f sq (F g ))), where f sq (·) represents the first full connection, f ad (·) represents the second parallel full connection, δ represents the ReLU operation, and F1 and F2 are the two separated features;.

[0097] (3): Use the Sigmoid activation function on F1 and F2 to obtain the weights w1 and w2 of the two modes, and multiply w1 and w2 with Pool-T and Pool-V respectively to obtain two weighted features. The operation of this step is expressed by the formula:

[0098] (4): Multiply the weights w1 and w2 with the original infrared and visible light features to obtain the two weighted features. Then concatenate the weighted features and use 1*1 convolution to reduce the dimension. Output Pool-W, denoted as P W The operation of this step is expressed by the formula: Where c and b represent the weight and bias of the convolution kernel respectively.

[0099] Step 3.7: Select the Conv3-T and Conv3-V feature pairs from ST and SV in step 3.1, cascade Conv3-T and Conv3-V and reduce the dimension to obtain the fusion feature Conv3-F, obtain the pooling feature Pool-conv3 on Conv3-F through step 3.5, use the high-low layer feature cascade module to perform feature cascade and dimension reduction on Pool-conv3, Pool-W, and Pool-F to obtain the features used for detection;

[0100] Step 3.8: Send the features obtained in step 3.7 to the fully connected layer to obtain the classification score and positioning box of the pedestrian target;

[0101] Step 3.9: Repeat the above steps and iterate the training until the optimal detection model is obtained. During the training phase, the overall loss function of the network is: It is the weighted sum of two groups of RAN loss, one group of RPN loss and one group of RCNN loss. Among them, λ1, λ2, λ3, λ4 are the weights of each sub-network, all set to 1. They are the regional attention network loss of the visible light stream, the regional attention network loss of the infrared stream, and the regional proposal generation network loss of the fusion stream; the RCNN loss is similar to the RPN loss in 3.2.4, the classification loss uses the cross entropy loss, which is the Smooth L1 loss, and the overall RCNN loss is the weighted sum of the position loss and the classification loss.

[0102] Step (4): Test the network model and set the pedestrian intersection over union (IOU) to 0.5:

[0103] Step 4.1: Input the paired infrared and visible light images in the test set Test-20 into the detection model, complete feature extraction, feature fusion, and classification regression, obtain the location and classification score of the pedestrian target, and output the detection results.

[0104] In order to demonstrate that the multi-spectral pedestrian detection method based on multi-stage feature fusion information reuse proposed in this application has achieved better performance than the existing pedestrian detection method using single-stage feature fusion, this application uses two methods to conduct simulation experiments on the KAIST dataset, see Figures 4 to 7 The thin borders are the original annotations provided by the dataset, and the thick borders are the predicted boxes output by the detector.

[0105] Depend on Figures 4 to 7 It can be seen that the shooting scene is a street environment in the daytime. The background of the street environment is relatively messy and pedestrian targets are easily confused. The multimodal networks of the two methods detect pedestrian targets. However, the multimodal network using the existing single-stage fusion strategy has false detections. Figure 4 and Figure 5 The detection results obtained using the existing single-stage fusion strategy shown in the figure mistakenly detect several black obstacles at the rightmost store door as pedestrians. In the visible light image, the black obstacles at the store door are indeed easily confused with pedestrians, but in the infrared image, the feature information of these objects is not significant.

[0106] This shows that the existing single-stage fusion strategy does not fully utilize the information of the two modalities, resulting in false detection. Figure 6 and Figure 7 In the detection results shown by the method of the present application, all pedestrian targets are detected without false detection or missed detection, which verifies the effectiveness of the multi-stage fusion strategy of the present application.

[0107] In order to further verify and highlight the effectiveness of the method proposed in this application, it is compared with the existing advanced methods ACF+T+THOG, Halfway-fusion, Fusion-RPN, IAF-RCNN, and TS-RPN on the KAIST multispectral pedestrian detection dataset, as shown in Table 1. It can be seen that this application has achieved the lowest average missed detection rate (MR): 27.63% and the highest average precision (AP): 76.24% in the whole period. Compared with the suboptimal algorithm Fusion-RPN, the AP value is increased by 0.9%, and the MR value is reduced by 2.1%. Compared with the other algorithms, the average missed detection rate is reduced by 27.17%, 8.37%, 3.67%, and 3.03%, respectively, and the average precision is increased by 23.01%, 7.36%, and 4.33%, respectively. Obviously, the performance is better than several other methods.

[0108] Table 1: Comparison results of relevant evaluation indicators of this application and different algorithms on the KAIST multispectral pedestrian detection set

[0109]

[0110] Among the existing advanced methods mentioned above:

[0111] ACF+T+THOG uses the ACF feature extraction algorithm to extract bimodal features, and then uses the feature cascade method to fuse multimodal features. Since it is the first multispectral pedestrian detection algorithm proposed for the KAIST multispectral dataset, it is often used as a baseline algorithm. Please refer to Hwang Soonmin's "Multispectral pedestrian detection: Benchmark dataset and baseline", which was published in "IEEE Conference on Computer Vision and Pattern Recognition" in 2015, pages 1037-1045.

[0112] Halfway-fusion uses the Faster R-CNN target detector as the basic framework for pedestrian detection, and adopts the intermediate layer feature channel stacking strategy to complete the fusion of infrared and visible light modalities, so as to maximize the integration of infrared and visible light information, while better retaining low-level detail information and high-level semantic information. Please refer to Liu Jingjing's "Multispectral deep neural networks for pedestrian detection", which was published in the "British Machine Vision Conference" in 2016, pages 73.1-73.13.

[0113] Fusion-RPN uses the region proposal generation network in the Faster R-CNN framework and the enhanced decision tree classification algorithm (Boosted Decision Trees, BDT) to jointly complete multispectral pedestrian detection. In the modal fusion part, the intermediate layer feature channel stacking strategy of Halfway-fusion is used. Please refer to Daniel Konig's "Fully convolutional region proposal networks for multispectral person detection", which was published in "IEEE Conference on Computer Vision and Pattern Recognition" in 2017, pages 49-56.

[0114] IAF-RCNN introduces an illumination perception mechanism into the multispectral pedestrian detection framework. It designs an illumination perception module to simulate the external lighting conditions and judge the probability of day or night. It then assigns different weights to the infrared and visible light modalities according to their different characteristics in different lighting environments for weighted fusion of the two. For details, please refer to Li Chengyang's "Illumination-aware faster R-CNN for robust multispectral pedestrian detection", which was published in Pattern Recognition in 2019, pages 161-171.

[0115] TS-RPN uses a dual-stream region proposal generation network. In the modal fusion part, it selects intermediate layer features for stacking fusion, and uses the annotation information of the multispectral dataset to iteratively mark pedestrian instances, automatically generating more reliable labels to improve the robustness of the pedestrian detection algorithm and reduce the cost of manual labeling. For reference, Cao Yanpeng's "Pedestrian detection with unsupervised multispectral feature learning using deep neural networks", which was published in "Information Fusion" on pages 206-217 in 2019.

[0116] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multispectral pedestrian detection method based on multi-stage feature fusion information multiplexing, characterized in that: The method comprises: The paired infrared and visible light images are respectively extracted from the intermediate layer features through the two-stream VGG16 network for early feature stacking and fusion to obtain early fusion features, and pedestrian suggestions are generated based on the early fusion features; each generated pedestrian suggestion is first mapped back to the three multimodal features for multi-feature pooling to obtain high-level pooling features, and then high-level pooling features are weighted fused, and high- and low-level feature pooling strategies are used to combine high- and low-level pooling features to complete the combination of high- and low-level features and obtain high- and low-level pooling joint features; finally, the high- and low-level pooling joint features are sent to the fully connected layer to obtain the classification score and positioning box of the pedestrian target; The pedestrian suggestion refers to the preliminary positioning information of the pedestrian; each pair of infrared and visible light images is an infrared image and a visible light image taken at the same time and in the same scene and registered; The method comprises: S1, obtaining pairs of infrared images and visible light images; each pair of images is an infrared image and a visible light image taken at the same time and in the same scene and registered; S2, extract low-level features, middle-level features, and high-level features from each pair of infrared images and visible light images through a two-stream VGG16 network; S3, perform feature fusion on the extracted infrared low-layer feature Conv3-T and the visible low-layer feature Conv3-V to obtain the low-layer fusion feature Conv3-F, and perform a region of interest pooling operation on the low-layer fusion feature Conv3-F to obtain the low-layer pooling feature Pool-conv3; perform early feature stacking fusion on the extracted infrared intermediate layer feature Conv4-T and the visible intermediate layer feature Conv4-V to obtain the early fusion feature Conv4-F, and perform a convolution operation on the early fusion feature Conv4-F to obtain the third modality high-level feature Conv5-F, and perform a region of interest pooling operation on the third modality high-level feature Conv5-F to obtain the third modality high-level pooling feature Pool-F; perform region of interest pooling operations on the extracted infrared high-level feature Conv5-T and the visible high-level feature Conv5-V to obtain the infrared high-level pooling feature Pool-T and the visible high-level pooling feature Pool-V respectively; S4, performing a post-pooling feature weighted fusion operation on the obtained infrared high-level pooling feature Pool-T and the visible high-level pooling feature Pool-V to obtain a post-weighted fusion pooling feature Pool-W, and performing a cascade operation on the post-weighted fusion pooling feature Pool-W, the third modality high-level pooling feature Pool-F and the low-level pooling feature Pool-conv3 to obtain a high- and low-level pooling joint feature, and finally sending the high- and low-level pooling joint feature to the fully connected layer to obtain the classification score and positioning box of the pedestrian target.

2. A multi-spectral pedestrian detection network framework system based on multi-stage feature fusion information reuse, characterized in that: The network framework includes: a feature extraction module, an intermediate layer feature fusion module, a region proposal generation module, a regional attention module, a late pooling feature fusion module, a high-low layer feature cascade module and a fully connected layer; wherein, the infrared intermediate layer feature Conv4-T and the visible intermediate layer feature Conv4-V are extracted by the feature extraction module and sent to the intermediate layer feature fusion module to obtain the early fusion feature Conv4-F, and then the early fusion feature Conv4-F is convolved to obtain the third modality high-level feature Conv5-F, which is input into the region proposal generation module to obtain the pedestrian suggestion, and at the same time, the infrared high-level feature Conv5-T and the visible high-level feature Conv5-V extracted by the feature extraction module are connected to the regional attention module to optimize the features, and then the pedestrian suggestion is mapped back to the feature suggestion. Take the low-layer fusion feature Conv3-F, infrared high-level feature Conv5-T, visible high-level feature Conv5-V extracted by the module, and the feature Conv5-F obtained by the intermediate layer feature fusion module, and pool them to obtain the low-layer pooling feature Pool-conv3, infrared high-level pooling feature Pool-T, visible high-level pooling feature Pool-V, and third modality high-level pooling feature Pool-F respectively, and then input Pool-T and Pool-V into the late pooling feature fusion module to obtain the late weighted fusion pooling feature Pool-W, and then input Pool-W, Pool-conv3, and Pool-F into the high- and low-layer feature cascade module to obtain the high- and low-layer pooling joint features, and input them into the fully connected layer to obtain the classification score and positioning box of the pedestrian target.

3. The network framework system according to claim 2, characterized in that: The feature extraction module and the fully connected layer use the VGG16 network pre-trained on ImageNet as the basic network.

4. The network framework system according to claim 3, characterized in that: The network framework adopts the following steps to perform training and testing to obtain the network framework under the optimal parameters: Step 1, construct training set Train-02 and test set Test-20; Step 2, training the network framework using the image training set Train-02; Step 2.1, sending the paired infrared and visible light images to the feature extraction module in turn to extract features until the infrared high-level features Conv5-T and the visible high-level features Conv5-V are generated respectively; the feature extraction module includes two networks with exactly the same network structure, which are used to extract low-level, middle-level and high-level features of the visible light stream SV and the infrared stream ST respectively; Step 2.2, select the last layer features of the fourth convolution from the visible light stream SV and the infrared stream ST with feature sizes of C×H×W: Conv4-T and Conv4-V are input into the intermediate layer feature fusion module to complete the early feature fusion operation; (1) Perform feature cascading in the channel dimension to obtain stacked features of size 2C×H×W; (2) Use 1*1 convolution to reduce the dimension and exchange information of the stacked features to obtain the early fusion feature Conv4-F with a size of C×H×W; Step 2.3: Send the early fusion feature Conv4-F into the fusion stream SF to generate the third modality high-level feature Conv5-F, where the fusion stream is the fifth convolution operation of the VGG16 network, including a maximum pooling layer, three groups of 3*3 convolution layers and ReLU activation layers; Step 2.4: After the third modality high-level feature Conv5-F, the region proposal generation module is connected to generate pedestrian proposals; The regional attention module is connected after the infrared high-level feature Conv5-T and the visible high-level feature Conv5-V; Step 2.5: Use the region of interest to map each generated pedestrian proposal back to Conv5-T, Conv5-V, Conv5-F and pool them to obtain pooled features Pool-T, Pool-V, Pool-F of size 7×7; Step 2.6: Input Pool-T and Pool-V into the late pooling feature fusion module for late pooling feature fusion: (1): Pool-T and Pool-V are stacked, and the stacked features are compressed using global average pooling to obtain global information; (2) The features obtained in the previous step are compressed and stacked using three fully connected layers to model the modality correlation and separate two features at the same time. (3): Use the Sigmoid activation function to obtain the weights w1 and w2 of the two modes for the two features obtained in the previous step. Multiply w1 and w2 with Pool-T and Pool-V respectively to obtain the corresponding two weighted features. (4): The two weighted features obtained in the previous step are concatenated and dimensionally reduced to obtain the late fusion feature Pool-W; Step 2.7: Select the infrared low-level feature Conv3-T and visible low-level feature Conv3-V from ST and SV in step 2.1, cascade Conv3-T and Conv3-V and reduce the dimension to obtain the low-level fusion feature Conv3-F, obtain the low-level pooling feature Pool-conv3 on Conv3-F through step 2.5, use the high-low-level feature cascade module to perform feature cascade and dimension reduction on Pool-conv3, Pool-W, Pool-T and Pool-V to obtain the features used for detection; Step 2.8: Send the features used for detection obtained in step 2.7 to the fully connected layer to obtain the classification score and positioning box of the pedestrian target; Step 2.9: Repeat steps 2.1 to 2.8 above and iterate the training until the network framework with the optimal parameters is obtained; Step (4): Test the network framework under the optimal parameters.

5. The network framework system according to claim 4, characterized in that: During the training process of the network framework, step 1 includes: Step 1.1, sample the training part video sequences of infrared and visible light videos in the multispectral pedestrian detection dataset every 2 frames to obtain the paired infrared and visible light image training set Train-02. Each pair of images with the same sequence number in the image training set Train-02 is an infrared image and a visible light image taken at the same time and in the same scene and after registration; The test part of the video sequence in the multispectral pedestrian detection dataset is sampled every 20 frames to obtain the paired infrared and visible light image test set Test-20. Each pair of images with the same sequence number in the image test set Test-20 is an infrared image and a visible light image taken at the same time and in the same scene and registered. Step 1.2: Flip all images in the image training set Train-02 to expand the training data.

6. The network framework system according to claim 5, characterized in that: The region proposal generation module in the network framework adopts the region proposal generation network RPN of the Faster RCNN detection network, and uses two 1*1 convolutions to respectively predict the position and classification score of the pedestrian.

7. The network framework system according to claim 6, characterized in that: The loss function of the RPN is the classification loss L cls and position loss L regs The weighted sum of: Where α is the weight set, which is set to 1.

8. The network framework system according to claim 7, characterized in that: The classification loss Lcls is: cls (p i ,g i )=-log[g i p i +(1-g i )(1-p i )], is the cross entropy loss, that is, the logarithmic loss is calculated for each sample and then divided by the total number of samples N; where p i is the probability that the i-th prediction box is predicted as a pedestrian target, g i ={1,0} represents the label values ​​of foreground and background respectively; in the region proposal generation network, Nregs represents the number of selected negative samples, which is set to 256; Position loss L regs for Among them, function R is the Smooth L1 loss expression, x represents represents the offset of the prediction of the i-th sample, With t i The dimensions are the same, indicating the actual offset of the i-th sample relative to the true value; x1, y1, w, and h represent the center point coordinates, width, and height of the detection box.

Images