Multispectral pedestrian detection method applied to intelligent traffic monitoring

Abstract

The invention belongs to the technical field of intelligent traffic and computer vision crossing, and particularly relates to a pedestrian detection method based on multispectral fusion. Aiming at the bottleneck problem that single-source data sensing reliability is insufficient in a complex traffic scene, the invention provides a visible light-infrared double-flow collaborative pedestrian detection framework based on a neural network. According to the method, complementary semantic information between visible light and infrared images is fully mined, a multispectral feature dynamic interaction mechanism composed of three stages of feature decoupling, interaction and reconstruction is established, and robust multispectral representation is generated through a feature fusion module based on channel attention. Compared with an existing method, the method has the advantages that target feature discrimination can be remarkably improved, higher adaptability is achieved in multi-scene, multi-scale and multi-target detection tasks, accurate positioning and recognition of pedestrian targets in the all-weather environment can be achieved, and reliable technical guarantee is provided for an intelligent traffic monitoring system.

Description

Technical Field

[0001] The present invention belongs to the cross - technical field of intelligent transportation and computer vision, and in particular relates to a pedestrian detection method based on multispectral fusion. Background Art

[0002] In recent years, significant progress has been made in object detection based on deep learning, and a large number of excellent frameworks have emerged. Currently, object detection is mainly divided into two - stage detection methods such as Faster RCNN and single - stage detection methods such as SSD and YOLO. For object detection in single - modality images, visible - light (RGB) images have rich texture and color information. However, in the face of challenges in adverse imaging conditions such as low illumination, rain, fog, etc., blurred target objects will greatly affect the detection performance. Therefore, RGB detectors are difficult to meet the actual needs of all - weather monitoring in traffic scenarios. Different from visible - light imaging, infrared imaging captures the thermal radiation of objects and is thus not restricted by illumination conditions. However, the lack of color and detail features also makes it difficult to accurately depict the target. To break through the limitations of single modality, multispectral pedestrian detection is on the rise.

[0003] Generally speaking, a common approach to multispectral pedestrian detection is to use a dual - path network to extract features from RGB and infrared images respectively, integrate RGB cues and infrared spectra to obtain rich target information, and then send it to the backend detection head. This way of simultaneously using image data from multiple spectral bands provides a more three - dimensional perspective for scene description and a more robust solution for target monitoring in complex scenarios. However, in these tasks, there are still two key challenges:

[0004] Considering that paired RGB - infrared images are highly correlated in terms of targets and scenes, it is challenging to better learn the potential semantic relationships between the two - modality data and obtain a compact and more discriminative multispectral feature representation. However, due to the different imaging methods of multispectral images, the RGB and infrared feature domains are essentially different. The huge difference in feature domains will cause a lack of compatibility between the two unimodal features, leading to the problem of insufficient feature fusion. Effective feature interaction can, to a certain extent, alleviate the modality differences and guide the enhancement of unimodal features. However, most current studies on RGB - infrared feature interaction adopt a dense feature transfer strategy, which is prone to information aliasing. Specifically, the two - modality features have a certain correlation. Each modality contains both homogeneous features that describe the common cues of the two modalities and specific features with unique components of each modality. When performing feature interaction, if only considering enriching the information of the other modality and transmitting features with complex spatial information without discrimination, it may cause the loss of modality - specific information due to feature aliasing and deepen the impact of redundant information on the detection performance.

[0005] To address the above challenges, we constructed a cross-attention-guided feature reconstruction module, which is applied to the visible-infrared feature interaction in the multi-spectral pedestrian detection model to enhance the model's discriminative ability for pedestrian targets. Through extensive experiments, it has been verified that the proposed multi-spectral pedestrian detection framework has stronger adaptability in the detection tasks of multi-scenes, multi-scales, and multi-targets, and can achieve accurate positioning and recognition of pedestrian targets in all-weather environments. Summary of the Invention

[0006] To solve the problem of insufficient reliability of single-source data perception in complex traffic scenarios, the present invention proposes a pedestrian detection method based on visible-infrared multi-spectral fusion of neural networks.

[0007] Given the input visible light image and infrared image . The purpose of the present invention is to utilize two-modal images to achieve accurate positioning of pedestrian targets in complex scenarios through four parts: dual-stream feature extraction, cross-modal feature interaction, multi-spectral feature fusion, and detection.

[0008] In the feature extraction stage, two independent backbone networks and are used to specifically extract visible-infrared multi-level features. ResNet50 is used as the feature extraction network, and the model is loaded with pre-trained weights on ImageNet to obtain two-modal multi-level features , i = 1, 2, 3, 4, and the number of feature channels is 256, 512, 1024, and 2048.

[0009] In the feature interaction stage, it is realized by cascading three modules: feature decoupling, dual-stream feature interaction, and modality reconstruction.

[0010] The feature decoupling unit uses group normalization weights to decouple the two-modal features into strong features and weak features respectively:

[0011]

[0012] where M represents the visible light modality R or the infrared modality I, and are the mean and standard deviation of respectively, and are trainable affine transformations, is the weight of each channel feature map, is a gating unit with a threshold of 0.5.

[0013] The two-stream feature interaction module consists of four cascaded cross-attention modules. It inputs the strong and weak features of the two modalities. The two modalities serve as the guiding domain and the base domain for each other. The guiding domain provides rich complementary information for the base domain to enhance the weak features of the base domain, and then outputs the weak features after interaction:

[0014]

[0015] Each cross-attention module first passes through 2 groups, each containing 3 depth-separable convolutions, to extract the query component, key tensor and value tensor of the two modalities respectively, where the query component is provided by weak features, and the key tensor and value tensor are provided by strong features. Then the query tensor and key tensor are used to establish attention weights through dot product operations and softmax activation, and then multiplied with the value vector, and the convolutional projection layer and residual connection are used to obtain the interactive two-modal weak features:

[0016]

[0017]

[0018]

[0019] Among them, Q, K, and V are query components, key tensors, and value tensors respectively; is the weight matrix of the depthwise separable convolutional layer; A is the attention weight.

[0020] The modality reconstruction unit restores the two modalities from the strong features and weak features by cross-adding them, aiming to further enhance the representative features and alleviate the information redundancy caused by feature interaction. Decompose the channels separately and get , then cross-add and concatenate along the channels:

[0021]

[0022] in, For point-by-point addition, Indicates a connection.

[0023] In the feature fusion stage, the two-modal features are first merged through channel aggregation operation to obtain the initial multispectral fusion features. Then, global maximum pooling and global average pooling are used to mine the channel correlation of multispectral fusion features, and the enhanced multispectral features are further obtained through channel aggregation operation and residual connection.

[0024]

[0025]

[0026] in Indicates channel connection, which is conv1 + Batchnormalization + ReLU. GAP represents global average pooling, and GMP is global max pooling.

[0027] In the detection stage, the obtained multi-level multi-spectral fusion features are first used for multi-scale feature fusion through the Feature Pyramid Network (FPN). Then, the Region Proposal Network (RPN) in Faster R-CNN is used to generate candidate regions, and accurate object localization and classification are achieved through the Region of Interest (RoI) head. Advantages and positive effects of the present invention

[0028] Compared with the prior art, the present invention has the following advantages and positive effects:

[0029] 1) The present invention proposes an all-weather pedestrian detection method applied to intelligent transportation monitoring, which fuses visible light and infrared multi-spectral images, solves the bottleneck problem of insufficient reliability of single-source data perception in complex traffic scenarios, designs an efficient multi-spectral feature adaptive fusion mechanism for the specific task of pedestrian detection, and establishes a multi-spectral pedestrian detection model that is robust to illumination.

[0030] 2) The present invention establishes a new paradigm for multi-spectral feature interaction of feature decoupling - interaction - reconstruction: a feature decomposition unit based on group normalization weights to decouple strong and weak features of different modalities, designs an iterative cross-attention module between modalities, establishes a cross-modal information interaction flow from strong features to weak features, and constructs a feature reconstruction unit to restore the two modalities.

[0031] 3) The proposed method of the present invention is tested on a pedestrian detection dataset in a traffic scenario. Experiments show that the proposed algorithm has stronger adaptability in detection tasks with multiple scenarios, multiple scales, and multiple targets, can achieve accurate localization and recognition of pedestrian targets in all-weather environments, and has significantly enhanced detection accuracy compared with existing methods. Description of the drawings

[0032] Figure 1 is the implementation flowchart of the present invention;

[0033] Figure 2 is a multi-spectral pedestrian detection framework diagram proposed by the present invention for intelligent transportation monitoring;

[0034] Figure 3 is the structure diagram of the feature interaction and fusion module in the multi-spectral pedestrian detection framework;

[0035] Figure 4 is the structure diagram of the feature decomposition unit in the feature interaction stage;

[0036] Figure 5 It is the structural diagram of the feature reconstruction unit in the feature interaction stage;

[0037] Figure 6 It is an example of the detection results of the present invention, MBNet, MLPD, and C2Former on the KAIST dataset when the confidence threshold is 0.4. Detailed implementation manners

[0038] To make the implementation manners, significance, and advantages of the present invention more clearly expressed, the present invention will be described in more detail below with reference to the following drawings and implementation examples.

[0039] (1) The multi-spectral feature extraction part consists of two branches of visible light and infrared. The ResNet50 backbone network is adopted, and the model is loaded with pre-trained weights on ImageNet, and the features of the first, second, third, and fourth stages of the two modalities are output respectively. The number of channels of the four hierarchical features is 256, 512, 1024, and 2048 respectively. The purpose is to obtain multi-level feature representations of the visible light and infrared modalities.

[0040] (2) The feature decoupling unit uses group normalization weights to decouple the features of the two modalities into strong features and weak features respectively. The purpose is to use strong features to retain the representative features of the modality, and weak features provide a container for modality interaction to avoid information aliasing caused by multi-spectral feature interaction.

[0041] (3) The feature interaction stage consists of 4 cascaded cross-attention modules. Each module first extracts the query component, key tensor, and value tensor of the two modalities through 2 groups, each group containing 3 depthwise separable convolutions. The query component is provided by the weak feature, and the key tensor and value tensor are provided by the strong feature. The channel-wise convolution kernel size in the depthwise separable convolution is 3, the stride is 1, and the pointwise convolution kernel size is 1. Then the query tensor and the key tensor are dot-product operated and activated by softmax to establish attention weights, and then multiplied by the value vector, and a convolutional projection layer and a residual connection are used to obtain the weak features of the two modalities after interaction. The kernel size of the convolutional projection layer is 3, and the stride is 1. The purpose of feature interaction is to establish a directional information interaction flow from strong features to weak features between the two modalities, avoid feature aliasing during interaction, and retain modality-specific information to the greatest extent.

[0042] (4) The feature reconstruction unit first evenly divides the strong features and weak features of each modality along the channel dimension, and then adopts the method of cross-adding and then channel connection to restore the features of the two modalities. The purpose is to further enhance the representative features and alleviate the information redundancy caused by feature interaction.

[0043] (5) The multi - spectral feature fusion part first merges the two - modality features through a channel aggregation operation to obtain the initial multi - spectral fusion features. The channel aggregation operation is specifically as follows: concatenate the features along the channels, then use a convolution with a kernel size of 1, batch normalization, and relu activation, and project back to the original channel dimension. Then, global max - pooling and global average - pooling are respectively used to mine the channel correlation of the multi - spectral fusion features, and further enhanced multi - spectral features are obtained through the channel aggregation operation and residual connection. The purpose is to model the channel correlation of the two modalities and adaptively aggregate the multi - spectral features.

[0044] (6) The multi - spectral detection part uses the obtained multi - level multi - spectral fusion features. First, multi - scale feature fusion is performed through a Feature Pyramid Network (FPN). Then, the Region Proposal Network (RPN) in Faster R - CNN is used to generate candidate regions, and accurate object localization and classification are achieved through the Region of Interest head (RoI head).

[0045] The hardware configuration of the simulation experiment of the present invention is: Intel(R) Xeon(R) Gold 6133 CPU @ 2.50GHz; NVIDIA GeForce RTX 4090 GPU.

[0046] The software configuration of the simulation experiment of the present invention is: ubuntu20.0 operating system, the simulation language is Python, and the software libraries are pytorch1.9.0 and MMDetection.

[0047] In the simulation experiment, the dataset used is the KAIST Dataset. This dataset includes visible - light - infrared image pairs captured in different scenes under different lighting conditions, with a resolution of 640×512. The present invention uses 8963 and 2,252 pairs for training and testing respectively.

[0048] The present invention is trained using a Stochastic Gradient Descent (SGD) optimizer with a momentum of 0.9, a weight decay factor of 0.0001, an initial learning rate of 0.01, a training batch size of 3, and the training is terminated at the 10th epoch.

[0049] The evaluation metric for detection accuracy is the Log - Average Miss Rate (LAMR), which represents the average miss rate (MR) at 9 False Positives per Image (FPPI) values, uniformly sampled in the logarithmic interval [10 - 2,1]. MR represents the percentage of missed objects (false negatives) among all ground truths (true positives and false negatives):

[0050]

[0051]

[0052] Among them, FN represents the number of samples that are actually positive but are wrongly predicted as negative; TP represents the number of samples that are actually positive and are correctly predicted as positive; FP represents the negative samples that are predicted as positive samples; and N is the total number of samples. The lower the value of LAMR, the better the performance.

[0053] Table 1 Comparison of LAMR (%) between the present invention and existing multispectral pedestrian detectors

[0054]

[0055] Table 1 shows the detection accuracies of different models under four partitioning methods of the KAIST dataset. The scale subsets are partitioned by the height of the pedestrians: "large" (115 ≤ height), "medium" (45 ≤ height < 115) and "small" (1 ≤ height < 45); the occlusion subsets are partitioned by the degree of occlusion of the target; the reasonable subsets are partitioned into day and night under the setting that the height of the pedestrian is greater than 55 pixels and the target is unoccluded or partially occluded. It can be seen from the table that under reasonable settings, the present invention has achieved results superior to the advanced level and can detect pedestrian targets with high accuracy both during the day and at night. In the more challenging full dataset containing small targets and occluded pedestrians, our method has achieved state-of-the-art results and can ensure accurate detection of targets under complex conditions and in full scenarios. Additionally, under the scale subsets and occlusion subsets, it is further proved that the present invention is more competitive under the conditions of small targets and partial occlusion.

[0056] Figure 6 Shows partial detection results of the present invention and three representative visible light-infrared object detectors, namely MBNet, MLPD, and C2Former, on the KAIST dataset, including different scenarios during the day and at night. It can be intuitively seen that the present invention can achieve lower false detection rates and missed detection rates. It can be seen from the first two rows that there are cases of missed detection for both MBNet and MLPD, and there is a case of false detection for C2Former in the third row. In comparison, in such a scenario with more targets, the present invention can basically ensure comprehensive and correct detection of the targets. In the last three rows of night scenarios, the present invention also shows more advanced performance and can still achieve accurate detection of pedestrians even under low light and complex lighting conditions.

Claims

1. A multispectral image pedestrian detection method for intelligent traffic monitoring, including multispectral feature extraction, interaction, fusion and detection. The characteristics of the method are as follows: (1) The multispectral feature extraction part consists of two branches: visible light and infrared. It uses the ResNet50 backbone network and loads the model with pre-trained weights on ImageNet to output the features of the first, second, third, and fourth stages of the two modalities. The number of channels of the four-level features is 256, 512, 1024, and 2048 respectively. The purpose is to obtain multi-level feature representations of the two modalities of visible light and infrared. (2) The multispectral feature interaction part consists of three stages: feature decoupling, interaction, and reconstruction. The purpose is to fully explore the complementary information between visible light and infrared features, alleviate modal differences, and provide a guarantee for the full fusion of multispectral features: 1) The feature decoupling unit uses group normalization weights to decouple the two modal features into strong features and weak features respectively. The purpose is to use strong features to retain the representative features of the modality, and weak features to provide a container for modal interaction to avoid information aliasing caused by multi-spectral feature interaction; 2) The feature interaction stage consists of 4 cascaded cross-attention modules. Each module first passes through 2 groups, each containing 3 depth-wise separable convolutions, to extract the query component, key tensor and value tensor of the two modalities respectively. The query component is provided by weak features, and the key tensor and value tensor are provided by strong features. The channel-by-channel convolution kernel size in the depth-wise separable convolution is 3, the stride is 1, and the point-by-point convolution kernel size is 1. Then the query tensor and key tensor are used to establish attention weights through dot product operations and softmax activation, and then multiplied with the value vector, and a convolutional projection layer and residual connection are used to obtain the weak features of the two modalities after interaction. The kernel size of the convolutional projection layer is 3 and the stride is 1. The purpose of feature interaction is to establish a directional information interaction flow from strong features to weak features between the two modalities, avoid feature aliasing during interaction, and retain modality-specific information to the maximum extent; 3) The feature reconstruction unit first divides the strong and weak features of each modality equally along the channel dimension, and restores the features of the two modalities by cross-adding and then connecting the channels. The purpose is to further enhance the representative features and alleviate the information redundancy caused by feature interaction; (3) The multispectral feature fusion part first merges the two-modal features through channel aggregation operation to obtain the initial multispectral fusion features. The channel aggregation operation is specifically as follows: splicing features along the channel, and then using convolution with a kernel size of 1, batch normalization and relu activation to project back to the original channel dimension. Then, global maximum pooling and global average pooling are used to mine the channel correlation of multispectral fusion features, and further through channel aggregation operation and residual connection, the enhanced multispectral features are obtained. The purpose is to model the channel correlation of the two modalities and adaptively aggregate multispectral features; (4) The multispectral detection part uses the acquired multi-level multispectral fusion features to first perform multi-scale feature fusion through the feature pyramid network (FPN). Then, the region proposal network (RPN) in Faster RCNN is used to generate candidate regions, and the region of interest head (RoI head) is used to achieve accurate positioning and classification of the target.

Images