Pedestrian detection method based on multi-modal information aggregation

Abstract

The invention discloses a pedestrian detection method based on multi-modal information aggregation, and the method comprises the steps: constructing an L-Det network model with a VGG16 network module structure, and obtaining the pooling features of an infrared image and the pooling features of a visible light image through a VGG16 network module, early-stage feature information and medium-stage feature information obtained in the VGG16 network module are fused through an early-stage and medium-stage feature fusion module to obtain pooling features of early-stage fusion features and pooling features of medium-stage fusion features respectively; and carrying out weighted feature fusion on the two pooling features through a pooling feature weighted fusion module to obtain weighted pooling features, cascading the pooling features of the early fusion features, the pooling features of the middle fusion features and the weighted pooling features, and then outputting a pedestrian detection result through a whole chain layer. According to the method, multi-stage fusion is realized, the limitation of single-stage fusion is made up, the feature information can be reused for multiple times, the loss of the feature information is reduced, and the robustness is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian detection, and in particular to a pedestrian detection method based on multimodal information aggregation. Background Art

[0002] Due to the safety issues of self-driving cars, robots and surveillance systems, high robustness pedestrian detection is becoming more and more important in the field of computer vision. Although some progress has been made in this field, high robustness in pedestrian detection is still a challenging problem due to the large number of unpredictable situations in the real world, as well as the influence of pedestrian posture, pedestrian occlusion and lighting environment.

[0003] Nowadays, all-weather sensor detection has become inevitable, and sensor data fusion has also become imperative. Pedestrian detection generally uses visible light waves with a wavelength of 390-750nm. Electromagnetic waves in this range can be perceived by humans. In real life, there are often bad weather or environments, such as poor lighting conditions, smoke, dust, rainy days, etc. These conditions can easily lead to the loss of pedestrian target information. However, for applications such as autonomous driving and intelligent monitoring, bad environments and weather are inevitable. Once the detection robustness is low, it is very dangerous and will pose a great threat to pedestrian safety.

[0004] Some existing multispectral pedestrian detection algorithms often only fuse infrared and visible light for the features of a certain feature layer, which wastes the features of other feature layers and affects the accuracy of pedestrian detection. In addition, most of the current direct stacking of infrared and visible light modes is a rough fusion strategy with low detection effect. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a pedestrian detection method based on multimodal information aggregation to solve the technical problems in the prior art.

[0006] The present invention provides a pedestrian detection method based on multimodal information aggregation, comprising the following steps:

[0007] Step 1: Construct an L-Det network model with a VGG16 network module structure. The L-Det network model also includes: an early and mid-term feature fusion module and a feature weighted fusion module;

[0008] Step 2: Acquire infrared image feature information and visible light image feature information including pedestrians;

[0009] Step 3: The infrared image feature information and the visible light image feature information are respectively obtained through the VGG16 network module to obtain the pooling features of the infrared image and the pooling features of the visible light image;

[0010] Step 4: The infrared image feature information and the early feature information and mid-term feature information of the visible light image obtained in the VGG16 network module are fused through the early and mid-term feature fusion module to obtain the pooling features of the early fusion features and the pooling features of the mid-term fusion features respectively;

[0011] Step 5: The pooling features of the infrared image and the pooling features of the visible light image are fused by a pooling feature weighted fusion module to obtain weighted pooling features;

[0012] Step 6: Cascade the pooled features of the early fusion features, the pooled features of the mid-term fusion features, and the weighted pooled features and output the pedestrian detection results through the full chain layer.

[0013] Furthermore, the specific process of step 3 is as follows:

[0014] The infrared image feature information and the visible light image feature information are respectively processed by three 3×3 convolutions in the VGG16 network module to generate the corresponding candidate regions, and the corresponding pooled features of the infrared image and the pooled features of the visible light image are obtained.

[0015] Furthermore, in step 3, the VGG16 network module also includes: a regional attention network; after the infrared image feature information and the visible light image feature information are convolutionally processed in the VGG16 network module, they are optimized by the regional attention network respectively, and then the optimized candidate regions are generated to obtain the pooled features of the infrared image and the pooled features of the visible light image.

[0016] Furthermore, the specific steps of step 4 are as follows:

[0017] Step 41: Obtain infrared image feature information and visible light image feature information. Obtain early feature information of the infrared image and early feature information of the visible light image after a 3×3 convolution process in the VGG16 network module.

[0018] Step 42: Obtain infrared image feature information and visible light image feature information. After two 3×3 convolutions in the VGG16 network module, the mid-term feature information of the infrared image and the mid-term feature information of the visible light image are obtained.

[0019] Step 43: Cascading the early feature information of the infrared image and the early feature information of the visible light image to obtain the early fusion feature; cascading the mid-term feature information of the infrared image and the mid-term feature information of the visible light image, and then performing two 3×3 convolution processes to obtain the mid-term fusion feature;

[0020] Step 44: Generate corresponding candidate regions through early fusion features and mid-term fusion features, and obtain pooled features of early fusion features and pooled features of mid-term fusion features after pooling the candidate regions.

[0021] Furthermore, in step 4, the early and mid-term feature fusion module also includes: a region proposal generation network; early feature information and mid-term feature information are used to generate pedestrian suggestions through the region proposal generation network, and the early feature information and mid-term feature information are combined with the pedestrian suggestions to generate candidate regions.

[0022] Furthermore, the specific steps of step 5 are as follows:

[0023] Step 51: cascading the pooling features of the infrared image and the pooling features of the visible light image to obtain a fused pooling feature;

[0024] Step 52: The fused pooled features are processed by a global pooling layer, a full chain layer, two parallel full chain layers, and two parallel Sigmoid gate functions in sequence to obtain normalized weights of infrared image pooling features and normalized weights of visible light image pooling features;

[0025] Step 53: multiplying the pooled features of the infrared image by the normalized weights of the corresponding pooled features, and multiplying the pooled feature level of the visible light image by the corresponding normalized weights;

[0026] Step 54: Cascade the two product results obtained in step 53 to obtain the weighted pooling feature.

[0027] Beneficial effects of the present invention:

[0028] The L-Det network proposed in the present invention is a multi-stage fusion, which makes up for the limitation of single-stage fusion, so that feature information can be reused multiple times, reduces feature information loss, and improves robustness.

[0029] In the present invention, the feature information of two inputs is connected to the regional attention network after being convolutionally processed in the VGG16 network module, so that the network can better utilize the feature information.

[0030] The present invention integrates the region proposal network into the early and mid-term feature fusion module, optimizes the performance of its respective branches, and can assist in the generation of candidate regions.

[0031] In order to retain the useful information in the pooled features and discard the irrelevant and redundant information, the present invention adds a pooled feature weighted fusion module (LR) to the network to filter the information to achieve more accurate detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0033] Figure 1 It is a schematic diagram of a flow chart of a specific embodiment of the present invention;

[0034] Figure 2 Schematic diagram of the L-Det network model structure of a specific embodiment of the present invention;

[0035] Figure 3 is a structural diagram of a pooling feature weighted fusion module in a specific embodiment of the present invention;

[0036] Figure 4 It is a schematic diagram of the working process of the region proposal generation network in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0038] The present invention is further illustrated below in conjunction with specific embodiments. Those skilled in the art should understand that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention, and modifications to various equivalent forms of the present invention fall within the scope defined by the appended claims of this application.

[0039] The present invention provides a pedestrian detection method based on multimodal information aggregation, such as Figure 1 As shown, the following steps are included:

[0040] Step 1: Build an L-Det network model with a VGG16 network module structure, such as Figure 2 As shown, the L-Det network model also includes: early and mid-term feature fusion module and feature weighted fusion module;

[0041] The loss function of the L-Det network model includes: RAN loss, RPN loss and RL loss, which is the weighted sum of the three losses. The RAN loss is divided into two groups. The specific formula is as follows:

[0042]

[0043] Among them, λ x ; x∈(1, 2, 3, 4) are the sub-weight values ​​of the network, which are all set to 1 in this experiment; is the RAN loss in the visible light flow; is the RAN loss in the infrared flow; RPN loss for the fusion of infrared and visible light streams.

[0044] Similarly, they are the weighted sum of classification loss and position loss, For example, it can be expressed as:

[0045]

[0046] RL loss is similar to RPN loss. The classification loss in RL loss uses cross entropy loss, and the formula is:

[0047]

[0048] Among them, k i is the true label category in the input sample, k i The value can be 0 or 1. When it is 1, it means the label is positive, and when it is 0, the label is a negative sample; Indicates the probability that the input sample is a pedestrian; the RL position loss uses Smooth L1 loss, and the specific formula is:

[0049]

[0050] Among them, g i = {g x ,g y ,g w ,g h}, the box coordinates representing the location of the true label, With g i The dimensions are the same, Represents the box coordinates of the predicted value. R(x) is specifically expressed as:

[0051]

[0052] The complete expression of RL loss function is and The weighted sum of is:

[0053]

[0054] Among them, β is the weighted value of classification loss and regression loss, and its value is 1; N regs Set to 128.

[0055] Step 2: Obtain infrared image feature information and visible light image feature information including pedestrians, as follows:

[0056] Step 21: Collect and create infrared and visible light datasets;

[0057] Step 22: Divide the obtained data set into training set, test set and validation set according to the ratio of 3:1:1.

[0058] Step 23: Align and label the dataset;

[0059] Step 24: Preprocess the collected and produced data set, adjust the image size to 640×480 pixels, and clean the data set. This completes the acquisition of infrared image feature information and visible light image feature information;

[0060] Step 3: The infrared image feature information and the visible light image feature information are respectively obtained through the VGG16 network module to obtain the pooling features of the infrared image and the pooling features of the visible light image.

[0061] A multi-step feature fusion network with VGG16 as the basic network is constructed. The 640×480×3 visible light image and infrared image are converted into 224×224×3 images and input into the VGG16 network respectively. After three 3×3 convolution processes, they are connected to the regional attention network (RAN) to optimize the dual-stream features. The candidate regions (Proposals) corresponding to the visible light image and the infrared image are generated through the optimized dual-stream features. The pooled features of the infrared image and the pooled features of the visible light image are obtained by pooling the candidate regions.

[0062] Step 4: The infrared image feature information and the early feature information and mid-term feature information of the visible light image obtained in the VGG16 network module are fused through the early and mid-term feature fusion module to obtain the pooling feature Pool-I of the early fusion feature and the pooling feature Pool-F of the mid-term fusion feature. The specific steps are as follows:

[0063] Step 41: Obtain infrared image feature information and visible light image feature information. Obtain early feature information of the infrared image and early feature information of the visible light image after a 3×3 convolution process in the VGG16 network module.

[0064] Step 42: Obtain infrared image feature information and visible light image feature information. After two 3×3 convolutions in the VGG16 network module, the mid-term feature information of the infrared image and the mid-term feature information of the visible light image are obtained.

[0065] Step 43: Cascading the early feature information of the infrared image and the early feature information of the visible light image to obtain the early fusion feature; cascading the mid-term feature information of the infrared image and the mid-term feature information of the visible light image, and then performing two 3×3 convolution processes to obtain the mid-term fusion feature;

[0066] The cascade (IME) operation can be written as:

[0067]

[0068] in, They represent the feature information from the visible light stream and the infrared stream respectively. C is the number of channels, which is set to 512. W and H represent the width and height of the feature map respectively. is the cascade feature after feature stacking, Indicates a cascade operation.

[0069] Step 44: The early feature information and the mid-term feature information are passed through the region proposal generation network (RPN) to generate pedestrian suggestions. The early feature information, the mid-term feature information and the pedestrian suggestions are combined to generate candidate regions. After pooling the candidate regions, the pooling features Pool-I of the early fusion features and the pooling features Pool-F of the mid-term fusion features are obtained.

[0070] like Figure 4 As shown in the figure, it is a region proposal generation network. The early feature information and the mid-term feature information are respectively input into the region proposal network. A 3*3 sliding window is used in the network to slide the window in the features. In this process, 9 anchors with different scales and aspect ratios are generated. Then each anchor will perform preliminary regression to obtain the probability value belonging to the foreground and the corresponding coordinate value of the original image. Finally, the first 300 candidate regions are selected as output from a large number of candidate regions, and then the candidate regions are mapped back to the feature map using the region of interest pooling (ROIPooling) operation to obtain pedestrian suggestions.

[0071] Step 5: Figure 3 As shown, the pooling feature Pool-T of the infrared image and the pooling feature Pool-V of the visible light image are weightedly fused through the pooling feature weighted fusion module to obtain the weighted pooling feature PooL-L. The specific steps are as follows:

[0072] Step 51: Cascading the pooling feature Pool-T of the infrared image and the pooling feature Pool-V of the visible light image to obtain a fused pooling feature;

[0073] Step 52: The fused pooled features are processed by a global pooling layer, a full chain layer, two parallel full chain layers, and two parallel Sigmoid gate functions in sequence to obtain normalized weights of infrared image pooling features and normalized weights of visible light image pooling features;

[0074] In the process, the global pooling layer (Global Pooling: Pg(x)) is used to compress the features. The formula is:

[0075]

[0076] Three fully connected layers are used to establish the correlation between the two modalities. The first fully connected layer is used to reduce the dimension to 256 to reduce the model parameters. Then two parallel FC layers are used to separate the features, separate the deeper infrared and visible light pooling features, and increase the dimension of the two features.

[0077] The two separated features are passed through the Sigmoid gate function to obtain the normalized weight w1 of the infrared image pooling feature and the normalized weight w2 of the visible light image pooling feature, which can be expressed as:

[0078]

[0079]

[0080] Step 53: Multiply the pooling feature Pool-T of the infrared image by the normalized weight w1 of the corresponding pooling feature, and multiply the pooling feature Pool-V of the visible light image by the corresponding normalized weight w2 to obtain the corresponding weighted feature.

[0081] Step 54: Cascade the two product results obtained in step 53 to obtain the weighted pooling feature PooL-L, the formula is:

[0082]

[0083] Among them, P is the weighted pooling feature PooL-L; c and b represent the weight and bias of the convolution kernel respectively, P V is the pooling feature Pool-V of the visible light image, P T It is the pooling feature PooL-T of infrared image.

[0084] Step 6: Cascade the pooled features of the early fusion features, the pooled features of the mid-term fusion features, and the weighted pooled features and output the pedestrian detection results through the full chain layer.

[0085] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A pedestrian detection method based on multimodal information aggregation, characterized in that: The steps include: Step 1: Construct an L-Det network model with a VGG16 network module structure. The L-Det network model also includes: an early and mid-term feature fusion module and a feature weighted fusion module; Step 2: Acquire infrared image feature information and visible light image feature information including pedestrians; Step 3: The infrared image feature information and the visible light image feature information are respectively obtained through the VGG16 network module to obtain the pooling features of the infrared image and the pooling features of the visible light image; Step 4: The infrared image feature information and the early feature information and mid-term feature information of the visible light image obtained in the VGG16 network module are fused through the early and mid-term feature fusion module to obtain the pooling features of the early fusion features and the pooling features of the mid-term fusion features respectively; Step 5: The pooling features of the infrared image and the pooling features of the visible light image are fused by a pooling feature weighted fusion module to obtain weighted pooling features; Step 6: Cascade the pooled features of the early fusion features, the pooled features of the mid-term fusion features, and the weighted pooled features and output the pedestrian detection results through the full chain layer.

2. The pedestrian detection method based on multimodal information aggregation as claimed in claim 1, characterized in that: The specific process of step 3 is as follows: The infrared image feature information and the visible light image feature information are respectively processed by three 3×3 convolutions in the VGG16 network module to generate the corresponding candidate regions, and the corresponding pooled features of the infrared image and the pooled features of the visible light image are obtained.

3. The pedestrian detection method based on multimodal information aggregation as claimed in claim 2, characterized in that: In the step 3, the VGG16 network module also includes: a regional attention network; after the infrared image feature information and the visible light image feature information are convolutionally processed in the VGG16 network module, they are optimized by the regional attention network respectively, and then the optimized candidate regions are generated to obtain the pooled features of the infrared image and the pooled features of the visible light image.

4. The pedestrian detection method based on multimodal information aggregation as claimed in claim 1, characterized in that: The specific steps of step 4 are as follows: Step 41: Obtain infrared image feature information and visible light image feature information. Obtain early feature information of the infrared image and early feature information of the visible light image after a 3×3 convolution process in the VGG16 network module. Step 42: Obtain infrared image feature information and visible light image feature information. After two 3×3 convolutions in the VGG16 network module, the mid-term feature information of the infrared image and the mid-term feature information of the visible light image are obtained. Step 43: cascading the infrared image early feature information and the visible light image early feature information to obtain an early fusion feature; After cascading the mid-term feature information of the infrared image and the mid-term feature information of the visible light image, the mid-term fusion feature is obtained after two 3×3 convolution processes; Step 44: Generate corresponding candidate regions through early fusion features and mid-term fusion features, and obtain pooled features of early fusion features and pooled features of mid-term fusion features after pooling the candidate regions.

5. The pedestrian detection method based on multimodal information aggregation as claimed in claim 4, characterized in that: In step 4, the early and mid-term feature fusion module also includes: a region proposal generation network; early feature information and mid-term feature information are used to generate pedestrian suggestions through the region proposal generation network, and the early feature information and mid-term feature information are combined with the pedestrian suggestions to generate candidate regions.

6. The pedestrian detection method based on multimodal information aggregation as claimed in claim 1, characterized in that: The specific steps of step 5 are as follows: Step 51: cascading the pooling features of the infrared image and the pooling features of the visible light image to obtain a fused pooling feature; Step 52: The fused pooled features are processed by a global pooling layer, a full chain layer, two parallel full chain layers, and two parallel Sigmoid gate functions in sequence to obtain normalized weights of infrared image pooling features and normalized weights of visible light image pooling features; Step 53: multiplying the pooled features of the infrared image by the normalized weights of the corresponding pooled features, and multiplying the pooled feature level of the visible light image by the corresponding normalized weights; Step 54: Cascade the two product results obtained in step 53 to obtain the weighted pooling feature.

Images