A lightweight pedestrian detection method in harsh environments based on deep learning

Abstract

The present invention discloses a lightweight pedestrian detection method in harsh environments based on deep learning, which comprises the following steps: (1) collecting multimodal data in complex road scenes for preprocessing and labeling; (2) fusing the collected multimodal data through DIVFusion to obtain fused data; (3) performing exposure correction and noise reduction on the fused data; (4) improving the YOLO-MS detection network; (5) completing pedestrian detection using the trained improved YOLO-MS detection model; the method adopts the technology of visible light and infrared fusion, designs a semantic segmentation module, a noise reduction module and a lightweight detection module, and has a significant improvement in pedestrian detection in harsh environments, especially in dark environments and complex environments such as rain, snow and fog.

Description

Technical Field

[0001] The present invention belongs to the field of target detection, and in particular, relates to a lightweight pedestrian detection method in harsh environments based on deep learning. Background Art

[0002] With the rapid development and improvement of my country's transportation network, people are no longer separated by distance. However, many harsh environments still exist, such as darkness and complex environments like rain, snow, and fog. It is urgent to update and iterate intelligent pedestrian detection and prediction and emergency response systems based on on-board monitoring equipment, and intelligent pedestrian detection and alarm systems based on fixed embedded monitoring equipment at intersections. These systems can effectively reduce traffic accidents and congestion caused by these harsh environments. However, there are still many challenges to overcome in the practical application of lightweight pedestrian detection in harsh environments based on deep learning.

[0003] In terms of image preprocessing, a real-time detection method that integrates visual perception and adaptive defogging (China Patent Authorization Announcement No.: CN116343144B, Authorization Announcement Date: August 11, 2023) proposes an advanced image defogging method. However, in actual applications, due to complex environmental changes, there will be many interferences such as rain, fog, blur, and occlusion. Most of the time, single image defogging or rain removal cannot meet the task of image cleaning, which is a temporary solution and not a fundamental solution. The infrared and visible light fusion images used in this patent can provide prominent important targets even in extremely harsh environments, greatly reducing unnecessary image preprocessing.

[0004] Regarding exposure correction, Andrey et al. (Ignatov Andrey, Kobyshev Nikolay, Timofte Radu, et al. Dslr-quality Photos on Mobile Devices with Deep Convolutional Networks [C] / / Proceedings of the IEEE International Conference on Computer Vision. 2017: 3277-3285.) used a conditional generative adversarial network model consisting of a complex stack of structures such as group deep attention blocks and spatial attention blocks to perform exposure correction. This model not only has high computational overhead and poor real-time performance, but also has poor ability to integrate low-dimensional and high-dimensional information. The improved MSEC used in this patent effectively avoids the above problems. While having low computational overhead, it also has a high ability to integrate low-dimensional and high-dimensional information.

[0005] For real-time detection, Lyu et al. (Lyu Chengqi, Zhang Wenwei, Huang Haian, et al. Rtmdet: An Empirical Study of Designing Real-time Object Detectors [J]. arxiv Preprint arxiv: 2212.07784, 2022.) invented an RTMDet detector. The model uses large-kernel depth convolution and additional point-wise convolution in the basic building blocks of the trunk and neck. Although this enhances the model's ability to capture global context, it increases the model depth and slows down the reasoning speed, which has a large negative impact on real-time detection based on embedded and mobile devices. Therefore, this patent improves the multi-scale YOLO-MS detection model that also has large-kernel convolution to improve the calculation speed.

[0006] In summary, although real-time detection of pedestrian positions in harsh environments based on deep learning has made some progress, it still needs to be improved to better meet the high-precision and high-speed requirements of intelligent real-time detection of pedestrian positions in harsh environments for embedded devices and mobile devices. To this end, this patent proposes a lightweight pedestrian detection method in harsh environments based on deep learning, and is committed to deploying a lightweight, high-performance mobile pedestrian detection device to reduce traffic accidents and traffic congestion caused by problems such as bad weather. Summary of the Invention

[0007] The present invention aims to overcome the deficiencies in the above-mentioned prior art and designs a lightweight pedestrian detection method in harsh environments based on deep learning to improve the efficiency and accuracy of pedestrian detection in traffic systems and vehicle-mounted systems in harsh environments.

[0008] To achieve the above invention, the present invention provides a lightweight pedestrian detection method in harsh environments based on deep learning, which specifically includes the following steps:

[0009] (1) Collect multimodal data in complex road scenes for preprocessing and annotation;

[0010] (2) Fusing the collected multimodal data through DIVFusion to obtain fused data;

[0011] (3) Exposure correction and noise reduction of the fused data;

[0012] (4) Improve the YOLO-MS detection network;

[0013] (5) Use the trained improved YOLO-MS detection model to complete pedestrian detection;

[0014] The preprocessing and annotation of the multimodal fixed-viewpoint dataset under complex road scenes refers to scaling all video images to 608×608 pixels using a bilinear interpolation algorithm. 2 , where the ratio of the training set to the test set is 8:2; the fusion of the collected multimodal data set by DIVFusion refers to the use of the DIVFusion fusion algorithm to perform 1:1 DIVFusion fusion on multiple visible light images and infrared images collected in complex road scenes, reducing data redundancy and generating high-quality images with significant contrast and rich textures; the trained improved YOLO-MS detection model refers to the training using the Adam optimizer, and the parameters during training are set as follows: the weights of the convolutional layer are set to Among them, Conv is the number of convolution kernels, Channel is the number of channels, the bias value is 0, the weight of the linear layer is 0.01, the bias value is 0, epochs is set to 300, the training data size of each batch is 32, the learning rate is 0.0001, the learning rate decay factor is 0.8, and the final detection model is obtained after the improved YOLO-MS network training.

[0015] Furthermore, the exposure correction and noise reduction of the fused data mentioned above refers to the fact that the DIVFusion fusion method cannot effectively eliminate the overexposure effect of the headlight area in complex road scenes. The improved MSEC is used for exposure correction and the IDN is introduced for noise reduction to optimize image quality.

[0016] The improved MSEC is to address the color imbalance problem caused by the lack of semantic information by designing a semantic segmentation network and fusing it with the MSEC Laplacian pyramid multi-layer structure in a layer-by-layer manner to solve the above problem;

[0017] The semantic segmentation network described in the present invention is designed to design a lightweight semantic segmentation network with strided connections, which consists of an initialization module, Block1, Block2, Block3, CARAFE residual, Block4, CARAFE residual, and a 1×1 fully connected layer. Block1 and Block2 are composed of an asymmetric convolutional residual structure with a pre-1×1 convolution and channel shuffling. Block3 is similar to Block1 in structure but without the asymmetric convolution. CARAFE is a general lightweight upsampling operator.

[0018] The layer-by-layer fusion refers to fusing the image visual features and image semantic features of each layer of the Laplacian pyramid by multiplying the tensors pixel by pixel to improve the image exposure restoration effect. The specific fusion method is as follows:

[0019]

[0020] Among them, F is the fusion result, H and W are the image width and height, is the image visual feature F x The pixels in the i-th and j-th rows of is the image semantic feature F y The pixels in the i-th and j-th rows of is pixel-by-pixel multiplication, and sigmoid is the activation function.

[0021] Furthermore, the improved YOLO-MS detection network refers to improving the YOLO-MS backbone and neck networks and removing the attention mechanism module of the YOLO-MS detection head;

[0022] The improved YOLO-MS backbone and neck network refers to the introduction of 3×3 small-core convolution blocks, asymmetric convolution layers, and residual blocks in the YOLO-MS backbone to improve the reverse bottleneck layers in the third 7×7 large core and the fourth 9×9 large core of the original MS-Block in the backbone. At the same time, the MS-Block branch structure is adjusted, and the neck network in YOLO-MS is replaced with a synthetic fusion pyramid network, thereby optimizing the network structure and alleviating the scale differences between the original feature maps.

[0023] The aforementioned YOLO-MS detection head attention mechanism module removes the SE attention mechanism of the YOLO-MS detection head, thereby significantly improving the speed at a relatively small loss of accuracy and enhancing the real-time performance of pedestrian detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of an embodiment of the present invention;

[0025] Figure 2 This is a structural diagram of an embodiment of the present invention;

[0026] Figure 3 A lightweight semantic segmentation network designed for embodiments of the present invention;

[0027] Figure 4 This is a network example diagram of the improved MSEC according to an embodiment of the present invention;

[0028] Figure 5 Schematic diagram of the improved YOLO-MS structure according to an embodiment of the present invention;

[0029] Figure 6 This is an example diagram of the original MS-Block, improved MS-Block, ConvModule, and SPPBlock structures in the improved YOLO-MS according to an embodiment of the present invention;

[0030] Figure 7This is an example diagram of the SFPN and SFM structures in the improved YOLO-MS according to an embodiment of the present invention; DETAILED DESCRIPTION

[0031] The technical approach of the present invention is further described in detail below through specific embodiments and drawings.

[0032] like Figures 1 to 7 As shown, an embodiment of the light-weight pedestrian detection method in harsh environments based on deep learning of the present invention includes the following steps:

[0033] 1. Collect multimodal datasets in complex road scenes for preprocessing and annotation

[0034] The infrared and visible light fixed-viewing angle image datasets in complex road scenes are collected at a 1:1 ratio, and all video images are scaled to 608×608 pixels using a bilinear interpolation algorithm. 2 The ratio of the training set to the test set is 8:2. Pedestrians are semi-automatically labeled using PPOCRLabel, and LabelImg is used to manually label parts that are not ideal.

[0035] 2. Use DIVFusion to fuse the collected multimodal data sets to obtain fused data

[0036] For the fusion of multimodal data sets, this patent adopts the fusion method mentioned by DIVFusion, which consists of SIDNet (Scene-Illumination Disentangled Network) and TCEFNet (Texture-Contrast Enhancement Fusion Network), which are used to generate features to enhance lighting effects and improve contrast and texture information respectively.

[0037] Specifically, SIDNet consists of an encoder, three parallel SE blocks, and three parallel decoders. The encoder includes four convolutional layers with a kernel size of 3×3 and an activation function of LRelu. The SE block includes a maximum pooling layer, two fully connected layers, an LRelu layer, a fully connected layer, and an S-shaped layer to enhance complementary information and eliminate degraded lighting. The last fully connected layer outputs a vector that is multiplied with the original decoder output to achieve the effect of attention. The decoder consists of four 3×3 convolutional layers, which decode the previously extracted features and restore them to the corresponding image. Except for the last 3×3 convolutional layer of the decoder, which corresponds to the activation function Tanh, the activation function of all other layers is LRelu. Tanh is used to map variables to (-1, 1) to prevent gradient explosion, and LRelu is used for hidden layer neuron output to reduce gradient disappearance.

[0038] TCEFNet consists of two modules: the Gradient Preserving Module (GRM) and the Contrast Enhancement Module (CEM). The GRM consists of a main stream and two residual streams. The main stream deploys two general 3×3 convolutional layers with LReLU and one 1×1 convolutional layer. The first residual stream integrates the Sobel operator to preserve the strong texture of the features and a 1×1 convolutional layer to eliminate differences in the channel dimension. The second residual stream uses the Laplacian operator to further extract the weak texture of the features. The results of the GRM are input to convolutional layers with kernel sizes of 1×1, 3×3, 5×5, and 7×7, respectively. Different convolution operations are each equipped with a pooling layer to reduce information loss. The four obtained features are then concatenated along the channel dimension and sent to the contrast block. The contrast block consists of a main stream and a residual stream. The residual stream includes a max pooling layer, a fully connected layer with LReLU, and a fully connected layer with Sigmoid. The final output of the residual stream is an activation vector representing the contrast weight of the feature. Subsequently, the features in the main stream and the activation vector of the residual stream are multiplied by channel-wise multiplication to generate the final enhanced features.

[0039] The decoder then reconstructs the fused image based on the obtained features and corrects the image color multiple times through the color consistency loss function to obtain the final infrared and visible light fusion image.

[0040] 3. Exposure correction and noise reduction of fused data

[0041] For pedestrian detection and prediction tasks, although DIVFusion fits the fusion image of infrared and visible light well, its only drawback is that it does not eliminate the overexposure effect of the vehicle headlight area in complex road scenes. Therefore, this patent uses an improved MSEC for exposure correction to optimize image quality.

[0042] MSEC consists of three main components: a Laplacian pyramid, a U-Net strided connection, and a loss function. The correction network uses a multi-branch strategy to process input images of different frequency bands, thereby sequentially enhancing color and detail information. The entire network is differentiable and capable of end-to-end training. The weight distribution of each subnetwork is adjusted based on the contribution of the subproblems—global color correction and detail enhancement—to the final result.

[0043] Let X represent a Laplacian pyramid with n levels, such that X(I) is the I-th level of X, X(1) captures the highest frequency information, X(n) captures the lowest frequency information, and so on for the intermediate levels. The low-frequency level contains global color information, while the medium and high-frequency levels contain the detail information of the image. These levels are used to reconstruct the full color image Y. In the Laplacian pyramid reconstruction process, each level of the Laplacian pyramid is corrected to enhance the details in turn, thereby improving the final image. For each level of the corrected Laplacian pyramid, a sub-network of the U-Net architecture is used for processing, and each sub-network has different weights to better handle global color correction and detail enhancement. The steps are as follows:

[0044] (a) The input image is passed through a U-Net sub-network for global color correction, processing low-frequency information, and generating an upsampled image Y(n).

[0045] (b) Add the intermediate frequency information X(n-1) of the previous level to Y(n) to obtain a residual layer.

[0046] (c) Add the residual layer to Y(n-1) to obtain the reconstructed image Y(n-1), which is equivalent to the corresponding level of the Gaussian pyramid.

[0047] Repeat the above steps until the final output image Y. Each U-Net subnetwork includes an encoder and a decoder part, which consists of a series of 2×2 transposed convolution layers, 3×3 convolution layers (stride=1, padding=1), 2×2 maximum pooling layers (stride=2), 1×1 convolution layers (stride=1, padding=1), activation function LRelu layers and skip connections, which are respectively used to gradually reduce the spatial resolution of the input image and extract high-level features and gradually restore the spatial resolution of the image and generate the final output. Since the specific output channel size, convolution kernel size and step size of the encoder and decoder are related to level I of the Laplacian pyramid, this patent does not describe their specific contents in detail.

[0048] In each U-Net subnetwork, there are skip connections between the encoder and the decoder. These skip connections connect the feature maps in the encoder with the corresponding feature maps in the decoder to help transfer low-level and high-level feature information, which helps improve the network's information transfer and feature reuse capabilities.

[0049] In addition, MSEC also uses a multi-level loss function to guide the training of each sub-network, including reconstruction loss (L rec ), pyramid loss (L pyr ) and adversarial loss (L adv); the reconstruction loss is used to measure the difference between the reconstructed image and the reference image, the pyramid loss is used to guide each sub-network to be trained according to the reconstruction process of the Laplacian pyramid, and the adversarial loss is used to improve the realism of the reconstructed image.

[0050] The total loss function formula is as follows:

[0051] L=L rec +L pyr +L adv (2)

[0052]

[0053]

[0054] L adv =3hwnlog(S(D(Y))) (5)

[0055] Among them, L rec represents the reconstruction loss, L pyr represents the pyramid loss, L adv represents the adversarial loss, h and w represent the height and width of the training image, n is the number of levels of the pyramid, p is the index of each pixel in the corrected image Y and the corresponding properly exposed reference image T, and h l and w l They are the lth in the Laplacian pyramid of the training image th The height and width of the level are twice, S is the sigmoid function, and D is the discriminator DNN.

[0056] Specifically, to improve MSEC, a lightweight semantic segmentation network is designed, and MSEC is constrained through the semantic segmentation network to make the extracted semantic information more robust. At the same time, when the input image has extremely dark areas, MSEC will inevitably introduce noise. For this reason, a lightweight reversible network, namely InvDN, is introduced to denoise the output results of MSEC. InvDN uses the same parameters for encoding and decoding, and has the characteristics of light model and information losslessness.

[0057] The structure and function of the semantic segmentation network are as follows:

[0058] like Figure 3 As shown in the figure, the lightweight semantic segmentation network first extracts shallow features through a 3×3 convolution with a stride of 2 and a maximum pooling initialization structure. This initialization structure effectively reduces the number of channels. Then, feature maps of 1 / 4, 1 / 8, and 1 / 16 of the original image are extracted in order from shallow to deep through Block1, Block2, and Block3.

[0059] For Block1, a large 5×5 convolution kernel is used to extract shallow features, and a 1×1 convolution is placed in front of the 5×5 large convolution kernel for information flow and dimensionality reduction between channels. At the same time, channel shuffling is added to effectively reduce the computational complexity of the 1×1 convolution. The BN (Batchnormalization) normalization, LRelu activation function and maximum pooling layer are combined, and an asymmetric convolution is performed instead of the 5×5 convolution. The asymmetric convolution consists of a 1×5 convolution and a 5×1 convolution. While replacing the 5×5 convolution to reduce the number of parameters, it also increases the receptive field. In addition, a residual branch is set here to maintain the network weight information transmission and update while the number of layers is deepened; for Block2, the structure is similar to Block1, but the convolution kernel size is changed from 5 to 3 to extract deeper features; for Block3, it is also similar to Block1, except that the convolution kernel size of Block1 is changed from 5 to 1, and since the convolution kernel size is 1, there is no need to use asymmetric convolution to replace 1×1 convolution, which also removes the asymmetric convolution and branch operations of Block3.

[0060] Then, upsampling is performed through CARAFE. Common upsampling operations such as nearest neighbor interpolation sampling and bilinear interpolation sampling do not take into account the relationship between pixels and only copy the pixel values ​​of the nearest neighbors for sampling. This may lead to the loss of image details and texture information during the upsampling process, and also produce obvious jagged artifacts when magnifying. When bilinear interpolation sampling is used for four adjacent pixels, the resulting surface is consistent in the neighborhood, but the slope is not consistent, and the smoothing effect of bilinear grayscale interpolation will cause the image details to degrade. CARAFE has the characteristics of a large receptive field, upsampling based on input content, and lightweight, which effectively avoids the above problems. Specifically, CARAFE is divided into two main modules: an upsampling kernel prediction module and a feature reconstruction module. The upsampling ratio is σ. For an input feature map of size H×W×C, the upsampling kernel prediction module is used to predict the upsampling kernel, and then the feature reconstruction module is used to complete the upsampling, obtaining an output feature map of size σH×σW×C.

[0061] Specifically, the upsampling kernel prediction module includes the following steps:

[0062] (1) Feature map channel compression: For an input feature map of size H×W×C, a 1×1 convolution is used to compress its number of channels to C. m ,The purpose is to reduce the computational effort of subsequent steps;

[0063] (2) Content encoding and upsampling kernel prediction: After the input feature map is compressed in (1), a k encoder ×kencoder The convolution layer is used to predict the upsampling kernel, and the number of input channels is C m , the number of output channels is Expand the channel dimension in the space dimension and get the shape Upsampling kernel of ;

[0064] (3) Upsampling kernel normalization: The upsampling kernel obtained in (2) is normalized using Softmax so that the sum of the convolution kernel weights is 1.

[0065] Specifically, the feature recombination module includes the following steps:

[0066] For each position in the output feature map, map it back to the input feature map and take out the k positions centered at that feature map. up ×k up The area of ​​​​the point is dot-producted with the predicted upsampling kernel of the point to get the output value, and different channels at the same position share the same upsampling kernel.

[0067] The output of CARAFE is then multiplied by the input channel by channel to enhance feature information and adjust channel correlation. After a 3×3 convolution, BN normalization, and Tanh activation function, unlike the previous LRelu activation function, the Tanh activation function is used here because it has a better gradient descent optimization effect for deep networks. The output result is divided into two branches, which are respectively connected to the channel attention part and the spatial attention part in Block4. At the same time, the output of Block2 is also added to this branch for jump connection, combining low-level spatial information with high-level semantic information to achieve high-quality segmentation.

[0068] For Block4, channel attention and spatial attention branches are introduced. The initial branch weights are set to 0.5 and 0.5 respectively. Backpropagation is used to dynamically optimize and update the weight values ​​during training. This branch helps improve the model's generalization ability, robustness, and ability to extract local context information. First, global average pooling is used to reduce the resolution of the input feature map to 1, and 1×1 convolution is implemented using BN normalization and Sigmoid activation function to adjust the value of the feature map. Then, it is multiplied with the input feature map to generate the output of the channel attention branch; for the spatial attention branch of Block4, 1×1 convolution of BN and ReLU is first used to reduce the number of channels of the input feature map. Then, maximum pooling and average pooling are used to generate two feature maps. Both feature maps have only one channel with the same resolution as the input feature map. They are concatenated and convolution of BN and Sigmoid is used to reduce the channel to one. The spatial attention mask is generated and multiplied with the input feature map to generate the output of the spatial attention branch. Finally, the outputs of the two branches are merged and further upsampled through 3×3 convolution, BN normalization, Tanh activation function and CARAFE upsampling operator with residual structure, so as to map the features back to the semantic segmentation result close to the original resolution. The 1×1 convolution replaces the fully connected layer to greatly reduce the amount of calculation and obtain the final semantic segmentation result.

[0069] The total loss function used by the semantic segmentation network is as follows:

[0070] L Main =αL GDL +βL WCE (6)

[0071] L GDL Refers to the variant of Generalized Dice loss, which is a region-based loss function. WCE It refers to a variation of Weighted Cross Entropy, a distribution-based loss function that combines the exponential and logarithmic transformations of the two to better balance category imbalance and capture details. The initial values ​​of α and β are 0.3 and 0.7, and the weight values ​​are dynamically optimized and updated through backpropagation during training.

[0072] Specifically, L GDL Formula and L WCE The formula is as follows:

[0073]

[0074] Among them, w l The weight of each category, r ln is the reference value of category l at the nth pixel, p lnis the corresponding predicted probability value, The nonlinearity used to improve the function, the initial value is set to 1,

[0075]

[0076] Among them, M represents the number of categories, and yc has only two values ​​0 and 1. If the category is the same as the sample category, it takes 1, otherwise it takes 0. is the weight parameter, N represents the total number of pixels, N c Indicates the number of pixels of the standard value category c, p c represents the probability that the predicted sample belongs to category c, The nonlinearity used for the boosting function is initially set to 1.

[0077] Finally, the results of the semantic segmentation network are gradually fused with the Laplacian multi-level features of the main network. Specifically, the image visual features of each layer of the Laplacian pyramid are fused with the image semantic features by multiplying the tensors pixel by pixel to improve the image exposure repair effect. The formula is as follows:

[0078]

[0079] Among them, F is the fusion result, H and W are the image width and height, is the image visual feature F x The pixels in the i-th and j-th rows of is the image semantic feature F y The pixels in the i-th and j-th rows of is pixel-by-pixel multiplication, and sigmoid is the activation function.

[0080] like Figure 4 As shown in Figure 3, semantic information is hierarchically introduced into the main network, enabling the network to better understand the semantic content of the image, thereby enhancing the exposure correction effect.

[0081] 4. Improve the YOLO-MS detection network

[0082] In order to be able to deploy and detect pedestrians in real time on mobile devices, this patent adopts the more advanced lightweight model YOLO-MS. The main parts of YOLO-MS are MS-Block and heterogeneous Kernel selection protocol. This structure is different from the previous real-time object detector that uses the same kernel size convolution mode in different encoder stages. YOLO-MS uses heterogeneous convolution at different stages to help capture richer multi-scale features; however, in the MS-Block in the heterogeneous Kernel selection protocol of YOLO-MS, there are 7x7 and 9x9 large kernel convolutions. Although large kernel convolution obtains a larger receptive field and better Global features, but the computing performance has a great impact, especially for real-time detection. Therefore, this patent optimizes the internal structure of MS-Block in YOLO-MS. At the same time, considering that steps 2 and 3 of this patent have done a lot of preprocessing for the input image of YOLO-MS, such as exposure correction and noise reduction, the SE attention mechanism after the 1x1 convolution of the network end detection head is removed, and the Neck structure of the original YOLO-MS is also improved. SFPN is used instead of the original Neck structure. SFPN creates a synthetic layer scaling prediction map between the original FPN layers to reduce the scale truncation problem and smooth feature fusion.

[0083] Specifically, if Figure 5 As shown, Figure 5 The bold module is in Figure 6 and Figure 7 For specific display, the size is 608×608 pixels 2 The image is input to Backbone and enters Stage 1. The original MS-Block with a convolution kernel size k of 3 is used. For MS-Block, let X∈R H×W×C Refers to the input features. After the conversion through 1×1 convolution, the channel dimension of X increases to n×C. Then, X is decomposed into n different groups, represented as {X i}, where i∈1, 2, 3, ..., n, in order to reduce the computational cost, n is 3. Each group goes through a reverse bottleneck layer, which consists of 1×1 convolution, k×k depthwise convolution, 1×1 convolution and partial information of the previous group, using IB k×k (Y i-1 +X i ), where k represents the kernel size, to obtain Y i , Y i The mathematical representation of is described as:

[0084]

[0085] In particular, this patent takes into account the existence of 7x7 and 9x9 large kernel convolutions in the MS-Block of YOLO-MS. Although the large kernel convolution obtains a larger receptive field and better global features, it has a greater impact on the computing performance, especially for real-time detection. Therefore, this patent optimizes the internal structure of the MS-Block in YOLO-MS. Specifically, Figure 6 As shown, for X1, a small reverse bottleneck block consisting of 1×1 convolution, 3×3 depth convolution and 1×1 convolution is added to perform preliminary feature extraction with a smaller amount of computation, thereby enhancing the feature extraction capability of the large kernels of X2 and X3; for X2, the input feature value of the original k×k depth convolution is reduced by half and passed through a Res Unit block composed of X residual components, and the other half is directly connected to the feature after the k×k depth convolution after the 1×1 convolution, thereby reducing the amount of computation; for X3, the k×k convolution is changed to an asymmetric convolution consisting of 1×k convolution and k×1 convolution. This asymmetric convolution not only replaces the k×k convolution to reduce the number of parameters, but also increases the receptive field. At the same time, the input feature at X2 is reduced by half, and the other half does not pass through the asymmetric convolution with the convolution kernel as k, and then the two parts of the feature are connected; in particular, this patent adds a branch of MS-Block, such as Figure 6 As shown, for X i The entrance to establish a connection with the subsequent X j , where i∈1,2,3,...,n and j∈i+1,i+2,i+3,...,n, effectively preserving the information of the previous layer.

[0086] At the end of the reverse bottleneck layer, a 1×1 convolution is applied to interact between all the segments. Each segment encodes features of different scales. When the network goes deeper, this 1×1 convolution also has the effect of adjusting the number of channels. Finally, a channel shuffling operation is performed for information flow and dimensionality reduction between channels. All the segments are connected and passed through a convolution block with a convolution kernel size of 3×3 and a stride of 2 to play the role of downsampling, reduce computational complexity, reduce noise and improve robustness.

[0087] The same is true for Stage2, Stage3 and Stage4. The difference is that at Stage3, an SPPBlock is added, such as Figure 5As shown, the network is fed with the original image without resizing it, and then the outputs of Stage2, Stage3, and Stage4 are connected to Neck. Considering that steps 2 and 3 of this patent have already done a lot of preprocessing on the input image of YOLO-MS, such as exposure correction and noise reduction, the Neck structure of the original YOLO-MS is improved, and SFPN is used instead of the original Neck structure. SFPN creates a synthetic layer scaling prediction map between the original FPN layers, thereby improving the accuracy and speed of the CNN backbone, reducing the complexity of the model, and making the Neck network more refined and accurate in processing features of different scales, reducing the scale truncation problem, and smoothing feature fusion. Specifically, as Figure 7 As shown in Figure 2, the addition of medium-scale feature maps makes the transition between different scales smoother, thereby improving the YOLO-MS target detection performance.

[0088] Specifically, the first is the smallest unit of SFPN, the synthetic fusion module (SFM), the architecture of SFM is as follows Figure 7 As shown, it is used to generate various synthetic layers between the original layers in order to scale the predicted map from the original 1 / 2, 1 / 4, 1 / 8, etc. to 1 / 2, 1 / 3, 1 / 4, 1 / 6, etc. By adding the synthetic layer, the original layer can reduce the feature loss, so that the model can better adapt to objects of different sizes and generate more appropriate bounding boxes. SFM includes three optional input steps. The first is linear scaling input, then pixel by pixel addition, and finally fusion through 3x3 convolution; SFB is constructed by multiple SFMs, which divides the layer into two batches. First, the features are passed from the first batch of layers to the second batch of layers, and then from the second batch of layers to the first batch of layers. Similar to the FPN structure, it integrates the top-down and bottom-up directions in the same block, that is, the features are concentrated and then radiated out, and SFPN is a stacked structure of three SFBs.

[0089] Then the three outputs of Neck are connected to the Head part. Although the SE attention mechanism in the Head brings a certain improvement in accuracy, the speed is also reduced a lot. Therefore, the SE attention mechanism after the 1x1 convolution at the end of the network is removed. Specifically, Figure 5 As shown in , each output is divided into two parts. These six parts are all connected through the series structure of ConvModule, ConvModule and 1x1 and the channel connection is merged to get the final result. Although the ConvModule structures of these six parts are the same and the BN statistics are calculated independently, their convolution weight sharing is different. ConvModule A and ConvModule A are shared between different layers, and ConvModule B and ConvModuleB are shared. They do not affect each other. The structure of ConvModule is as follows Figure 6As shown; in particular, in order to distinguish the ConvModule of the Head part, this patent divides the ConvModule of the Head part into ConvModule A and ConvModule B, but the structure is essentially the same.

[0090] 5. Use the trained improved YOLO-MS detection model to complete pedestrian detection

[0091] The pedestrian detection model corresponding to the optimal parameters obtained through multiple rounds of loss function training optimization is used for real-time pedestrian detection. The detection results are then subjected to bounding box correction and threshold filtering. A classification confidence threshold is set and only prediction results above the threshold are retained, thereby reducing false detections, improving accuracy, and performing efficient and accurate pedestrian detection on the input image.

[0092] Specifically, the Adam optimizer is used for training with the following parameters: the weights of the convolutional layer are set to Among them, Conv is the number of convolution kernels, Channel is the number of channels, the bias value is 0, the weight of the linear layer is 0.01, the bias value is 0, epochs is set to 300, the training data size of each batch is 32, the learning rate is 0.0001, the learning rate decay factor is 0.8, and the final detection model is obtained after the improved YOLO-MS network training.

[0093] Compared with the prior art, the present invention has the following beneficial effects:

[0094] (1) This patent designs a new semantic segmentation network, which is a lightweight semantic segmentation network with multiple stride connections consisting of an initialization module, Block1, Block2, Block3, CARAFE residual, Block4, CARAFE residual and a 1×1 fully connected layer; this semantic segmentation network makes up for the shortcomings of MSEC in correcting image areas that lack semantic information.

[0095] (2) This patent improves the reverse bottleneck layer in the 7×7 and 9×9 large-core MS-Block in the original YOLO-MS, introduces 3×3 depth convolution, a residual structure with halved input channels, and asymmetric convolution, and adjusts the branch structure of MS-Block, thereby optimizing the upstream and downstream relationships of the network while maintaining accuracy, improving the calculation speed, and reducing parameters; at the same time, the neck network of YOLO-MS is improved to SFPN, reducing the scale truncation problem, smoothing feature fusion, and significantly improving the calculation speed.

[0096] (3) This patent provides a feasible and lightweight method for real-time detection of pedestrians in harsh environments. Because it adopts the technology of visible light and infrared fusion, and designs a semantic segmentation module, a noise reduction module and a lightweight detection module, it has a significant improvement in pedestrian detection in harsh environments, especially in dark environments and complex environments such as rain, snow, and fog, and provides a feasible method for reducing traffic accidents caused by blurred vision.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific implementation methods of the present invention can still be modified or replaced without departing from the spirit and scope of the present invention. Any modification or replacement should be included within the scope of protection of the claims of the present invention.

Claims

1. A lightweight pedestrian detection method in harsh environments based on deep learning, characterized by The multimodal data collected in complex road scenes are fused by the DIVFusion fusion algorithm, and the over-exposed parts in the data are removed by the improved MSEC (Learning Multi-Scale Photo Exposure Correction). Then, the multimodal pedestrian position detection is realized by the improved YOLO-MS, which includes the following steps: (1) Collect multimodal data in complex road scenes for preprocessing and annotation: All video images are scaled to 608×608 pixels by the bilinear interpolation algorithm. 2 , LabelImg is used to label pedestrian targets; (2) Fusing the collected multimodal data through DIVFusion to obtain fused data: using the DIVFusion fusion algorithm to perform 1:1 DIVFusion fusion of multiple visible light images and infrared images collected in complex road scenes; (3) performing exposure correction and noise reduction on the fused data; this refers to using an improved MSEC for exposure correction and introducing an IDN (Invertible Denoising Network) for noise reduction; the improved MSEC refers to a semantic segmentation network that is fused with a Laplacian pyramid multi-layer structure of MSEC in a layer-by-layer manner; the semantic segmentation network refers to designing a network consisting of an initialization module, Block1, Block2, Block3, and CARAFE (Content-Aware ReAssembly of FEatures) residual, Block4, CARAFE residual and 1×1 fully connected layer with stride connection lightweight semantic segmentation network; Among them, Block1 and Block2 are composed of asymmetric convolution residual structure with pre-1×1 convolution and channel shuffling, Block3 removes asymmetric convolution compared with Block1 structure, CARAFE is a universal lightweight upsampling operator; further, the loss function of the semantic segmentation network is reconstructed , calculated according to formula (1), ， in, and The initial values ​​of are 0.3 and 0.7, and the weight values ​​are dynamically optimized and updated through back propagation during training; For each category weight, For category In the n The baseline value of pixels (Ground True), is the corresponding predicted probability value, The initial value is set to 1; M represents the number of categories, There are only two values ​​0 and 1. If the predicted category is the same as the sample category, it takes 1, otherwise it takes 0. is the weight parameter, N Indicates the total number of pixels, N c Indicates the benchmark value category c The number of pixels, Indicates that the predicted sample belongs to the category c The probability of The initial value is set to 1, and the , to improve the nonlinearity of the loss function; the layer-by-layer fusion refers to fusing the image visual features and image semantic features of each layer of the Laplacian pyramid by multiplying the tensor pixels one by one. The specific fusion method is calculated according to formula (2). ， in, F For the fusion result, H and W are the image width and height, is the image visual feature No. i Row and j row pixels, is the image semantic feature No. i Row and j row pixels, is pixel-by-pixel multiplication, is the activation function; (4) improve the YOLO-MS detection network; (5) use the trained improved YOLO-MS detection model to complete pedestrian detection.

2. The method for detecting pedestrians in harsh environments based on deep learning according to claim 1, characterized in that: The improved YOLO-MS detection network refers to improving the trunk and neck networks of YOLO-MS and removing the attention mechanism module of the YOLO-MS detection head; the improved trunk and neck networks of YOLO-MS refer to introducing 3×3 small-core convolution blocks, asymmetric convolution layers, and residual blocks in the YOLO-MS trunk to improve the third 7×7 large core and the fourth 9×9 large core in the original MS-Block in the trunk, while adjusting the MS-Block branch structure and replacing the neck network in YOLO-MS with a synthetic fusion pyramid network SFPN (Synthetic Feature Pyramid Networks); the removal of the attention mechanism module of the YOLO-MS detection head refers to removing the SE attention mechanism of the YOLO-MS detection head.

Images