Lightweight severe environment pedestrian detection method based on deep learning
Through multimodal data fusion and improvement of YOLO-MS detection network, the problem of large calculation overhead and poor real-time performance of pedestrian detection in harsh environments is solved, and high-precision and high-speed pedestrian detection is realized, suitable for embedded devices and mobile devices , reducing the risk of traffic accidents.
Patent Information
- Application Number
- CN202411062312.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-05
AI Technical Summary
In actual application of existing pedestrian detection technology based on lightweight harsh environments based on deep learning, there are problems such as complex image preprocessing, large computing overhead, poor real-time performance and insufficient integration capabilities of low-dimensional and high-dimensional information, which is difficult to meet the embedded devices and High precision and high speed requirements for mobile devices in harsh environments.
Using multimodal data fusion, improved YOLO-MS detection network, fusion of infrared and visible light data through DIVFusion, designed improved MSEC for exposure correction and noise reduction, introduced lightweight semantic segmentation network and SFPN structure, and removed YOLO-MS detection head The attention mechanism module is used to optimize the network structure, improve computing speed and detection accuracy.
It realizes high-precision pedestrian detection in harsh environments, significantly reduces calculation overhead, improves detection speed and accuracy, and is suitable for embedded devices and mobile devices, reducing the risk of traffic accidents and congestion.
Smart Images

Figure CN118982843A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of target detection, and in particular, relates to a lightweight pedestrian detection method in harsh environments based on deep learning. Background Art
[0002] With the rapid development and improvement of my country's transportation network, people are no longer separated by distance. However, there are still many harsh environments such as dark environments and complex environments such as rain, snow, and fog. It is urgent to update and iterate the intelligent detection and prediction of pedestrians and emergency systems based on vehicle-mounted monitoring equipment, and the intelligent detection of pedestrians and alarm systems based on fixed embedded monitoring equipment at intersections. This can effectively reduce traffic accidents and traffic jams caused by harsh environments and other problems. However, there are still many problems to be overcome in the practical application of lightweight pedestrian detection in harsh environments based on deep learning.
[0003] Regarding image preprocessing, a real-time detection method that integrates visual perception and adaptive defogging (China Patent Authorization Announcement No.: CN116343144B, Authorization Announcement Date: August 11, 2023) proposes an advanced image defogging method. However, in actual applications, due to complex environmental changes, there will be many interferences such as rain, fog, blur, and occlusion. Most of the time, a single image defogging or rain removal cannot meet the task of image cleaning, which is a temporary solution and not a fundamental solution. The infrared and visible light fusion images used in this patent can provide prominent important targets even in extremely harsh environments, greatly reducing unnecessary image preprocessing.
[0004] Regarding exposure correction, Andrey et al. (Ignatov Andrey, Kobyshev Nikolay, Timofte Radu, et al. Dslr-quality Photos on Mobile Devices with Deep Convolutional Networks [C] / / Proceedings of the IEEE International Conference on Computer Vision. 2017: 3277-3285.) used a conditional generative adversarial network model composed of a complex stack of deep attention blocks and spatial attention blocks to perform exposure correction. This model not only has high computational overhead and poor real-time performance, but also has poor ability to integrate low-dimensional and high-dimensional information. The improved MSEC used in this patent effectively avoids the above problems. While having a low computational overhead, it also has a high ability to integrate low-dimensional and high-dimensional information.
[0005] For real-time detection, Lyu et al. (Lyu Chengqi, Zhang Wenwei, Huang Haian, et al. Rtmdet: An Empirical Study of Designing Real-time Object Detectors [J]. arxiv Preprint arxiv: 2212.07784, 2022.) invented an RTMDet detector. The model uses large-kernel deep convolutions and additional point-wise convolutions in the basic building blocks of the trunk and neck. Although this enhances the model's ability to capture global context, it increases the model depth and slows down the reasoning speed, which has a large negative impact on real-time detection based on embedded and mobile devices. Therefore, this patent improves the multi-scale YOLO-MS detection model that also has large-kernel convolution to increase the calculation speed.
[0006] In summary, although real-time detection of pedestrian positions in harsh environments based on deep learning has made some progress, it still needs to be improved to better meet the high-precision and high-speed requirements of intelligent real-time detection of pedestrian positions in harsh environments for embedded devices and mobile devices. To this end, this patent proposes a lightweight pedestrian detection method in harsh environments based on deep learning, and is committed to deploying a lightweight, high-performance mobile pedestrian detection device to reduce traffic accidents and traffic jams caused by problems such as bad weather. Summary of the invention
[0007] The present invention aims to overcome the deficiencies in the above-mentioned prior art and designs a lightweight pedestrian detection method in harsh environments based on deep learning to improve the efficiency and accuracy of pedestrian detection in traffic systems and vehicle-mounted systems in harsh environments.
[0008] In order to achieve the above invention, the present invention provides a lightweight pedestrian detection method in harsh environment based on deep learning, which specifically includes the following steps:
[0009] (1) Collect multimodal data in complex road scenes for preprocessing and annotation;
[0010] (2) Fusing the collected multimodal data through DIVFusion to obtain fused data;
[0011] (3) Perform exposure correction and noise reduction on the fused data;
[0012] (4) Improve the YOLO-MS detection network;
[0013] (5) Use the trained improved YOLO-MS detection model to complete pedestrian detection;
[0014] The preprocessing and annotation of the multimodal fixed-view dataset under complex road scenes refers to scaling all video images to 608×608 pixels using a bilinear interpolation algorithm. 2 , where the ratio of the training set to the test set is 8:2; the fusion of the collected multimodal data set by DIVFusion refers to the use of the DIVFusion fusion algorithm to perform 1:1 DIVFusion fusion of multiple visible light images and infrared images collected in complex road scenes to reduce data redundancy and generate high-quality images with significant contrast and rich texture; the trained improved YOLO-MS detection model refers to the training using the Adam optimizer, and the parameters during training are set as follows: the weight of the convolutional layer is set to Among them, Conv is the number of convolution kernels, Channel is the number of channels, the deviation value is 0, the weight of the linear layer is 0.01, the deviation value is 0, epochs is set to 300, the training data size of each batch is 32, the learning rate is 0.0001, the learning rate attenuation factor is 0.8, and the final detection model is obtained after the improved YOLO-MS network training.
[0015] Furthermore, the exposure correction and noise reduction of the fused data refers to the use of improved MSEC for exposure correction and introduction of IDN for noise reduction to optimize image quality, as the DIVFusion fusion method cannot effectively eliminate the overexposure effect of the headlight area in complex road scenes;
[0016] The improved MSEC refers to a semantic segmentation network designed to be integrated with the Laplacian pyramid multi-layer structure of MSEC in a layer-by-layer manner to solve the color imbalance problem caused by lack of semantic information;
[0017] The design of a semantic segmentation network refers to designing a lightweight semantic segmentation network with stride connection consisting of an initialization module, Block1, Block2, Block3, CARAFE residual, Block4, CARAFE residual and a 1×1 fully connected layer; wherein Block1 and Block2 are composed of an asymmetric convolution residual structure with a pre-1×1 convolution and channel shuffle, Block3 is similar to Block1 in structure but removes the asymmetric convolution, and CARAFE is a universal lightweight upsampling operator;
[0018] The layer-by-layer fusion refers to fusing the image visual features and image semantic features of each layer of the Laplacian pyramid by multiplying the tensor pixels one by one to improve the image exposure restoration effect. The specific fusion method is as follows:
[0019]
[0020] Among them, F is the fusion result, H and W are the image width and height, is the image visual feature F x The pixels in the i-th and j-th rows of is the image semantic feature F y The pixels in the i-th and j-th rows of is pixel-by-pixel multiplication, and sigmoid is the activation function.
[0021] Furthermore, the improved YOLO-MS detection network refers to improving the backbone and neck networks of YOLO-MS and removing the attention mechanism module of the YOLO-MS detection head;
[0022] The improved YOLO-MS backbone and neck network refers to introducing a 3×3 small-core convolution block, an asymmetric convolution layer, and a residual block in the YOLO-MS backbone to improve the reverse bottleneck layer in the third 7×7 large core and the fourth 9×9 large core original MS-Block in the backbone, adjusting the MS-Block branch structure, and replacing the neck network in YOLO-MS with a synthetic fusion pyramid network, thereby optimizing the network structure and alleviating the scale differences of different feature maps.
[0023] The module for removing the attention mechanism of the YOLO-MS detection head refers to removing the SE attention mechanism of the YOLO-MS detection head part, which exchanges a relatively small loss of accuracy for a significant speed improvement, thereby enhancing the real-time performance of pedestrian detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 is a flow chart of an embodiment of the present invention;
[0025] Figure 2 It is a structural diagram of an embodiment of the present invention;
[0026] Figure 3 A lightweight semantic segmentation network designed for embodiments of the present invention;
[0027] Figure 4 This is a network example diagram of the improved MSEC according to an embodiment of the present invention;
[0028] Figure 5 Schematic diagram of the improved YOLO-MS structure in an embodiment of the present invention;
[0029] Figure 6 This is an example diagram of the structure of the original MS-Block, improved MS-Block, ConvModule and SPPBlock in the improved YOLO-MS according to an embodiment of the present invention;
[0030] Figure 7This is an example diagram of the SFPN and SFM structures in the improved YOLO-MS according to an embodiment of the present invention; DETAILED DESCRIPTION
[0031] The technical route of the present invention is further described in detail below through specific embodiments and drawings.
[0032] like Figures 1 to 7 As shown, an embodiment of a lightweight pedestrian detection method in harsh environments based on deep learning of the present invention includes the following steps:
[0033] 1. Collect multimodal data sets in complex road scenes for preprocessing and annotation
[0034] The infrared and visible fixed-viewing angle image datasets in complex road scenes are collected at a 1:1 ratio, and all video images are scaled to 608×608 pixels using a bilinear interpolation algorithm. 2 The ratio of the training set to the test set is 8:2. Pedestrians are semi-automatically labeled using PPOCRLabel, and LabelImg is used to manually label parts that are not ideal.
[0035] 2. Use DIVFusion to fuse the collected multimodal data sets to obtain fused data
[0036] For the fusion of multimodal data sets, this patent adopts the fusion method mentioned by DIVFusion, which consists of SIDNet (Scene-Illumination Disentangled Network) and TCEFNet (Texture-Contrast Enhancement Fusion Network), which are used to generate features to enhance lighting effects and improve contrast and texture information respectively.
[0037] Specifically, SIDNet consists of an encoder, three parallel SE blocks, and three parallel decoders. The encoder includes four convolutional layers with a kernel size of 3×3 and an activation function of LRelu; for the SE block, it includes a maximum pooling layer, two fully connected layers, an LRelu layer, a fully connected layer, and an S-shaped layer, which are used to enhance complementary information and eliminate degraded lighting. The last fully connected layer outputs a vector, which is multiplied by the original decoder output result to achieve the effect of attention; the decoder consists of four 3×3 convolutional layers, which decode the previously extracted features and restore them to the corresponding image. Except for the activation function corresponding to the last 3×3 convolutional layer of the decoder, which is Tanh, the activation functions of other layers are LRelu. Tanh is used to map variables to (-1, 1) to prevent gradient explosion, and LRelu is used for hidden layer neuron output to reduce gradient disappearance.
[0038] TCEFNet consists of two modules, namely the gradient preservation module (GRM) and the contrast enhancement module (CEM); GRM contains a mainstream and two residual streams. The mainstream deploys two general 3×3 convolutional layers with LRelu and a 1×1 convolutional layer. The first residual stream integrates the Sobel operator to retain the strong texture of the feature and integrates the 1×1 convolutional layer to eliminate the difference in the channel dimension. The second residual stream uses the Laplacian operator to further extract the weak texture of the feature; the result of GRM is input to the convolutional layers with kernel sizes of 1×1, 3×3, 5×5, and 7×7, respectively. Different convolution operations are equipped with pooling layers to reduce information loss. Then, the obtained 4-part features are cascaded in the channel dimension and sent to the contrast block. The contrast block contains a mainstream and a residual stream. The residual stream contains a maximum pooling layer, a fully connected layer with LRelu, and a fully connected layer with Sigmoid. The final output of the residual stream is the activation vector representing the contrast weight of the feature. Subsequently, the features in the main stream and the activation vector of the residual stream are multiplied by channel-wise multiplication to generate the final enhanced features.
[0039] The decoder then reconstructs the fused image based on the obtained features and corrects the image color multiple times through the color consistency loss function to obtain the final infrared and visible light fused image.
[0040] 3. Exposure correction and noise reduction for fused data
[0041] For pedestrian detection and prediction tasks, although DIVFusion fits the fusion image of infrared and visible light well, its drawback is that it does not eliminate the over-exposure effect of the vehicle headlight area in complex road scenes. Therefore, this patent uses improved MSEC for exposure correction to optimize image quality.
[0042] MSEC mainly consists of three parts: Laplacian pyramid, U-Net stride connection, and loss function. The correction network uses a multi-branch strategy to process input images of different frequency bands, thereby enhancing color and detail information in sequence. At the same time, the entire network is differentiable and can be trained end-to-end. The weight distribution of each sub-network is adjusted according to the contribution of the sub-problems, namely global color correction and detail enhancement, to the final result.
[0043] Let X represent a Laplacian pyramid with n levels, such that X(I) is the Ith level of X, X(1) captures the highest frequency information, X(n) captures the lowest frequency information, and so on for the intermediate levels. The low-frequency level contains global color information, while the medium and high-frequency levels contain the detail information of the image. These levels are used to reconstruct the full color image Y. In the Laplacian pyramid reconstruction process, the details are enhanced in turn by correcting each level of the Laplacian pyramid, thereby improving the final image. For each level of the corrected Laplacian pyramid, a sub-network of the U-Net architecture is used for processing, and each sub-network has different weights to better handle global color correction and detail enhancement. The steps are as follows:
[0044] (a) The input image passes through a U-Net sub-network for global color correction, processes low-frequency information, and generates an upsampled image Y(n).
[0045] (b) Add the intermediate frequency information X(n-1) of the previous level to Y(n) to obtain a residual layer.
[0046] (c) Add the residual layer to Y(n-1) to obtain the reconstructed image Y(n-1), which is equivalent to the corresponding level of the Gaussian pyramid.
[0047] Repeat the above steps until the final output image Y. Each U-Net subnetwork includes an encoder and a decoder part, which consists of a series of 2×2 transposed convolution layers, 3×3 convolution layers (stride=1, padding=1), 2×2 maximum pooling layers (stride=2), 1×1 convolution layers (stride=1, padding=1), activation function LRelu layers and jump connections, which are used to gradually reduce the spatial resolution of the input image and extract high-level features and gradually restore the spatial resolution of the image and generate the final output. Since the specific output channel size, convolution kernel size and step size of the encoder and decoder are related to level I of the Laplace pyramid, this patent does not describe its specific content in detail.
[0048] In each U-Net subnetwork, there are skip connections between the encoder and the decoder. These skip connections connect the feature maps in the encoder with the corresponding feature maps in the decoder to help transfer low-level and high-level feature information, which helps improve the network's information transfer and feature reuse capabilities.
[0049] In addition, MSEC also uses a multi-level loss function to guide the training of each sub-network. The loss function includes reconstruction loss (L rec ), pyramid loss (L pyr ) and adversarial loss (L adv); the reconstruction loss is used to measure the difference between the reconstructed image and the reference image, the pyramid loss is used to guide each sub-network to be trained according to the reconstruction process of the Laplacian pyramid, and the adversarial loss is used to improve the realism of the reconstructed image.
[0050] The total loss function formula is as follows:
[0051] L=L rec +L pyr +L adv (2)
[0052]
[0053]
[0054] L adv =3hwnlog(S(D(Y))) (5)
[0055] Among them, L rec represents the reconstruction loss, L pyr represents the pyramid loss, L adv represents the adversarial loss, h and w represent the height and width of the training image, n is the number of levels of the pyramid, p is the index of each pixel in the corrected image Y and the corresponding properly exposed reference image T, and h l and w l They are the lth in the Laplacian pyramid of the training image. th The height and width of the level are twice, S is the sigmoid function, and D is the discriminator DNN.
[0056] Specifically, to improve MSEC, a lightweight semantic segmentation network is designed, and MSEC is constrained through the semantic segmentation network to make the extracted semantic information more robust. At the same time, when the input image has extremely dark areas, MSEC will inevitably introduce noise. For this reason, a lightweight reversible network, namely InvDN, is introduced to denoise the output results of MSEC. InvDN encoding and decoding use the same parameters, and has the characteristics of light model and information losslessness.
[0057] The structure and function of the semantic segmentation network are as follows:
[0058] like Figure 3 As shown in the figure, the lightweight semantic segmentation network first extracts shallow features through a 3×3 convolution with a step size of 2 and a maximum pooling initialization structure. This initialization structure effectively reduces the number of channels, and then extracts feature maps of 1 / 4, 1 / 8 and 1 / 16 of the original image in order from shallow to deep through Block1, Block2, and Block3.
[0059] For Block1, a large 5×5 convolution kernel is used to extract shallow features, and a 1×1 convolution is placed in front of the 5×5 large convolution kernel for information flow and dimensionality reduction between channels. At the same time, channel shuffling is added to effectively reduce the computational complexity of the 1×1 convolution. The BN (Batchnormalization) normalization, LRelu activation function and maximum pooling layer are combined, and an asymmetric convolution is used to replace the 5×5 convolution. The asymmetric convolution consists of a 1×5 convolution and a 5×1 convolution. While replacing the 5×5 convolution to reduce the number of parameters, it also increases the receptive field. In addition, a residual branch is set here to maintain the network weight information transmission and update while the number of layers is deepened; for Block2, the structure is similar to Block1, but the convolution kernel size is changed from 5 to 3 to extract deeper features; for Block3, it is also similar to Block1, the difference is that the convolution kernel size of Block1 is changed from 5 to 1, and because the convolution kernel size is 1, there is no need to use asymmetric convolution instead of 1×1 convolution, which also removes the asymmetric convolution and branch operations of Block3.
[0060] Then, upsampling operation is performed through CARAFE. Common upsampling operations such as nearest neighbor interpolation sampling method and bilinear interpolation sampling method. The nearest neighbor interpolation sampling method does not take into account the relationship between pixels, and only copies the pixel values of the nearest neighbors for sampling, which may lead to the loss of image details and texture information during the upsampling process. At the same time, obvious jagged artifacts will be generated when zooming in. When the bilinear interpolation sampling method uses bilinear interpolation for four adjacent pixels, the resulting surface is consistent in the neighborhood, but the slope is not consistent, and the smoothing effect of bilinear grayscale interpolation will cause the details of the image to degenerate. CARAFE has the characteristics of large receptive field, upsampling based on input content and lightweight, which effectively avoids the above problems. Specifically, CARAFE is divided into two main modules, namely, upsampling kernel prediction module and feature recombination module, with an upsampling ratio of σ. For the input feature map of size H×W×C, the upsampling kernel prediction module is used to predict the upsampling kernel, and then the feature recombination module is used to complete the upsampling to obtain an output feature map of size σH×σW×C.
[0061] Specifically, the upsampling kernel prediction module includes the following steps:
[0062] (1) Feature map channel compression: For an input feature map of size H×W×C, a 1×1 convolution is used to compress its number of channels to C. m ,The purpose is to reduce the amount of calculation in subsequent steps;
[0063] (2) Content encoding and upsampling kernel prediction: The compressed input feature map in (1) is used to generate a k encoder ×kencoder The convolution layer is used to predict the upsampling kernel, and the number of input channels is C m , the number of output channels is Expand the channel dimension in the space dimension, and the shape is The upsampling kernel of
[0064] (3) Upsampling kernel normalization: The upsampling kernel obtained in (2) is normalized using Softmax so that the sum of the convolution kernel weights is 1.
[0065] Specifically, the feature recombination module includes the following steps:
[0066] For each position in the output feature map, map it back to the input feature map and take the k positions centered at that feature map. up ×k up The area is dot-producted with the predicted upsampling kernel of the point to get the output value, and different channels at the same position share the same upsampling kernel.
[0067] The output of CARAFE is then multiplied by the input channel by channel to enhance feature information and adjust channel correlation. It then goes through a 3×3 convolution, BN normalization, and Tanh activation function. Unlike the previous LRelu activation function, the Tanh activation function is used here because it has a better gradient descent optimization effect for deep networks. The output result is divided into two branches that are connected to the channel attention part and the spatial attention part in Block4 respectively. At the same time, the output of Block2 is also added to this branch for jump connection, combining low-level spatial information with high-level semantic information to achieve high-quality segmentation.
[0068] For Block4, channel attention and spatial attention branches are introduced, and the initial branch weights are set to 0.5 and 0.5 respectively. Back propagation is used to dynamically optimize and update the weight values during training. This branch helps improve the model's generalization ability, robustness, and ability to extract local context information. First, the resolution of the input feature map is reduced to 1 using global average pooling, and 1×1 convolution is implemented using BN normalization and Sigmoid activation function to adjust the value of the feature map. Then, it is multiplied with the input feature map to generate the output of the channel attention branch; for the spatial attention branch of Block4, first use 1×1 convolution of BN and ReLU to reduce the number of channels of the input feature map, and then use maximum pooling and average pooling to generate two feature maps, and both feature maps have only one channel with the same resolution as the input feature map. Connect them and use BN and Sigmoid convolution to reduce the channel to one, generate a spatial attention mask and multiply it with the input feature map to generate the output of the spatial attention branch. Finally, the outputs of the two branches are merged and further upsampled through 3×3 convolution, BN normalization, Tanh activation function, and CARAFE upsampling operator with residual structure, so as to map the features back to semantic segmentation results close to the original resolution. The 1×1 convolution replaces the fully connected layer to greatly reduce the amount of calculation and obtain the final semantic segmentation result.
[0069] The total loss function used by the semantic segmentation network is as follows:
[0070] L Main =αL GDL +βL WCE (6)
[0071] L GDL Refers to the variant of Generalized Dice loss, which is a region-based loss function. WCE It refers to a variation of Weighted Cross Entropy, which is a distribution-based loss function. It combines the exponential and logarithmic transformations of the two to better balance category imbalance and capture details. The initial values of α and β are 0.3 and 0.7, and the weight values are dynamically optimized and updated through back-propagation during training.
[0072] Specifically, L GDL Formula and L WCE The formula is as follows:
[0073]
[0074] Among them, w l The weight of each category, r ln is the reference value of the nth pixel of category l, p lnis the corresponding predicted probability value, The nonlinearity used to improve the function, the initial value is set to 1,
[0075]
[0076] Among them, M represents the number of categories, and yc has only two values: 0 and 1. If the category is the same as the sample category, it takes 1, otherwise it takes 0. is the weight parameter, N represents the total number of pixels, N c represents the number of pixels of the standard value category c, p c represents the probability that the predicted sample belongs to category c, The nonlinearity used for the boosting function is initially set to 1.
[0077] Finally, the results of the semantic segmentation network are gradually fused with the Laplacian multi-level features of the main network. Specifically, the image visual features of each layer of the Laplacian pyramid are fused with the image semantic features by multiplying the tensors pixel by pixel to improve the image exposure restoration effect. The formula is as follows:
[0078]
[0079] Among them, F is the fusion result, H and W are the image width and height, is the image visual feature F x The pixels in the i-th and j-th rows of is the image semantic feature F y The pixels in the i-th and j-th rows of is pixel-by-pixel multiplication, and sigmoid is the activation function.
[0080] like Figure 4 As shown in Figure 3, semantic information is introduced into the main network in a hierarchical manner, so that the network can better understand the semantic content of the image, thereby enhancing the exposure correction effect.
[0081] 4. Improve the YOLO-MS detection network
[0082] In order to be able to deploy and detect pedestrians in real time on mobile devices, this patent adopts a relatively advanced lightweight model YOLO-MS. The main parts of YOLO-MS are MS-Block and heterogeneous Kernel selection protocol. This structure is different from the previous real-time object detector that uses convolutions with the same kernel size in different encoder stages. YOLO-MS uses heterogeneous convolutions at different stages to help capture richer multi-scale features; however, in the MS-Block in the heterogeneous Kernel selection protocol of YOLO-MS, there are 7x7 and 9x9 large kernel convolutions. Although large kernel convolutions obtain larger receptive fields and better Global features, but the computing performance has a great impact, especially for real-time detection. Therefore, this patent optimizes the internal structure of MS-Block in YOLO-MS. At the same time, considering that steps 2 and 3 of this patent have done a lot of preprocessing for the input image of YOLO-MS, such as exposure correction and noise reduction, the SE attention mechanism after the 1x1 convolution of the network end detection head is removed, and the Neck structure of the original YOLO-MS is also improved. SFPN is used instead of the original Neck structure. SFPN creates a synthetic layer scaling prediction map between the original FPN layers to reduce the scale truncation problem and smooth feature fusion.
[0083] Specifically, Figure 5 As shown, Figure 5 The bold module is Figure 6 and Figure 7 For specific display, the size is 608×608 pixels 2 The image is input to Backbone and enters Stage 1. The original MS-Block with a convolution kernel size k of 3 is used. For MS-Block, let X∈R H×W×C Refers to the input features. After the conversion through 1×1 convolution, the channel dimension of X increases to n×C. Then, X is decomposed into n different groups, represented as {X i}, where i∈1, 2, 3, ..., n, in order to reduce the computational cost, n is 3 here. Each group goes through a reverse bottleneck layer, which consists of a 1×1 convolution, a k×k depth convolution, and a 1×1 convolution with part of the information of the previous group, using IB k×k (Y i-1 +X i ), where k is the kernel size, to obtain Y i , Y i The mathematical representation of is described as:
[0084]
[0085] In particular, this patent takes into account that the MS-Block in YOLO-MS has 7x7 and 9x9 large kernel convolutions. Although the large kernel convolution obtains a larger receptive field and better global features, it has a greater impact on the computing performance, especially for real-time detection. Therefore, this patent optimizes the internal structure of the MS-Block in YOLO-MS. Specifically, Figure 6 As shown, for X1, a small reverse bottleneck block consisting of 1×1 convolution, 3×3 depth convolution and 1×1 convolution is added to perform preliminary feature extraction with a small amount of calculation, thereby enhancing the feature extraction capability of the large kernels of X2 and X3; for X2, the input feature value of the original k×k depth convolution is reduced by half and passes through a Res Unit block consisting of X residual components, and the other half is directly connected to the features after the k×k depth convolution after passing through the 1×1 convolution, thereby reducing the amount of calculation; for X3, the k×k convolution is changed to an asymmetric convolution consisting of 1×k convolution and k×1 convolution, which replaces the k×k convolution to reduce the number of parameters while also increasing the receptive field. At the same time, the input feature at X2 is reduced by half, and the other half does not pass through the asymmetric convolution with the convolution kernel being k, and then the two parts of the feature are connected; in particular, this patent adds branches of MS-Block, such as Figure 6 As shown, for X i The entrance to establish your own connection with the subsequent X j , where i∈1,2,3,…,n and j∈i+1,i+2,i+3,…,n, thereby effectively retaining the information of the previous layer.
[0086] At the end of the reverse bottleneck layer, a 1×1 convolution is applied to interact between all the segments. Each segment encodes features of different scales. When the network goes deeper, this 1×1 convolution also has the function of adjusting the number of channels. Finally, the channel shuffle operation is used for information flow and dimensionality reduction between channels, and all the segments are connected and passed through a convolution block with a convolution kernel size of 3×3 and a stride of 2 to play the role of downsampling, reduce computational complexity, reduce noise and improve robustness.
[0087] The same is true for Stage2, Stage3, and Stage4. The difference is that at Stage3, an SPPBlock is added, such as Figure 5As shown, the network is allowed to input the original image without resizing it, and then the outputs of Stage2, Stage3, and Stage4 are connected to Neck. Considering that steps 2 and 3 of this patent have done a lot of preprocessing for the input image of YOLO-MS, such as exposure correction and noise reduction, the Neck structure of the original YOLO-MS is improved, and SFPN is used to replace the original Neck structure. SFPN creates a synthetic layer scaling prediction map between the original FPN layers, thereby improving the accuracy and speed of the CNN backbone, reducing the complexity of the model, and making the Neck network more refined and accurate in processing features of different scales, reducing the scale truncation problem, and smoothing feature fusion. Specifically, as Figure 7 As shown in Figure 2, the addition of medium-scale feature maps makes the transition between different scales smoother, thereby improving the YOLO-MS target detection performance.
[0088] Specifically, the first is the smallest unit of SFPN, the synthetic fusion module (SFM). The architecture of SFM is as follows Figure 7 As shown, it is used to generate various synthetic layers between the original layers so as to scale the predicted images from the original 1 / 2, 1 / 4, 1 / 8, etc. to 1 / 2, 1 / 3, 1 / 4, 1 / 6, etc. By adding the synthetic layer, the original layer can reduce the feature loss, so that the model can better adapt to objects of different sizes and generate more appropriate bounding boxes. SFM includes three optional input steps. The first is linear scaling input, then adding pixel by pixel, and finally fusing through 3x3 convolution; SFB is constructed by multiple SFMs, which divides the layers into two batches. First, the features are passed from the first batch of layers to the second batch of layers, and then from the second batch of layers to the first batch of layers. Similar to the FPN structure, it integrates the top-down and bottom-up directions in the same block, that is, the features are concentrated and then radiated, and SFPN is a stacked structure of three SFBs.
[0089] Then the three outputs of Neck are connected to the Head part. Although the SE attention mechanism in the Head brings a certain improvement in accuracy, the speed is also reduced a lot, so the SE attention mechanism after the 1x1 convolution at the end of the network is removed. Specifically, Figure 5 As shown in the figure, each output is divided into two parts. These six parts are connected through the series structure of ConvModule, ConvModule and 1x1 and the channel connection is merged to get the final result. Although the ConvModule structures of these six parts are the same and the BN statistics are calculated independently, their convolution weight sharing is different. ConvModule A and ConvModule A are shared between different layers, and ConvModule B and ConvModule B are shared. They do not affect each other. The structure of ConvModule is as follows: Figure 6As shown; in particular, in order to distinguish the ConvModule of the Head part, this patent divides the ConvModule of the Head part into ConvModule A and ConvModule B, but the structure is essentially the same.
[0090] 5. Use the trained improved YOLO-MS detection model to complete pedestrian detection
[0091] The pedestrian detection model corresponding to the optimal parameters obtained through multiple rounds of training and optimization of the loss function is used for real-time pedestrian detection, and the detection results are corrected by bounding boxes and threshold filtered. A classification confidence threshold is set, and only the prediction results above the threshold are retained, thereby reducing false detections, improving accuracy, and performing efficient and accurate pedestrian detection on the input image.
[0092] Specifically, the Adam optimizer is used for training with the following parameters: The weights of the convolutional layer are set to Among them, Conv is the number of convolution kernels, Channel is the number of channels, the deviation value is 0, the weight of the linear layer is 0.01, the deviation value is 0, epochs is set to 300, the training data size of each batch is 32, the learning rate is 0.0001, the learning rate attenuation factor is 0.8, and the final detection model is obtained after the improved YOLO-MS network training.
[0093] Compared with the prior art, the present invention has the following beneficial effects:
[0094] (1) This patent designs a new semantic segmentation network, which is a lightweight semantic segmentation network with multiple stride connections consisting of an initialization module, Block1, Block2, Block3, CARAFE residual, Block4, CARAFE residual and a 1×1 fully connected layer; the semantic segmentation network makes up for the deficiency of MSEC in correcting image areas lacking semantic information.
[0095] (2) This patent improves the reverse bottleneck layer in the 7×7 and 9×9 large core MS-Block in the original YOLO-MS, introduces 3×3 deep convolution, residual structure with halved input channels and asymmetric convolution, and adjusts the branch structure of MS-Block, thereby optimizing the upstream and downstream relationship of the network while maintaining accuracy, improving the calculation speed and reducing parameters; at the same time, the neck network of YOLO-MS is improved to SFPN, reducing the scale truncation problem, smoothing feature fusion and greatly improving the calculation speed at the same time.
[0096] (3) This patent provides an implementable lightweight method for real-time detection of pedestrians in harsh environments. Because it adopts visible light and infrared fusion technology, and designs a semantic segmentation module, a noise reduction module and a lightweight detection module, it can greatly improve the detection of pedestrians in harsh environments, especially in dark environments and complex environments such as rain, snow, and fog, and provides a feasible method for reducing traffic accidents caused by blurred vision.
[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific implementation methods of the present invention can still be modified or replaced without departing from the spirit and scope of the present invention. Any modification or replacement should be included in the scope of protection of the claims of the present invention.
Claims
1. A lightweight pedestrian detection method in harsh environments based on deep learning, characterized in that The multimodal data collected in complex road scenes are fused through the DIVFusion (Darkness-free Infrared and Visible Image Fusion) fusion algorithm, and the over-exposed parts in the data are removed using the improved MSEC (Learning Multi-Scale Photo Exposure Correction). Then, the multimodal pedestrian position detection is realized through the improved YOLO-MS, which includes the following steps: (1) Collect multimodal data in complex road scenes for preprocessing and annotation: All video images are scaled to 608×608 pixels using the bilinear interpolation algorithm. 2 , LabelImg is used to label pedestrian targets; (2) The collected multimodal data are fused through DIVFusion to obtain fused data: the DIVFusion fusion algorithm is used to perform 1:1 DIVFusion fusion of multiple visible light images and infrared images collected in complex road scenes to reduce data redundancy; (3) exposure correction and noise reduction are performed on the fused data; (4) the YOLO-MS detection network is improved; (5) pedestrian detection is completed using the trained improved YOLO-MS detection model.
2. According to the light-weight pedestrian detection method in harsh environment based on deep learning of claim 1, it is characterized in that: The exposure correction and denoising of the fused data mentioned above refers to the use of improved MSEC for exposure correction and the introduction of IDN (Invertible Denoising Network) for denoising in view of the fact that the DIVFusion fusion method cannot effectively eliminate the overexposure effect of the headlight area in complex road scenes; the improved MSEC refers to the design of a semantic segmentation network and the MSEC's Laplacian pyramid multi-layer structure to fuse them layer by layer to solve the color imbalance problem caused by the lack of semantic information; the design of a semantic segmentation network refers to the design of a lightweight semantic segmentation network with strided connection composed of an initialization module, Block1, Block2, Block3, CARAFE (Content-Aware ReAssembly of FEatures) residual, Block4, CARAFE residual and a 1×1 fully connected layer; wherein Block1 and Block2 are composed of a pre-1×1 convolution and channel shuffle (Channel Compared with the Block1 structure, Block3 removes the asymmetric convolution. CARAFE is a universal lightweight upsampling operator. Furthermore, the loss function L of the semantic segmentation network is reconstructed. GDL and L WCE , calculated according to formula (1), Among them, the initial values of α and β are 0.3 and 0.7, and the weight values are dynamically optimized and updated through back propagation during the training process; w l The weight of each category, r ln is the reference value (Ground True) of category l at the nth pixel, p ln is the corresponding predicted probability value, The initial value is set to 1; M represents the number of categories, y c There are only two values 0 and 1. If the category is the same as the sample category, it takes 1, otherwise it takes 0. is the weight parameter, N represents the total number of pixels, N c represents the number of pixels of the reference value category c, p c represents the probability that the predicted sample belongs to category c, The initial value is set to 1, and the L GDL and L WCE , to improve the nonlinearity of the loss function; the layer-by-layer fusion refers to fusing the image visual features of each layer of the Laplacian pyramid with the image semantic features by multiplying the tensor pixels one by one, which is used to improve the image exposure restoration effect. The specific fusion method is calculated according to formula (2), Among them, F is the fusion result, H and W are the image width and height, is the image visual feature F x The pixels in the i-th and j-th rows of is the image semantic feature F y The pixels in the i-th and j-th rows of is pixel-by-pixel multiplication, and sigmoid is the activation function.
3. According to the light-weight pedestrian detection method in harsh environment based on deep learning of claim 1, it is characterized in that: The improved YOLO-MS detection network refers to improving the trunk and neck network of YOLO-MS and removing the attention mechanism module of the YOLO-MS detection head; the improved YOLO-MS trunk and neck network refers to introducing 3×3 small core convolution blocks, asymmetric convolution layers and residual blocks in the YOLO-MS trunk to improve the reverse bottleneck layer in the third 7×7 large core and the fourth 9×9 large core original MS-Block in the trunk, while adjusting the MS-Block branch structure and replacing the neck network in YOLO-MS with a synthetic fusion pyramid network SFPN (Synthetic Feature Pyramid Networks), thereby optimizing the network structure and alleviating the scale differences of the original different feature maps; the removal of the attention mechanism module of the YOLO-MS detection head refers to removing the SE attention mechanism of the YOLO-MS detection head part to enhance the real-time performance of pedestrian detection.
Citation Information
Patent Citations
Exposure correction method based on sample relevance
CN117014730A
Semi-supervised domain adaptive weak light environment pedestrian detection method
CN117854044A
Target detection method and apparatus
US20180157938A1
Cited By
Deep learning-based human body posture recognition method and system in dark environment
CN120412074A
Industrial flexible object detection method based on machine vision
CN120876399A
An industrial flexible object detection method based on machine vision
CN120876399B
Method and system for detecting nonferrous metal target of scraped car
CN120953758A
