A highway occluded pedestrian detection method based on a proposal box generation enhancement algorithm

Abstract

The application discloses a highway pedestrian detection method based on a proposal box generation enhancement algorithm, and comprises the following steps: acquiring a to-be-detected image; inputting the to-be-detected image into a trained pedestrian detection model to perform pedestrian detection, wherein the pedestrian detection model comprises a feature extraction module, a target proposal box generation module and a detection head module, and the step of inputting the to-be-detected image into the trained pedestrian detection model to perform pedestrian detection comprises the following steps: S1, performing feature extraction on the to-be-detected image, and performing feature fusion by adopting a combination of bilinear difference and 1x1 convolution to generate a feature map of the to-be-detected image; S2, generating a target proposal box according to the feature map of the to-be-detected image generated in S1 by using a proposal box generation enhancement algorithm; and S3, projecting the target proposal box onto the feature map to perform processing, and obtaining a detection result of the to-be-detected image. The method improves the detection effect on a blocked pedestrian target.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent traffic information, and designs a highway pedestrian detection method based on a proposal box generation enhancement algorithm, which is suitable for detecting occluded pedestrian targets in a highway scene. BACKGROUND

[0002] In a highway scene, applying pedestrian detection technology to a highway monitoring and management system can quickly identify and locate pedestrian targets within the monitoring range, enhance the perception of highway traffic anomalies, and improve the overall management level of highways, which is of great significance to improving traffic efficiency and ensuring traffic safety.

[0003] Due to the characteristics of the highway scene, the pedestrian targets captured by the highway camera are often accompanied by different degrees of occlusion, which reduces the detection accuracy of the pedestrian detection model. Therefore, it is necessary to explore a detection scheme for occluded pedestrian targets according to the actual situation of the scene to improve the detection efficiency of the model. To address the occlusion problem, researchers have proposed corresponding solutions from both one-stage and two-stage model categories.

[0004] For two-stage detection models, existing solutions design some detection heads that improve the quality of bounding box regression in the detection head by focusing on the visible part of the pedestrian target proposal box, improving the target proposal box modeling algorithm, and other ways. Typical representative methods include those disclosed in patent application CN115690849A and patent application CN113610037A. However, these approaches do not improve the quality of the proposal boxes obtained by the proposal box generation network, which requires detailed tuning of the proposal boxes in the detection head, resulting in the consumption of more computational resources and time, affecting the model efficiency.

[0005] For one-stage detection models, some existing solutions choose to classify each pixel on the feature map based on the center point and scale and regress the possible target corresponding to the point, and then introduce a focal loss to handle the imbalance between positive and negative samples. Typical representatives include the CSP network. These approaches abandon the idea of generating rough proposal boxes first and then refining them in the detection head in two-stage models, and the network has good convergence, improving the detection speed of the model. However, these approaches will produce more false positives, and this phenomenon will be exacerbated in the detection of occluded pedestrians. SUMMARY

[0006] Therefore, in the highway scene, the detection of the occluded pedestrian must consider the problems of the decline of the target image feature quality caused by the occlusion and the false positive of the detection result, and the detection effect of the model on the occluded pedestrian target is improved by fully extracting and using the image features to be detected and improving the target proposal box generation method.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] A highway pedestrian detection method based on a proposal box generation enhancement algorithm, comprising: obtaining an image to be detected; inputting the image to be detected into a trained pedestrian detection model for pedestrian detection, wherein the pedestrian detection model comprises a feature extraction module, a target proposal box generation module and a detection head module, the feature extraction module comprises a feature extraction network HRNetV2-W32, and the inputting of the image to be detected into the trained pedestrian detection model for pedestrian detection comprises:

[0009] S1: in the feature extraction module, feature extraction is performed on the image to be detected, and feature fusion is performed by using a combination of bilinear interpolation and 1*1 convolution to generate a feature map of the image to be detected;

[0010] S2: in the target proposal box generation module, a proposal box generation enhancement algorithm is used to generate a target proposal box according to the feature map of the image to be detected generated in S1;

[0011] S3: in the detection head module, the target proposal box is projected onto the feature map for processing to obtain a detection result of the image to be detected.

[0012] Further, S1 specifically comprises:

[0013] S1.1: inputting the image to be detected of W*H into the feature extraction network HRNetV2-W32 in the feature extraction module, obtaining four feature maps of different scales in stage 4 after the first three stages, wherein W represents the width of the image to be detected, and H represents the height of the image to be detected;

[0014] S1.2: using a combination of bilinear interpolation and 1*1 convolution to upsample the four feature maps respectively and using a convolution kernel of 1*1*128, 1*1*128, 1*1*256 and 1*1*256 on the four feature maps respectively, and finally generating a feature map with a size of

[0015] S1.3: processing the four scale feature maps to obtain a feature map with a size of ​​​The feature maps of size are spliced to obtain feature maps of size

[0016] Further, the S2 specifically comprises:

[0017] S2.1: performing convolution processing on the feature maps of size by using a 3*3*256 convolution kernel to reduce the channel dimension to 256;

[0018] S2.2: inputting the feature maps after the convolution processing into two 1*1 convolution branches, the branches including a pedestrian center point prediction branch and a pedestrian scale prediction branch, to respectively predict and regress the target center point and the scale to generate a target proposal box.

[0019] Further, the loss function used in the pedestrian center point prediction branch is:

[0020]

[0021]

[0022]

[0023]

[0024]

[0025] wherein L c represents the pedestrian center point prediction branch loss function, K represents the number of pedestrian targets in the image, p ij and y ij respectively represent the pedestrian center point probability predicted by the model and the corresponding pedestrian center point true value, a ij represents the weight of the point at the (i,j) position, CE(p ij , y ij ) represents the cross-entropy loss function, M ij represents the Gaussian mask near the true pedestrian center point, x k , y k , w k and h k respectively represent the coordinates and width and height of the kth pedestrian target center point, and the variance of the Gaussian mask is γ and β represent weight coefficients.

[0026] Further, the loss function used in the pedestrian scale prediction branch is:

[0027]

[0028] wherein L srepresents a pedestrian scale prediction branch loss function, K represents the number of pedestrian targets in the image, L1 represents a standard L1 loss function, s k and t k respectively represent the scale prediction value and the true value of the kth pedestrian target.

[0029] Further, the overall loss function used by the target proposal box generation module is

[0030] L RPN = λ c L c + λ s L s

[0031] wherein, L RPN represents the overall loss function of the target proposal box generation module, λ c represents the weight of the pedestrian center point prediction branch loss function, λ s represents the weight of the pedestrian scale prediction branch loss function.

[0032] Further, the S3 specifically comprises:

[0033] S3.1: using ROIAlign to map the target proposal box into a feature map to obtain a feature matrix with a size of 7x7;

[0034] S3.2: inputting the feature matrix into a 3x3x256 convolution layer and a 1024-dimensional fully connected layer for processing to generate an initial detection result;

[0035] S3.2: using NMS to process the initial detection result to remove false positives in the initial detection result to obtain a final detection result.

[0036] Further, the loss function used by the detection head module is

[0037]

[0038] wherein, L head represents the loss function of the model detection head part, K represents the number of pedestrian targets in the image, z k is the binary label of the kth pedestrian target, which takes the value of 0 or 1, p k is the prediction probability value of the kth pedestrian target.

[0039] Further, the highway occluded pedestrian detection method based on the proposal box generation enhancement algorithm further comprises training a pedestrian detection model using a highway pedestrian target data set, wherein training the pedestrian detection model comprises the following steps:

[0040] S4.1: the network parameters of the HRNetV2-W32 are initialized by using an ImageNet pre-training classification model, and the network parameters of the target proposal frame generation module are trained separately by using the expressway pedestrian target data set;

[0041] S4.2: the network parameters of the target proposal frame generation part are fixed, the network parameters of the HRNetV2-W32 are reinitialized by using an ImageNet pre-training classification model, and the network parameters of the detection head module are trained by using the target proposal frame generated by the target proposal frame generation module;

[0042] S4.3: the network parameters of the HRNetV2-W32 are fixed, and the network parameters of the target proposal frame generation part are trained separately by using the expressway pedestrian target data set;

[0043] S4.4: the network parameters of the HRNetV2-W32 are fixed, and the network parameters of the detection head module are trained separately by using the expressway pedestrian target data set;

[0044] S4.5: the learning rate is fixed as 10-4, the network parameters of the pedestrian detection model are obtained by training according to the preset training round, that is, the pedestrian detection model is determined.

[0045] The beneficial effects of the present application are:

[0046] The present application improves the feature fusion module in the HRNetV2-W32 network from the actual scene of the expressway and the characteristics of the pedestrian target, and designs an efficient target proposal frame generation network based on the CSP network, and designs a corresponding detection head for the model, fully fuses and uses the feature information from multiple levels contained in the to-be-detected picture, improves the quality of target proposal frame generation and regression, and improves the effect of detecting occluded pedestrian targets in the expressway scene.

[0047] Other advantages, objects and features of the present application will be set forth in part in the following specification, and in part will become apparent to those skilled in the art from the examination of the following text, or can be learned from the practice of the present application. The objects and other advantages of the present application can be realized and obtained by the following specification. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings, in which:

[0049] Figure 1 The present application is a general flowchart;

[0050] Figure 2 a schematic diagram for extracting a feature map of an input image;

[0051] Figure 3 a flowchart for generating a target proposal box;

[0052] Figure 4 a flowchart for generating a detection result. DETAILED DESCRIPTION

[0053] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present application, and are not intended to limit the protection scope of the present application.

[0054] The present application provides a pedestrian detection method in a highway scene considering occlusion, as shown in the following formula (I), the method comprises the following steps: Figure 1 acquiring an image to be detected; inputting the image to be detected into a trained pedestrian detection model (also referred to as an "occlusion pedestrian detection model") for pedestrian detection. The pedestrian detection model comprises a feature extraction module, a target proposal box generation module, and a detection head module. In some embodiments, the feature extraction module can comprise a feature extraction network HRNetV2-W32.

[0055] The inputting the image to be detected into the trained pedestrian detection model for pedestrian detection can comprise:

[0056] S1: in the feature extraction module, performing feature extraction on the image to be detected, and performing feature fusion by combining bilinear difference and 1x1 convolution (i.e. Figure 1 fusion of multi-layer features) as shown in the following formula (II), to generate a feature map of the image to be detected;

[0057] S2: in the target proposal box generation module, generating a target proposal box according to the feature map of the image to be detected generated in S1 by using a proposal box generation enhancement algorithm;

[0058] S3: in the detection head module, projecting the target proposal box onto the feature map for processing to obtain a detection result of the image to be detected.

[0059] Figure 2 a schematic diagram for extracting a feature map of an input image. As shown in 2, S1 specifically comprises:

[0060] S1.1: inputting a WxH image to be detected into a feature extraction network HRNetV2-W32 in the feature extraction module, obtaining four feature maps of different scales in stage 4 after the first three stages, wherein W represents the width of the image to be detected, and H represents the height of the image to be detected.

[0061] ​S1.2: Use a combination of bilinear interpolation and 1×1 convolution to respectively... Four feature maps are upsampled and then convolutional kernels of 1×1×128, 1×1×128, 1×1×256, and 1×1×256 are applied to the four feature maps respectively, resulting in a final output of size [missing information]. The feature map. This operation replaces the original upsampling operation in stage 4 of the HRNetV2-W32 network with a combination of bilinear interpolation and 1×1 convolution, respectively, for The four feature maps are upsampled by 1, 2, 4, and 8 times to generate feature maps of the same size.

[0062] S1.3: The size of the feature maps after processing at all four scales is... The feature maps are concatenated to obtain a size of [size missing]. The feature map.

[0063] Therefore, this invention improves the original feature fusion method of the HRNetV2-W32 network by using bilinear interpolation and 1×1 convolution to construct a feature map with rich semantic information using image features from different levels.

[0064] Figure 3 The flowchart illustrates the generation of the target suggestion box. This invention designs a suggestion box generation algorithm, which first utilizes 3×3 convolutions to process the obtained suggestion boxes... The feature map is convolved, and then two convolutional branches are used to generate target proposal boxes for pedestrian targets. For example... Figure 3 As shown, S2 specifically includes:

[0065] S2.1: Use a 3×3×256 convolution kernel. The feature map is convolved to reduce its channel dimension to 256.

[0066] S2.2: Input the convolutional feature map into two 1×1 convolutional branches, which include a pedestrian center point prediction branch and a pedestrian scale prediction branch, to predict and regress the target center point and scale respectively, thereby generating target proposal boxes.

[0067] In some embodiments, the loss function used in the pedestrian center point prediction branch can be:

[0068]

[0069]

[0070]

[0071]

[0072]

[0073] wherein L c represents the loss function of the pedestrian center point prediction branch, K is the number of pedestrian targets in the image, p ij and y ij are the probability of the pedestrian center point predicted by the model and the real value of the corresponding pedestrian center point, respectively, a ij is the weight value of the point at the (i, j) position, CE(p ij , y ij ) represents the cross-entropy loss function, M ij represents the Gaussian mask near the real pedestrian center point, x k , y k , w k and h k are the coordinates and width and height of the kth pedestrian target center point, and the variance of the Gaussian mask is which is proportional to the width and height of the single object, and γ and β represent the weight coefficients. and are used to assign the focal weight value based on the prediction confidence, so as to control the proportion of easy-to-classify samples and difficult-to-classify samples in the loss function, and make the model more focused on the processing of difficult-to-classify samples. (1-M ij β The term is used to reduce the influence of false positives close to the real center point on the loss function. After a large number of experiments, the optimal training effect can be obtained when the weight coefficients are γ = 2 and β = 4.

[0074] In some embodiments, the loss function used in the pedestrian scale prediction branch can be:

[0075]

[0076] wherein L s represents the loss function of the pedestrian scale prediction branch, K is the number of pedestrian targets in the image, L1 indicates the standard L1 loss function, s k and t k represent the scale prediction value and the real value of the kth pedestrian target, respectively.

[0077] In some embodiments, the overall loss function used by the target proposal frame generation module can be:

[0078] L RPN = λ c L c + λ s L s

[0079] wherein L RPN ​denotes the overall loss function of the target proposal box generation module, λ c denotes the weight of the pedestrian center point prediction branch loss function, λ s denotes the weight of the pedestrian scale prediction branch loss function. After a large number of experiments, λ c = 0.05 and λ s = 0.01 can make the model obtain the optimal convergence effect.

[0080] As shown in FIG. 1, after the feature map of the image to be detected and the target proposal box are determined, the target proposal box can be projected onto the feature map for operation (such as ROIAlign) in the detection head module to generate a detection result (i.e., a preliminary detection result, which may contain false positives) and perform post-processing (for example, an operation of removing false positives) on the detection result, thereby obtaining a final detection result.

[0081] Figure 4 is a flowchart of generating a detection result. As shown in FIG. 1, the S3 specifically can include: Figure 4

[0082] S3.1: using ROIAlign to map the target proposal box into the feature map to obtain a feature matrix with a size of 7x7;

[0083] S3.2: inputting the feature matrix into a convolution layer with a size of 3x3x256 and a fully connected layer with a dimension of 1024 to process and generate an initial detection result;

[0084] S3.2: using NMS to process the initial detection result (i.e., post-processing in the Figure 4 ) to remove false positives in the initial detection result to obtain a final detection result.

[0085] The present application can realize the detection of pedestrians in the image to be detected by using the trained pedestrian detection model. Specifically, the high-resolution features in the image to be detected are fully extracted through the HRNetV2-W32 network, and the feature maps with rich semantic information are constructed by using the image features from different levels through the improved feature fusion method. The present application also improves the target proposal box generation network (i.e., first using 3x3 convolution to respectively perform convolution processing on the obtained feature maps, and then generating the target proposal box of the pedestrian target through two convolution branches), generates high-quality target proposal boxes based on the feature maps, maps the target proposal boxes onto the feature maps for operation in the detection head, and performs post-processing to obtain the final detection result.

[0086] In some embodiments, the loss function used by the detection head module can be:

[0087]

[0088] ​wherein, L head represents a loss function of the model detection head part, K is the number of pedestrian targets in the image, z k is the binary label of the kth pedestrian target, and its value is 0 or 1, p k is the prediction probability value of the kth pedestrian target.

[0089] In some embodiments, the expressway occluded pedestrian detection method based on the proposal box generated enhancement algorithm further comprises training a pedestrian detection model using an expressway pedestrian target data set, wherein training the pedestrian detection model can include the following steps:

[0090] S4.1: initialize the network parameters of HRNetV2-W32 using the ImageNet pre-trained classification model, and train the network parameters of the target proposal box generation module separately using the expressway pedestrian target data set;

[0091] S4.2: fix the network parameters of the target proposal box generation part, re-initialize the network parameters of HRNetV2-W32 using the ImageNet pre-trained classification model, and train the network parameters of the detection head module using the target proposal box generated by the target proposal box generation module;

[0092] S4.3: fix the network parameters of HRNetV2-W32, and train the network parameters of the target proposal box generation part separately using the expressway pedestrian target data set;

[0093] S4.4: fix the network parameters of HRNetV2-W32, and train the network parameters of the detection head module separately using the expressway pedestrian target data set;

[0094] S4.5: fix the learning rate to 10-4, train according to the preset training rounds (for example, 80 times), and obtain the network parameters of the pedestrian detection model, i.e., determine the pedestrian detection model.

[0095] The present application trains the designed occluded pedestrian detection model on the self-built expressway pedestrian target data set until the model converges, and saves the model weight generated by training. Loading and running the pedestrian detection model and the corresponding weight obtained by training can realize effective detection of occluded pedestrian targets in the expressway scene.

[0096] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced equivalently without departing from the purpose and scope of the technical solutions, and all should be covered in the scope of the claims of the present application.

Claims

1. A highway pedestrian detection method based on a proposal box generated enhanced algorithm, characterized in that, The method comprises the following steps: obtaining an image to be detected; inputting the image to be detected into a trained pedestrian detection model for pedestrian detection, wherein the pedestrian detection model comprises a feature extraction module, a target proposal box generation module and a detection head module, the feature extraction module comprises a feature extraction network HRNetV2-W32, and the inputting the image to be detected into the trained pedestrian detection model for pedestrian detection comprises the following steps: S1: in the feature extraction module, performing feature extraction on the image to be detected, and performing feature fusion by using a combination of bilinear interpolation and 1x1 convolution to generate a feature map of the image to be detected; S1.1: input a to-be-detected image of WxH into a feature extraction network HRNetV2-W32 in the feature extraction module, and obtain four feature maps of different scales in stage 4 after the first three stages wherein W represents the width of the to-be-detected image, and H represents the height of the to-be-detected image. S1.2: upsample the four feature maps respectively by the combination of bilinear interpolation and 1x1 convolution, and use 1x1x128, 1x1x128, 1x1x256, 1x1x256 convolution kernels respectively on the four feature maps, finally generate feature maps with size of ​​ S1.3: The four feature maps processed in S1.2 are spliced to obtain a feature map with a size of . ​ S2: in the target proposal box generation module, generating a target proposal box according to the feature map of the image to be detected generated in S1 by using a proposal box generation enhancement algorithm; S2.1: Use a 3×3×256 convolution kernel. The feature map is convolved to reduce its channel dimension to 256. S2.2: inputting the feature map after convolution into two 1x1 convolution branches, the branches comprising a pedestrian center point prediction branch and a pedestrian scale prediction branch, to respectively predict and regress the target center point and scale, and generate the target proposal box; S3: in the detection head module, projecting the target proposal box onto the feature map for processing to obtain a detection result of the image to be detected.

2. The highway pedestrian detection method based on proposal box generation enhancement algorithm of claim 1, wherein, The loss function used in the pedestrian center point prediction branch is: where L c represents the pedestrian center point prediction branch loss function, K is the number of pedestrian targets in the image, p ij and y ij are the model predicted pedestrian center point probability and the corresponding pedestrian center point true value respectively, α ij is the weight value of the point at the (i,j) position, CE(p ij , y ij ) represents the cross-entropy loss function, M ij represents the Gaussian mask near the true pedestrian center point, x k , y k , w k and h k are the coordinates and width and height of the kth pedestrian target center point respectively, and the variance of the Gaussian mask is γ and β represent the weight coefficients.

3. The highway pedestrian detection method based on proposal box generation enhancement algorithm of claim 2, wherein, The loss function used in the pedestrian scale prediction branch is: wherein L s represents a pedestrian scale prediction branch loss function, K is the number of pedestrian targets in the image, L1 indicates a standard L1 loss function, s k and t k respectively represent the scale prediction value and the true value of the kth pedestrian target.

4. The highway pedestrian detection method based on proposal box generation enhancement algorithm of claim 3, wherein, The overall loss function used in the target proposal box generation module is: L RPN = λ c L c + λ s L s wherein, L RPN represents the overall loss function of the target proposal box generation module, λ c represents the weight of the pedestrian center point prediction branch loss function, λ s represents the weight of the pedestrian scale prediction branch loss function. 5.The highway pedestrian detection method based on proposal box generation enhancement algorithm of claim 1, wherein, S3 specifically comprises: S3.1: using ROIAlign to map the target proposal box into the feature map to obtain a feature matrix with a size of 7x7; S3.2: inputting the feature matrix into a 3x3x256 convolution layer and a 1024-dimensional fully connected layer for processing to generate an initial detection result; S3.2: using NMS to process the initial detection result to remove false positives in the initial detection result to obtain a final detection result. 6.The highway pedestrian detection method based on proposal box generation enhancement algorithm of claim 1, wherein, The loss function used in the detection head module is: wherein L head represents a loss function of the model detecting the head part, K is the number of pedestrian targets in the image, z k is a binary label of the kth pedestrian target, whose value is 0 or 1, p k is a predicted probability value of the kth pedestrian target. 7.The highway pedestrian detection method based on proposal box generation enhancement algorithm of claim 1, wherein, The method further comprises training the pedestrian detection model by using a highway pedestrian target data set, wherein the training of the pedestrian detection model comprises the following steps: S4.1: initializing the network parameters of HRNetV2-W32 by using an ImageNet pre-trained classification model, and training the network parameters of the target proposal box generation module separately by using the highway pedestrian target data set; S4.2: fixing the network parameters of the target proposal box generation part, reinitializing the network parameters of HRNetV2-W32 by using the ImageNet pre-trained classification model, and training the network parameters of the detection head module by using the target proposal box generated by the target proposal box generation module; S4.3: fixing the network parameters of HRNetV2-W32, and training the network parameters of the target proposal box generation part separately by using the highway pedestrian target data set; S4.4: fixing the network parameters of HRNetV2-W32, and training the network parameters of the detection head module separately by using the highway pedestrian target data set; S4.5: The fixed learning rate is 10 -4 According to the preset training round, the network parameters of the pedestrian detection model are obtained, that is, the pedestrian detection model is determined.

Images