A pedestrian detection method for road scenes

Abstract

The application discloses a pedestrian detection method for a road scene, and the method comprises the following steps: in the single multi-scale target detection task of road pedestrians, in order to realize information interaction and supplement between cross-scale branches, inputting Conv3_3, Conv4_3, Conv7 and Conv8_2 layers into a jump reverse feature pyramid structure designed in the application in a path from shallow to deep to realize feature fusion; merging information on the feature layers to be fused in a certain way to obtain features with better judgment than original features; and repeatedly considering the attention of the row and column where the target pixel features are located by using a cross attention module to capture the dependency relationship between non-local feature semantics. The application can solve the problem that key information of each detection branch cannot be integrated, and can also solve the problem that the fusion mode of directly adding operation does not consider the contribution degree of the feature to be fused to the high-level feature obtained after fusion.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of pedestrian detection, and particularly relates to a pedestrian detection method for a road scene. BACKGROUND

[0002] Artificial intelligence technology is constantly improving and developing, gradually expanding from the production field to the living field, and becoming part of human life. Detecting pedestrians in a traffic road scene is a sub-task in computer vision, which is used to determine whether there is a pedestrian in the collected image or video data. If there is a pedestrian, a rectangle or other frame is used to determine the specific location of the target and give the judgment accuracy of a single target. This technology can save time and cost, reduce the use of human resources, produce rich economic and social benefits, and has a very wide range of applications.

[0003] With the joint efforts of all researchers in the world, pedestrian detection algorithms have gradually made breakthroughs from low-level to high-level and from local to universal. The literature (Zhou Y F, Li W L, Hu R R. Multi-scale feature fusion of dual-channel SSD pedestrian head detection algorithm [J]. Laser and Optoelectronics Progress, 2021, 58(24): 383-394.) designs a dual-channel SSD network and fuses the high and low layer feature maps to increase the model detection accuracy by adjusting the prior box of SSD. The literature (Wang H L, Dai S, Liu D, et al. Small-scale pedestrian detection research of improved SSD algorithm in traffic scene [J]. Computer Engineering and Applications, 2022, 58(02): 201-207.) proposes a Resnet50-based SSD network architecture and uses feature fusion method to improve the model's detection ability for small-scale pedestrians. The literature (Qiang X Y, Yang J B, Zhang J H. Pedestrian detection based on improved DSSD algorithm [J]. China Instrumentation, 2021(05): 21-27.) combines the improved ResNeXt feature extraction model as the pre-network of the DSSD detection framework, and performs deconvolution operation on the deep network to extract features of different scales, and uses the same FPN fusion strategy to fuse the features of different scales with the shallow network. The literature (Dong Y C, Shan Y G, Yuan J. Pedestrian detection method based on improved SSD algorithm [J]. Computer Engineering and Design, 2020, 41(10): 2921-2926. DOI:10.16208 / j.issn1000-7024.2020.10.037.) uses DenseNet as the basis network of SSD, adds four convolutional layers to build a new network, and extracts the target frame from the last four layers of the new network and the last two DenseBlocks of DenseNet to improve the model detection accuracy. Although the above algorithms have improved the detection accuracy in specific environments, the design of multi-scale target detection structure for specific tasks lacks pertinence, and different semantic level features cannot be efficiently fused, and the positioning of occluded targets is still not accurate. In addition, the related work cannot balance the accuracy and detection speed, and there is still room for further improvement. Designing an efficient pedestrian detection system with detection accuracy and speed has become a research hotspot in the field of intelligent auxiliary driving.

[0004] SSD is a classical multi-scale single-stage target detection algorithm based on regression idea and Anchor mechanism. It normalizes the original input image and adjusts it to a fixed size of 300*300 as the input of the model. VGG-16 is used as the basic backbone network. In the process of continuous downsampling, the features of the input image are extracted through 6 multi-scale feature layers with sizes of 38*38, 19*19, 10*10, 5*5, 3*3 and 1*1, and anchor boxes are defined on the prediction branch. Finally, the best prediction box is retained by using non-maximum suppression to output the detection result.

[0005] The SSD model structure can be divided into two parts: the basic network and the additional network. The basic network is used to extract key features and is composed of a modified VGG-16 network. The original VGG-16 network has 16 layers, including 5 groups of convolution.

[0006] The modified part is: the first 13 convolutional layers of the VGG-16 network are retained; to improve the receptive field size of the model and adapt to the change of pool5 from the original 2*2 convolution with a stride of 2 to 3*3 convolution with a stride of 1, smoothing processing is performed, i.e. replacing the first and second fully connected layers with dilated convolution and ordinary convolution respectively. At the same time, all Dropout layers and the third fully connected layer in the network are removed.

[0007] The additional network is used to increase the receptive field of the model and obtain high-level semantic feature information, which is composed of 4 groups of convolution, each group having 2 convolutional layers, a total of 8 convolutional layers.

[0008] The final model selects the Conv4 layer in the basic network and the Conv7 layer, the Conv8_2 layer, the Conv9_2 layer, the Conv10_2 layer and the Conv11_2 layer in the additional network as the detection branch.

[0009] The SSD model uses a multi-scale prediction method: using large-scale feature layers to predict small targets; using small-scale feature layers to predict large targets, to solve the problem of target size variation in the detection process. However, the 6 detection branches are independent of each other, and the position information and classification confidence of the prediction box are calculated synchronously through aggregation operation. At this time, the bottom features such as Conv4_3 do not have high-level semantic information and are not friendly to the target classification task; the high-level features such as Conv11_2 lack detailed information with the increase of convolution operation times and are not friendly to the target positioning task. The one-way propagation path structure from shallow to deep makes the feature information carried by the detection branch insufficient, and the detection ability of the model has a lot of room for improvement. In addition, in the same scale of the prediction feature layer, the size of each target is different, and in order to identify all targets on this layer, the context information needs to be grasped to facilitate target positioning. SUMMARY

[0010] Based on the above deficiencies of the prior art, the present application proposes a pedestrian detection method for a road scene, which can solve the problem of not integrating key information of each detection branch and the problem of not considering the contribution degree of the to-be-fused features to the high-level features obtained after fusion in the fusion mode of directly performing addition operation.

[0011] In order to realize the above application, a pedestrian detection method for a road scene is proposed, comprising the following steps:

[0012] S1, building an SSD target detection model based on a Pytorch framework on a GUP platform, including a feature extraction backbone network, a feature extraction network and a classification detection module;

[0013] S2, inputting part of the detection branches of the SSD model to a jump reverse feature pyramid structure for feature fusion from shallow to deep, so as to fully interact and supplement the information between the cross-scale branches;

[0014] S3, merging the information on the to-be-fused feature layer in a certain way to obtain a feature with better judgment than the original feature;

[0015] S4, repeatedly considering the attention of the row and column where the target pixel feature is located by referring to a cross attention module, and capturing the dependency relationship between non-local feature semantics;

[0016] S5, setting corresponding training parameters to train the model, and stopping training and saving the obtained model and weight file when the model reaches a fitting state or reaches a set maximum training iteration number;

[0017] S6, reading test data for testing according to the trained model, visualizing and comparing the test pictures, and saving the test results.

[0018] Preferably, in the step S2, part of the detection branches Conv3_3, Conv4_3, Conv7 and Conv8_2 of the SSD model are input to the jump reverse feature pyramid structure, and finally three layers of prediction feature layers are output, so that the shallow layer information is integrated into the deep layer feature, the model positioning ability is enhanced, and the information difference between adjacent sub-features of the high-level semantic features is enhanced.

[0019] Preferably, in the step S3, the to-be-fused features adjusted to the same resolution are respectively passed through a global average pooling and a global maximum pooling module, and after calculation, information distribution in each channel of the two feature layers is obtained, and through one-dimensional convolution operation, the information of adjacent channels is interacted; then the corresponding channel information is one-dimensionally spliced, the information amount carried in each channel of the two is interacted through a Sigmoid function, attention is formed at the channel level, and the attention is taken as an updated information weight, and then the corresponding channel is distributed back, and non-local information analysis is performed on the two to-be-fused features, the prior contribution degree to the fused feature is obtained through learning, interaction and comparison of non-local information between the channels of the to-be-fused feature layers, and the prior contribution degree is taken as a weight, more attention is paid to key information, and relevant information is directly enhanced or inhibited, and the feature layers are fused through the element-by-element addition method.

[0020] Preferably, the training parameters in the step S5 include a training batch size, a learning rate, an iteration number, a confidence threshold and a non-maximum suppression threshold.

[0021] From the above, the pedestrian detection method for a road scene has the following beneficial effects:

[0022] (1) The multi-scale prediction feature layers of the SSD model are independent of each other, the feature extraction mode of the model is researched, and the problem that the key information of each detection branch cannot be integrated is solved.

[0023] (2) The feature fusion mode is researched in the feature fusion process, and the problem that the fusion mode of directly adding operation does not consider the contribution degree of the to-be-fused features to the obtained high-level features after fusion is solved.

[0024] (3) The problem that the target scales in the prediction feature layers of the same scale are different and simultaneously participate in target positioning prediction, the model is not accurate in target detection under the condition of occlusion, and target missed detection or false detection is caused is solved.

[0025] The above description is only a summary of the technical scheme of the present application, in order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following will be combined with preferred embodiments and the drawings, and will be described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical scheme of the embodiment of the present application, the drawings of the embodiment will be briefly introduced as follows.

[0027] Fig. 1 The flowchart of the pedestrian detection method for a road scene of the present application is shown in the figure.

[0028] Fig. 2 The SRFPN structure diagram is shown in the figure.

[0029] Fig. 3 The structure diagram of the adaptive feature fusion module. DETAILED DESCRIPTION

[0030] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings, which are part of this specification, and illustrate the principles of the present application by way of examples. Other aspects, features, and advantages of the present application will become apparent from this detailed description. In the drawings, the same or similar components are denoted by the same reference numbers throughout the different figures.

[0031] As shown in the figure, the pedestrian detection method for road scenes of the present application comprises the following steps: Figs. 1 to 3

[0032] S1, building an SSD target detection model based on a Pytorch framework on a GUP platform, including a feature extraction backbone network, a feature extraction network, and a classification detection module;

[0033] S2, inputting the partial detection branches of the SSD model to a jump reverse feature pyramid structure for feature fusion on the path from shallow to deep, to fully interact and supplement the information between the cross-scale branches;

[0034] S3, merging the information on the feature layers to be fused in a certain way to obtain features with better judgment than the original features;

[0035] S4, repeatedly considering the attention of the target pixel features in the rows and columns by referencing a cross-attention module, to capture the dependency relationship between non-local feature semantics;

[0036] S5, setting corresponding training parameters to train the model, and stopping training and saving the obtained model and weight file when the model reaches the fitting state or reaches the set maximum training iteration number;

[0037] S6, reading test data for testing according to the trained model, and visualizing and comparing the test pictures to save the test results.

[0038] Jump reverse feature pyramid structure

[0039] In the SSD model, each detection branch is independent of each other, lacking effective information complementarity and relevance. The FPN structure performs recursive fusion on adjacent features from deep to shallow, and the PAN structure performs feature fusion from deep to shallow and then from shallow to deep. NAS-FPN and BiFPN increase the feature expression capability by stacking the pyramid structure.

[0040] ​The self-deep-to-shallow path has limitations on the pedestrian detection task. In the fusion feature pyramid structure, the continuous up-sampling of the features will destroy the pixel continuity in the original feature layer. The extraneous information introduced by the interpolation method will continue to propagate forward with the convolution operation, and after being fused with the shallow features introduced in the self-shallow-to-deep path, it is easy to form a positional deviation, which is not conducive to target positioning. In addition, pedestrian detection is a single target detection task, and the integration of high-level semantic features into shallow feature layers is beneficial to target classification. In this case, the information that is beneficial to positioning in the shallow feature layer should be considered for complete retention.

[0041] The way of integrating shallow information into deep features has two aspects of help to improve the detection ability of the model:

[0042] (1) Enhance the positioning ability of the model

[0043] The information is sent into the backbone network and then to the detection branch. After layer-by-layer convolution operation and down-sampling, the resolution gradually decreases, and the number of sub-features on each feature layer also gradually decreases, and the positioning information is gradually lost. However, the semantic information of deep features is relatively higher. However, due to the small resolution of deep information, the information contained between adjacent features gradually becomes similar. Under the same resolution, due to the small receptive field of shallow features, they have more detailed information such as position and edge than deep features, and have better distinguishability. After feature fusion, the adjacent sub-features of deep features are enhanced due to the integration of the detailed information carried in the shallow features, and the difference between the adjacent sub-features is enhanced, and the target position is more sensitive.

[0044] (2) Enhance the information difference between adjacent sub-features of high-level semantic features

[0045] In complex real-life environments, the target to be detected may be occluded. The main difficulty in detecting the occluded target is that the visible pixels are limited, and the higher the degree of overlap between them, the higher the proportion of shared information in the feature layer. The introduction of shallow features supplements the detailed information between similar adjacent features, making them more different, which provides a strong help to solve the occlusion problem.

[0046] Therefore, a skip reverse feature pyramid network (SRFPN) is designed. In order to interact and supplement the information between the cross-scale branches, the Conv3_3, Conv4_3, Conv7, and Conv8_2 layers are input into the SRFPN structure in the self-shallow-to-deep path, and the prediction feature layers P1, P2, and P3 are obtained.

[0047] The main contents of the SRFPN structure include adjusting the number of feature channels, feature splicing, and activation function ReLU. The structure is as shown in Fig. 2 .

[0048] First, the Conv3_3 layer is down-sampled and given a 2-fold channel upgrade, from the original 256 to 512; the Conv7 layer is down-sampled from the original 1024 channels to 512; the Conv4_3 and Conv8_2 layers are subjected to 1x1 convolution to maintain the number of channels. Subsequently, the Conv3_3 layer with the changed scale is added and fused with the Conv4_3 layer, and the fused features continue to be down-sampled and fused with deeper features. This is an iterative operation, which will not be described again. Finally, three layers of fused high-level features are obtained. Since the addition and fusion will increase the difference between adjacent pixel-level sub-features, a 3x3 convolution is used to smooth the pixel-level sub-features of the high-level features, and the number of channels is uniformly adjusted to 256. Then, the number of channels of the Conv4_3, Conv7, and Conv8_2 layers is uniformly down-sampled to 256. Finally, data normalization (indicated by BN in the figure) and ReLU activation function are performed, respectively, and then feature splicing operation is performed at the channel level to obtain 512-channel P1, P2, and P2 prediction feature layers.

[0049] The splicing operator calculation process is represented as:

[0050]

[0051] wherein Z concat represents the features after the splicing operator, x i and y i represent the features to be fused, k i is the convolution kernel, and * is the convolution operation.

[0052] Adaptive feature fusion module based on attention mechanism

[0053] The purpose of feature fusion is to combine the information on the feature layer in a certain way to obtain features with better judgment than the original features. Generally, feature fusion needs to select feature layers of different scales. Low-level features have low semantic information and contain more detailed information, while high-level features are not sensitive to detailed information but have higher-level semantic information. How to combine the advantages of both for target detection tasks and remove irrelevant or even harmful information for efficient fusion is the key to improving the detection ability of the model.

[0054] There are two common feature fusion methods: one is concatenation, which splices feature layers of the same scale in the channel dimension; the other is addition, which adds corresponding elements one by one. The addition fusion can increase the amount of information while maintaining the size of the original feature dimension, and its computational complexity is less than that of concatenation.

[0055] In addition, when performing feature fusion, it is necessary to adjust different resolution feature layers to the same scale through upsampling or downsampling. However, the semantic information carried by different resolution feature layers is different in level, and in the feature fusion process, the correlation of the two in the channel level is inconsistent, and the importance of targets of different scales in space is also different, which may cause aliasing effect and position information offset in the feature layer, and confuse the positioning and recognition tasks. In order to avoid the above problems and affect the detection accuracy of the model, the application uses attention mechanism and one-dimensional convolution operation to propose a plug-and-play adaptive feature fusion module, the structure of which is shown in Fig. 3

[0056] The module adjusts the to-be-fused features of the same resolution to pass through the global average pooling (GAP) and global maximum pooling module (GMP) respectively, and the information distribution in each channel of the two feature layers is obtained after calculation. After one-dimensional convolution operation, the information of adjacent channels is interacted. Then the corresponding channel information is one-dimensionally spliced, the information amount carried in each channel of the two is interacted through the Sigmoid function, attention is formed in the channel level, and the attention is taken as the updated information weight and distributed back to the corresponding channel. The purpose is to analyze the non-local information of the two to-be-fused features, learn, interact and compare the non-local information between the channels of the to-be-fused feature layers, obtain the prior contribution degree of the fused feature, and use it as a weight to pay more attention to the key information, directly enhance or suppress the relevant information, and fuse the feature layers through the element-by-element addition method.

[0057] The module can comprehensively analyze and calculate the information carried by the to-be-fused feature layers, compare the key information content between the corresponding channels of the two, adaptively enhance the representation of key information, suppress redundant information and conflict information, and achieve the purpose of adaptively paying attention to key targets in the feature fusion process.

[0058] Non-local feature positioning enhancement module

[0059] In the same scale feature layer, the sizes of the targets are different, and in order to identify all the targets on the feature layer, different scale context information is needed to assist in distinguishing. Therefore, the feature information with different sizes in the feature layer of the same semantic level feature should be treated equally.

[0060] The cross attention module (CCN) only collects the context information of all pixel-level sub-features on the cross path in the corresponding width and height directions based on the target pixel position, obtains the non-local information dependency relationship, and reduces the calculation amount of the model.

[0061] ​In the spatial dimension of the feature layer, the cross attention module repeatedly considers the attention of the target pixel feature in the row and column, capturing the dependency between non-local feature semantics. The present application introduces this structure into the positioning branch of the SSD detection model, forming a non-local feature positioning enhancement module, which accurately calculates the predicted frame position and increases the positioning ability of the target.

[0062] First, detect the branch F, F element of R C×W×H Parallelly pass through 1x1 convolution to obtain feature layer K, Q (C' < C) with scale size C'xWxH and scale-invariant feature layer V, respectively, and obtain the index i corresponding to each pixel feature point in K. Vector M u Element of R C'×1×1 and the set Omega composed of H+W-1 feature vectors in the corresponding row and column in the feature layer Q u Element of R (H+W-1)×C Through correlation calculation, M i obtain the correlation between Omega i , and then obtain the attention A i Element of R (H+W-1) , the process can be represented as:

[0063] A i = softmax (Omega i M i ) (1)

[0064] Similarly, in the feature layer V, the vector set Phi i Element of R (H+W-1)×C is obtained, and the non-local context information F i Element of R (C×1×1) of the corresponding position i is obtained through aggregation calculation, the process can be represented as:

[0065] F i '= Phi i T A i +F i (2)

[0066] After the index is traversed, the final feature layer F' is obtained through the above steps. To avoid sparse information obtained by cross-intersection form to obtain long-distance feature dependency, it needs to pass through the module twice. After obtaining the information through this operation, the predicted frame position information of the corresponding detection branch is obtained through the original positioning branch convolution calculation.

[0067] In order to verify the effectiveness of the improved model, the training is carried out on the public benchmark PASCAL VOC dataset, and the original SSD model, the improved SSD model series algorithm, the two-stage representative algorithm, the YOLO series algorithm and the current latest literature are compared under the dataset. The detection results of different algorithms on the PASCAL VOC dataset "person" label are shown in Table 1.

[0068] Table 1: Comparison results of PASCAL VOC dataset "person" label

[0069]

[0070]

[0071] As can be seen from the table, the detection results of the improved model are 7.1% and 1.7% higher than those of the two-stage representative algorithm Faster R-CNN and ION respectively; the accuracy is 0.5% and 6.8% higher than that of the single-stage representative model YOLOv3 and DC-SPP-YOLO algorithm respectively; compared with the improved SSD model series algorithm: DSSD, RSSD, EDF-SSD and AFP-SSD, the detection results are 4.1%, 3.6%, 2.8% and 4.4% higher respectively; compared with the original basic model SSD, the detection accuracy is 4.4% higher.

[0072] The above is the preferred embodiment of the present application, of course, cannot be limited by this to limit the scope of the present application, it should be pointed out that, for ordinary skilled in the art, without departing from the principles of the present application, can also make a number of improvements and changes, these improvements and changes are also considered to be within the scope of the present application.

Claims

1. A pedestrian detection method for a road scene, characterized in that, Comprise the following steps: S1, build an SSD target detection model based on the Pytorch framework on the GUP platform, including a feature extraction backbone network, a feature extraction network, and a classification and detection module; S2, on the path from shallow to deep, input the partial detection branches of the SSD model into a jump reverse feature pyramid structure for feature fusion, fully interact and supplement the information between the cross-scale branches; S3, merge the information on the to-be-fused feature layers in a certain way to obtain features with better judgment than the original features; The to-be-fused features adjusted to the same resolution are respectively subjected to global average pooling, and the information distribution in each channel of the two feature layers obtained after calculation is subjected to one-dimensional convolution operation to interact the information of adjacent channels, and then the corresponding channel information is one-dimensionally spliced, the amount of information carried in each channel of the two is interacted through a Sigmoid function, attention is formed at the channel level, and the attention is taken as updated information weight and distributed back to the corresponding channel. The purpose is to analyze the non-local information of the two to-be-fused features, learn, interact, and compare the non-local information between the channels of the to-be-fused feature layers, obtain the prior contribution degree of the fused feature, take this as the weight, invest more attention on the key information, directly enhance or suppress the related information, and fuse the feature layers in a targeted manner through element-by-element addition; S4, repeatedly consider the attention of the target pixel feature in the row and column by referring to the cross attention module, and capture the dependency relationship between the non-local feature semantics; S5, set corresponding training parameters to train the model, stop training and save the obtained model and weight file when the model reaches the fitting state or reaches the set maximum training iteration number; S6, read test data for testing according to the trained model, and visualize and compare the test pictures, and save the test results.

2. The pedestrian detection method for a road scene according to claim 1, wherein, In the step S2, the partial detection branches Conv3_3, Conv4_3, Conv7, and Conv8_2 of the SSD model are input into the jump reverse feature pyramid structure, and three layers of prediction feature layers are finally output, the shallow layer information is fused into the deep layer feature, the model positioning ability is enhanced, and the information difference between adjacent sub-features of the high-level semantic features is enhanced.

3. The pedestrian detection method for a road scene according to claim 1, wherein, In the step S3, the to-be-fused features adjusted to the same resolution are respectively subjected to global average pooling and global maximum pooling modules, and the information distribution in each channel of the two feature layers obtained after calculation is subjected to one-dimensional convolution operation to interact the information of adjacent channels; then the corresponding channel information is one-dimensionally spliced, the amount of information carried in each channel of the two is interacted through a Sigmoid function, attention is formed at the channel level, and the attention is taken as updated information weight and distributed back to the corresponding channel. The purpose is to analyze the non-local information of the two to-be-fused features, learn, interact, and compare the non-local information between the channels of the to-be-fused feature layers, obtain the prior contribution degree of the fused feature, take this as the weight, invest more attention on the key information, directly enhance or suppress the related information, and fuse the feature layers in a targeted manner through element-by-element addition.

4. The pedestrian detection method for a road scene according to claim 1, wherein, The training parameters in step S5 include: training batch size, learning rate, iteration number, confidence threshold, non-maximum suppression threshold.

Images