A pedestrian detection method and system based on improved RFBNet algorithm

Abstract

The present invention discloses a pedestrian detection method and system based on an improved RFBNet algorithm, comprising the following steps: Step 1: obtaining a real-time image of a road surface in a current frame; Step 2: inputting the real-time image of the road surface in a current frame into a trained RFB_Nano feature extraction network model, and using a TensorRT acceleration engine to run the trained RFB_Nano feature extraction network model to obtain a detection result of the real-time image of the road surface in the current frame, wherein the detection result includes a pedestrian detection frame and position information of the pedestrian detection frame; Step 3: determining whether the proportion of the area of ​​the pedestrian detection frame to the area of ​​the entire single-frame real-time image of the road surface exceeds a threshold, and if so, issuing a warning signal, and if not, obtaining the next frame of the real-time image of the road surface and executing Step 2.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to a pedestrian detection method and system based on an improved RFBNet algorithm. Background Art

[0002] Vision-based pedestrian detection technology is a key technology that has been focused on and studied in recent years. Among various sensors, visual sensors have the advantages of richer information, higher accuracy, richer functions and more reasonable costs. Therefore, vision-based pedestrian detection technology is a research hotspot and guides future development trends. It is one of the best choices for studying the field of intelligent driving.

[0003] Regarding the actual deployment of pedestrian detection, because vision-based data processing is usually large data such as pictures and videos, it is deployed in the cloud (that is, the mobile device transmits the generated input data to the cloud, and the cloud processes it and then transmits it back). This method cannot meet the real-time requirements of the automatic driving assistance system, and is prone to problems such as unstable network signals, resulting in major defects in the pedestrian detection system; therefore, with the development of edge computing, the deployment of edge devices on vehicle terminals can realize the direct processing of image data and inference calculations for detection on the edge detection devices.

[0004] In response to the accuracy defects of the SSD (Single Shot MultiBox Detector) algorithm, Liu et al. proposed the RFBNet (Reception Field Block Net) algorithm, which was mainly inspired by human bionics and verified the influence of the size of the receptive field and the size of the eccentricity on improving the robustness of features. However, due to the characteristics of edge devices, factors such as memory space, computing power, power consumption and cost all affect the performance of pedestrian detection. At present, it is difficult to directly apply the target detection model with good performance on edge devices to achieve the ideal effect, and there may be performance problems such as high latency and low frame rate. There are two important measurement indicators for the performance of the target detection model applied on the edge device: the mAP (mean Average Precision) and the FPS (Frame Per Second). Therefore, studying how to balance the performance in terms of accuracy and speed and deploy the pedestrian detection system on the restricted edge devices is a hot issue in current research and has profound practical significance. Summary of the invention

[0005] Purpose of the invention: In order to solve the problem that the existing target detection model is difficult to apply directly on edge devices and has performance issues such as high latency and low frame rate, the present invention proposes a pedestrian detection method and system based on an improved RFB algorithm model, which greatly reduces the amount of calculation while controlling the accuracy and achieves an effective balance between accuracy and real-time performance.

[0006] Technical solution: A pedestrian detection method based on an improved RFBNet algorithm includes the following steps:

[0007] Step 1: Get the real-time image of the road surface in the current frame;

[0008] Step 2: Input the current frame road real-time image into the trained RFB_Nano feature extraction network model, and use the TensorRT acceleration engine to run the trained RFB_Nano feature extraction network model to obtain the detection result of the current frame road real-time image, which includes the pedestrian detection frame and the pedestrian detection frame position information;

[0009] Step 3: Determine whether the ratio of the area of ​​the pedestrian detection frame to the area of ​​the current frame road real-time image exceeds the threshold. If so, issue a warning signal. If not, obtain the next frame road real-time image and execute step 2.

[0010] Further, in step 2, the RFB_Nano feature extraction network model is established according to the following steps:

[0011] The VGG16 skeleton network model is used as the basis of the feature extraction network, and the RFB module is introduced;

[0012] Add a 3*3 convolution layer to the first convolution branch of the RFB module;

[0013] Starting from the third group of convolutional layers, add a 1*1 convolutional layer before the 3*3 convolutional layer in each group of convolutional layers;

[0014] Insert a BN layer before each convolutional layer;

[0015] Using the multi-scale feature fusion mechanism, 4 default boxes are extracted from each unit of the feature map of conv4_3; 4 default boxes are extracted from each unit of the feature map of conv7; 4 default boxes are extracted from each unit of the feature map of conv8_2; 4 default boxes are extracted from each unit of the feature map of conv9_2; 4 default boxes are extracted from each unit of the feature map of conv10_2; 3 default boxes are extracted from each unit of the feature map of conv11_2;

[0016] The number of convolution kernels in each layer is reduced to half of the original number, and one 3*3 convolution layer is removed from the third to fifth convolution layers, and one maximum pooling layer is removed from the fifth convolution layer;

[0017] The category confidence and bounding box position prediction of each default box is performed through 3*3 convolution on conv4_3, conv7, conv8_2, conv9_2, conv10_2, and conv11_2.

[0018] Furthermore, the use of the TensorRT acceleration engine to run the trained RFB_Nano feature extraction network model is specifically performed as follows:

[0019] The trained RFB_Nano feature extraction network model is converted into a network model_trt that supports the TensorRT acceleration engine through the torch2trt converter in the TensorRT acceleration engine.

[0020] Use the TensorRT acceleration engine to run the network model_trt to obtain the detection results of the real-time road image of the current frame.

[0021] Furthermore, the specific training operations of the RFB_Nano feature extraction network model include:

[0022] Set the default box pre-selection box position according to the rules and use MSRA to initialize the network weights;

[0023] The SmoothL1 Loss of the RFB_Nano model is used as the positioning loss, and the class-weighted cross entropy is used as the classification loss. The network weights are updated by forward propagation and back propagation to minimize the positioning loss and classification loss, and a trained RFB_Nano feature extraction network model is obtained.

[0024] Furthermore, for the 3*3 convolutional layer added in the first convolutional branch of the RFB module, its dilated convolution factor is 2.

[0025] The present invention also discloses a pedestrian detection system based on an improved RFBNet algorithm, comprising:

[0026] Video acquisition module, used to acquire real-time video data of the road surface;

[0027] An image processing module is used to divide the real-time road surface video data into single-frame images and perform image preprocessing on the single-frame images to obtain a real-time road surface image of the current frame;

[0028] The pedestrian detection module is used to input the current frame of the real-time road image into the trained RFB_Nano feature extraction network model, and use the TensorRT acceleration engine to run the trained RFB_Nano feature extraction network model to obtain the detection result of the current frame of the real-time road image, which includes the pedestrian detection frame and the pedestrian detection frame position information;

[0029] The early warning module is used to determine whether the proportion of the pedestrian detection frame area to the entire single-frame real-time road image area exceeds a threshold based on the pedestrian detection frame obtained by the pedestrian detection module. If so, an early warning signal is issued; if not, the next frame of real-time road image is obtained and the pedestrian detection module is called.

[0030] Furthermore, the pedestrian detection module includes a trained RFB_Nano feature extraction network model and a TensorRT acceleration engine; the trained RFB_Nano feature extraction network model generates a network model_trt that can support the TensorRT acceleration engine through the torch2trt converter in the TensorRT acceleration engine; the network model_trt is run using the TensorRT acceleration engine to obtain the detection result of the real-time image of the road surface in the current frame.

[0031] Furthermore, the RFB_Nano feature extraction network model structure is:

[0032] The VGG16 skeleton network model is used as the basis of the feature extraction network, and the RFB module is introduced;

[0033] Add a 3*3 convolution layer to the first convolution branch of the RFB module;

[0034] Starting from the third group of convolutional layers, add a 1*1 convolutional layer before the 3*3 convolutional layer in each group of convolutional layers;

[0035] Insert a BN layer before each convolutional layer;

[0036] Using the multi-scale feature fusion mechanism, 4 default boxes are extracted from each unit of the feature map of conv4_3; 4 default boxes are extracted from each unit of the feature map of conv7; 4 default boxes are extracted from each unit of the feature map of conv8_2; 4 default boxes are extracted from each unit of the feature map of conv9_2; 4 default boxes are extracted from each unit of the feature map of conv10_2; 3 default boxes are extracted from each unit of the feature map of conv11_2;

[0037] The number of convolution kernels in each layer is reduced to half of the original number, and one 3*3 convolution layer is removed from the third to fifth convolution layers, and one maximum pooling layer is removed from the fifth convolution layer;

[0038] The category confidence and bounding box position prediction of each default box is performed through 3*3 convolution on conv4_3, conv7, conv8_2, conv9_2, conv10_2, and conv11_2.

[0039] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0040] (1) The method of the present invention greatly reduces the number of network parameters and improves the speed of training and detection by reducing the number of network model channels and convolutions while ensuring reliable detection accuracy;

[0041] (2) The present invention trains the network from scratch by adding a BN (Batch Normalization) layer, using 1*1 convolution, reducing the number of default boxes, and optimizing the loss function. This makes the number of parameters of the network model only about 8.5% of that of the RFBNet algorithm, and the detection accuracy mAP on the test set reaches 80.7%. The detection speed on the server side can reach 112FPS, which is 4.3 times that of the RFBNet algorithm.

[0042] (3) The present invention can be transplanted to the edge device Jetson Nano, and uses the deep learning acceleration library TensorRT to optimize the neural network model, thereby speeding up the pedestrian detection speed deployed on the edge device, while effectively reusing the memory, reducing the amount of calculation, and further improving the detection speed of the pedestrian detection system. Experimental verification shows that the mAP of the RFB_Nano_trt detection model remains unchanged compared to the RFB_Nano detection model, but the FPS increases from 12.8 to 31.6, an increase of nearly 1.5 times. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Detection model for RFB_Nano;

[0044] Figure 2 This is the workflow diagram of the pedestrian detection system;

[0045] Figure 3 Training flow chart for pedestrian detection model;

[0046] Figure 4 This is the pedestrian detection flow chart of the RFB_Nano algorithm;

[0047] Figure 5 This is a flow chart for determining the system warning level;

[0048] Figure 6 This is a schematic diagram of the RFB module;

[0049] Figure 7 Schematic diagram of the VGG16-3 network model structure. DETAILED DESCRIPTION

[0050] The technical solution of the present invention is now further described in conjunction with the accompanying drawings and embodiments.

[0051] See also Figure 2 , a pedestrian detection method based on an improved RFBNet algorithm and a TensorRT acceleration framework of the present invention comprises the following steps:

[0052] Step 1: Connect the camera correctly, process the video input through the camera driver V4L2 and the video codec component Gstreamer, and use OpenCV to split the real-time video image into single frame images;

[0053] Step 2: Load the trained RFB_Nano feature extraction network model, use the TensorRT acceleration engine to perform model inference, and obtain the detection results of a single frame image;

[0054] Step 3: Based on the detection results obtained in step 2, a judgment is made; when a pedestrian target appears in front of the vehicle's driving area and the proportion of the pedestrian detection box size to the entire input image area exceeds the threshold, a warning signal is issued; the specific process can be seen in Figure 5 ;

[0055] Step 4: Process the next frame of image and repeat steps 2 to 4.

[0056] The establishment of the RFB_Nano feature extraction network model mentioned in step 2 is now further explained.

[0057] The RFB_Nano proposed in the present invention is based on the RFBNet model, which is an improvement based on SSD. The improvement of RFBNet over SSD is the addition of the RFB module (multi-branch convolution and dilated convolution).

[0058] The improvement of RFB_Nano proposed in the present invention over RFBNet is that the feature extraction network VGG16 is model compressed, reduced to one RFB module, 1*1 convolution and batch normalization BN layers are added, and the classification loss function is improved because the ratio between objects such as pedestrians and the background is small. Among them, the multi-scale feature fusion of RFB_Nano is consistent with the mechanism in RFBNet, but the difference is that RFBNet takes 6, 6, 6, 6, 4, 4 default boxes for each unit on the feature maps of conv4_3, conv7, conv8_2, conv9_2, conv10_2, and conv11_2, respectively, while RFB_Nano takes 4, 4, 4, 4, 4, 3 default boxes on the feature maps of the same layer, respectively, reducing the number of parameters while ensuring that the accuracy loss is not too much.

[0059] The improvements of RFB_Nano proposed in the present invention are now further described.

[0060] In the RFB_Nano model proposed in this invention, the RFB module is introduced and the first convolution branch is improved, such as Figure 6 As shown in the figure, a 3*3 convolution layer is added, and the original 3*3 hole convolution factor is changed from 1 to 2.

[0061] The feature extraction network of the RFB_Nano detection model proposed in this invention is designed based on the improved VGG16 skeleton network. Figure 7 As shown in the figure, the VGG16 skeleton network consists of 5 groups of 13 convolutional layers, 3 fully connected layers and 5 pooling layers (the pooling layer has no parameters to be learned, so it is not counted in the number of network layers). Since a 5*5 convolution can be realized by two 3*3 convolutions in series, and the receptive field size is still 5*5, the entire VGG16 network uses 3*3 convolution kernels of the same size for stacking. This connection method not only reduces the number of parameters in the network model, but also the multi-layer ReLU activation function makes the network have more nonlinear changes, thereby making the CNN feature extraction ability stronger; in addition, a 2*2 maximum pooling layer is used after each group of convolutional layers to reduce the size of the feature map and reduce the amount of calculation; the network also has 3 fully connected layers at the end, of which the first two fully connected layers each contain 4096 neurons. Through the added dropout mechanism, some fully connected layer nodes are randomly dropped to prevent overfitting. The last fully connected layer outputs the probability of each category in the image, and the final classification result of the image is the category with the highest probability.

[0062] However, since the number of parameters of the VGG16 network model is as high as over 100 million and it occupies about 500M of memory space, in order to enable the pedestrian detection model to achieve the best performance on the edge device, it is necessary to improve the detection model and reduce the number of parameters and calculations without sacrificing accuracy, so as to achieve the goal of fast and accurate detection.

[0063] The RFB_Nano detection model proposed in the present invention first compresses the feature extraction network. The idea of ​​the SSD algorithm is continued in RFBNet, and the fully connected layer in VGG16 is replaced by a convolutional layer, which will greatly improve the speed. According to the network characteristics that the feature extraction network contains a large number of convolutional layers and each group of convolutions contains a large number of convolutional kernels, the present invention adopts the method of reducing the number of network channels (i.e., the number of convolutional kernels) and the number of convolutional layers: the number of convolutional kernels in each layer is reduced to half of the original number, and the maximum number of convolutional kernels is 256; and a 3*3 convolutional layer is removed from each of the convolutional layers of the third to fifth groups, and a maximum pooling layer is also removed from the fifth group. Such a model reduction method greatly reduces the number of model parameters, providing convenience and possibility for subsequent further improvements to the network model.

[0064] In the RFB_Nano model of the present invention, starting from the third group of convolutional layers, a 1*1 convolutional layer is added before the 3*3 convolutional layer in each group of convolutional layers; 1*1 convolution is a convolution with a convolution kernel size of 1. The 1*1 convolution is added after the conventional convolution to deepen and widen the network structure and improve the abstract representation of feature extraction of the network. Generally speaking, for a convolution kernel with a size of n*n (n>1), the number of edge 0s and the step length of each convolution movement are usually considered, but when the convolution kernel size is 1*1, the number of convolution kernels can be controlled according to specific needs, thereby achieving dimensionality reduction (increase); the convolution process of the 1*1 convolution kernel is equivalent to the calculation process of the fully connected layer, and a nonlinear activation function is also added through the 1*1 convolution layer, thereby increasing the nonlinearity of the network, so that the network can express more complex features; in common detection models, since more convolution operations are required, the amount of calculation is very large, so the amount of calculation can be reduced while ensuring accuracy by introducing 1*1 convolution.

[0065] A BN (Batch Normalization) layer is inserted before each layer input of the RFB_Nano model. The main principle of the BN layer is to pre-process the input data of each layer to normalize it before entering the next layer of the network. In the RFB_Nano detection model proposed in the present invention, on the one hand, since the BN layer has the characteristics of fast training convergence, a larger initial learning rate is selected, and the learning rate decays quickly, which greatly improves the training speed and reduces the dependence on initialization compared to RFBNet; on the other hand, since the BN layer has the characteristics of improving the generalization ability of the network, the drop out layer and the selection of parameters are omitted.

[0066] A multi-scale feature fusion mechanism is adopted to take 4, 4, 4, 4, 4, 3 default boxes for each unit on the feature maps of conv4_3, conv7, conv8_2, conv9_2, conv10_2, and conv11_2, respectively. Among them, conv4_3 represents the third convolution kernel in the fourth group of convolution layers; conv7 represents the seventh group of convolution layers; conv8_2 represents the second convolution kernel in the eighth group of convolution layers; conv9_2 represents the second convolution kernel in the ninth group of convolution layers; conv10_2 represents the second convolution kernel in the tenth group of convolution layers; and conv11_2 represents the second convolution kernel in the eleventh group of convolution layers.

[0067] The number of convolution kernels in each layer of the RFB_Nano model is reduced to half of the original number, and the maximum number of convolution kernels is 256; and one 3*3 convolution layer is removed from each of the third to fifth convolution layers, and one maximum pooling layer is removed from the fifth convolution layer. The present invention achieves compression of the feature extraction network by reducing the network channels (i.e., the number of convolution kernels) and the number of convolution layers.

[0068] For the positioning loss function, the SmoothL1 Loss of the RFB_Nano model is used:

[0069]

[0070]

[0071] Among them, x is the input.

[0072] Since the background is dominant in the object detection problem, it is necessary to deal with the serious imbalance between different categories. In image classification tasks, the only possible error is the misclassification between foreground categories, but in object detection tasks, errors often occur in the distinction between foreground and background, rather than in the classification between several types of objects in the foreground. In pedestrian detection, since objects such as pedestrians account for a small proportion of the background, in order to solve this problem, class-weighted cross entropy is used as the classification loss:

[0073]

[0074] Where, j = 0, 1, 2, ..., K-1; K is the number of categories; S j is the jth value of the softmax output vector S, which indicates the probability that the sample belongs to the jth category; wc is the weight; y i It is a 1*T vector, where only the value of the position corresponding to the true label is 1, and the other T-1 values ​​are 0. Use a larger weight in the background class and a relatively smaller weight in other classes, for example, the weight used for the background class is 1.5 and the weight used for the pedestrian class is 1.

[0075] The category confidence and bounding box position of each default box are predicted through 3*3 convolution on the six feature layers of conv4_3, conv7, conv8_2, conv9_2, conv10_2, and conv11_2. Then the entire network is converted into a network model_trt that supports TensorRT through the torch2trt converter in TensorRT. The accelerated pedestrian detection network model is directly run using the TensorRT acceleration engine, and pedestrian information is output in real time.

[0076] The detection method of the present invention is further described by way of examples.

[0077] This embodiment uses a USB camera (resolution of 640*480, frame rate of 30fps) for video acquisition, driven by V4L2 (Video for Linux 2). Secondly, the Gstreamer component is used to accelerate the encoding, decoding and output process of the video stream, and minimize the delay of the pedestrian detection task.

[0078] After the prepared training set samples are input into the network, RFB_Nano will perform image preprocessing, mainly for image enhancement, including cropping, random left and right flipping, color change, whitening, etc. The main purpose is to make the algorithm more robust to input targets of different sizes and shapes.

[0079] See also Figure 3, the preprocessed data is put into the RFB_Nano feature extraction network. First, the pedestrian detection model is initialized by setting the default box pre-selection box position according to the rules and using MSRA to initialize the network weights. Then, the training process of updating the network weights by forward propagation and back propagation is carried out to minimize the positioning loss and the category confidence loss as much as possible, so as to obtain the pedestrian detection network model. After that, the trained RFB_Nano feature extraction network model is put into the validation set samples, and the samples are image preprocessed. The forward propagation of the validation set is carried out to evaluate the detection accuracy of the pedestrian detection model.

[0080] See also Figure 4 First, the unlabeled data set is subjected to certain image preprocessing, including resizing, mean-reduced kernel channel swapping, and after passing through the feature extraction network, the category confidence and bounding box position of each default box are predicted by 3*3 convolution on the six feature layers of conv4_3, conv7, conv8_2, conv9_2, conv10_2, and conv11_2. Each default box predicts the confidence of each category, and the categories of the present invention include pedestrians and background categories; the coordinate information of the predicted bounding box corresponding to the default box, the coordinate information includes the center coordinates and width and height. For each predicted box obtained, the maximum value of its category confidence is taken. If the maximum value is greater than the confidence threshold, the category corresponding to the maximum value is the category of the predicted box, otherwise the box is filtered out; for the retained predicted box, its true position parameters are decoded according to its corresponding priori box, and then it is sorted in descending order according to the confidence of the category to which it belongs, and the top-k predicted boxes are taken. Finally, NMS is performed to filter out the predicted boxes with large overlap, and finally the detection results are obtained.

[0081] Then the entire network is converted into a network model_trt that supports TensorRT through the torch2trt converter in TensorRT, and the TensorRT acceleration engine is used to directly run the accelerated pedestrian detection network model and perform real-time detection and output of pedestrian information.

[0082] The network training data set used in this embodiment has a total of 117,000 images, and the validation set uses 2,097 human test sets of VOC2007, 3,220 images of BDD100K, and 683 self-made data sets, totaling 6,000 images. The public data sets and self-made data sets are not homologous in scene regions, etc., which can better test the performance of the system in non-homologous scenes.

Claims

1. A pedestrian detection method based on an improved RFBNet algorithm, characterized in that: The following steps are involved: Step 1: Get the real-time image of the road surface in the current frame; Step 2: Input the current frame road real-time image into the trained RFB_Nano feature extraction network model, and use the TensorRT acceleration engine to run the trained RFB_Nano feature extraction network model to obtain the detection result of the current frame road real-time image, which includes the pedestrian detection frame and the pedestrian detection frame position information; Step 3: Determine whether the ratio of the area of ​​the pedestrian detection frame to the area of ​​the current frame road real-time image exceeds the threshold. If so, issue a warning signal. If not, obtain the next frame road real-time image and execute step 2. In step 2, the RFB_Nano feature extraction network model is established according to the following steps: The VGG16 skeleton network model is used as the basis of the feature extraction network, and the RFB module is introduced; Add a 3*3 convolution layer to the first convolution branch of the RFB module; Starting from the third group of convolutional layers, add a 1*1 convolutional layer before the 3*3 convolutional layer in each group of convolutional layers; Insert a BN layer before each convolutional layer; Using a multi-scale feature fusion mechanism, 4 default boxes are extracted from each unit of the feature map of conv4_3; 4 default boxes are extracted from each unit of the feature map of conv7; 4 default boxes are extracted from each unit of the feature map of conv8_2; 4 default boxes are extracted from each unit of the feature map of conv9_2; 4 default boxes are extracted from each unit of the feature map of conv10_2; 3 default boxes are extracted from each unit of the feature map of conv11_2; The number of convolution kernels in each layer is reduced to half of the original number, and one 3*3 convolution layer is removed from the third to fifth convolution layers, and one maximum pooling layer is removed from the fifth convolution layer; The category confidence and bounding box position prediction of each default box is performed through 3*3 convolution on conv4_3, conv7, conv8_2, conv9_2, conv10_2, and conv11_2.

2. The pedestrian detection method based on the improved RFBNet algorithm according to claim 1, characterized in that: The specific operation of using the TensorRT acceleration engine to run the trained RFB_Nano feature extraction network model is as follows: The trained RFB_Nano feature extraction network model is converted into a network model_trt that supports the TensorRT acceleration engine through the torch2trt converter in the TensorRT acceleration engine. Use the TensorRT acceleration engine to run the network model_trt to obtain the detection results of the real-time road image of the current frame.

3. The pedestrian detection method based on the improved RFBNet algorithm according to claim 1, characterized in that: The specific training operations of the RFB_Nano feature extraction network model include: Set the default box pre-selection box position according to the rules and use MSRA to initialize the network weights; The SmoothL1 Loss of the RFB_Nano model is used as the positioning loss, and the class-weighted cross entropy is used as the classification loss. The network weights are updated by forward propagation and back propagation to minimize the positioning loss and classification loss, and a trained RFB_Nano feature extraction network model is obtained.

4. The pedestrian detection method based on the improved RFBNet algorithm according to claim 1, characterized in that: For the 3*3 convolutional layer added in the first convolutional branch of the RFB module, its dilated convolution factor is 2.

5. A pedestrian detection system based on the improved RFBNet algorithm, characterized by: include: Video acquisition module, used to acquire real-time video data of the road surface; An image processing module is used to divide the real-time road surface video data into single-frame images and perform image preprocessing on the single-frame images to obtain a real-time road surface image of the current frame; The pedestrian detection module is used to input the current frame of the real-time road image into the trained RFB_Nano feature extraction network model, and use the TensorRT acceleration engine to run the trained RFB_Nano feature extraction network model to obtain the detection result of the current frame of the real-time road image, which includes the pedestrian detection frame and the pedestrian detection frame position information; The early warning module is used to determine whether the ratio of the pedestrian detection frame area to the entire single-frame road real-time image area exceeds a threshold based on the pedestrian detection frame obtained by the pedestrian detection module. If so, an early warning signal is issued. If not, the next frame of road real-time image is obtained and the pedestrian detection module is called. The pedestrian detection module includes a trained RFB_Nano feature extraction network model and a TensorRT acceleration engine; the trained RFB_Nano feature extraction network model generates a network model_trt that can support the TensorRT acceleration engine through the torch2trt converter in the TensorRT acceleration engine; the network model_trt is run using the TensorRT acceleration engine to obtain the detection result of the real-time image of the road surface in the current frame; The RFB_Nano feature extraction network model structure is: The VGG16 skeleton network model is used as the basis of the feature extraction network, and the RFB module is introduced; Add a 3*3 convolution layer to the first convolution branch of the RFB module; Starting from the third group of convolutional layers, add a 1*1 convolutional layer before the 3*3 convolutional layer in each group of convolutional layers; Insert a BN layer before each convolutional layer; Using a multi-scale feature fusion mechanism, 4 default boxes are extracted from each unit of the feature map of conv4_3; 4 default boxes are extracted from each unit of the feature map of conv7; 4 default boxes are extracted from each unit of the feature map of conv8_2; 4 default boxes are extracted from each unit of the feature map of conv9_2; 4 default boxes are extracted from each unit of the feature map of conv10_2; 3 default boxes are extracted from each unit of the feature map of conv11_2; The number of convolution kernels in each layer is reduced to half of the original number, and one 3*3 convolution layer is removed from the third to fifth convolution layers, and one maximum pooling layer is removed from the fifth convolution layer; The category confidence and bounding box position prediction of each default box is performed through 3*3 convolution on conv4_3, conv7, conv8_2, conv9_2, conv10_2, and conv11_2.

Images