A deep learning-based road target detection system, method and storage medium

Abstract

This invention claims protection for a deep learning-based road target detection system, method, and storage medium, belonging to the field of autonomous driving environmental perception. The method uses a Faster R-CNN neural network structure, employing a Residual Neural Network (ResNet) as the backbone for image feature extraction; a Feature Pyramid Network (FPN) as the neck for feature enhancement and connecting the backbone and head networks; and a Region Generation Network (RPN) as the head for generating multiple region candidate bounding boxes. Region of Interest (ROI) pooling is used to output feature maps of the same size, which are then input to a fully connected layer. Classification and regression are performed, using Softmax to classify targets and a bounding box regressor to predict and correct target bounding boxes. The method iteratively trains on a road target detection dataset, optimizing the loss to generate a more ideal target detection model. This invention proposes an attention-based upsampling method to replace the nearest neighbor interpolation method, which can effectively convey multi-scale target semantics and improve the accuracy of small target detection.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving environmental perception, and specifically relates to a road target detection method based on deep learning. Background Technology

[0002] The rise of artificial intelligence and machine learning has greatly propelled the development of computer vision technology. Computer vision technology is widely used in scenarios such as video surveillance and security, medical diagnosis, and autonomous driving. Examples include detecting anomalies in surveillance videos; medical image diagnosis, such as CT scans and X-rays; and real-time perception of the surrounding environment for autonomous driving, such as roads, vehicles, and pedestrians. Among these, environmental perception for autonomous driving is a hot research area for automotive companies, internet companies, and numerous universities. Autonomous driving environmental perception primarily involves acquiring real-time environmental information about the vehicle's surroundings through sensors such as cameras, LiDAR, and millimeter-wave radar, processing this information, and sending it to the decision center for more accurate and reliable control decisions.

[0003] Currently, in the field of autonomous driving environmental perception, target detection technology is relatively mature. Numerous methods based on cameras, LiDAR, millimeter-wave radar, and multi-source sensor fusion have been applied in real-world vehicles. Among these, camera-based target detection methods are relatively simple and low-cost. Currently, camera-based road target detection mainly uses deep learning neural network methods to extract road target features from real-time images captured by onboard cameras, thereby identifying and classifying targets. Subsequently, filtering algorithms are used to track the state of the detected target in real time, sending the target category and target state (relative longitudinal and lateral distances to the driving vehicle, relative speed, etc.) to the controller for appropriate judgment and to make accurate control decisions.

[0004] CN114648745A discloses a deep learning-based road detection system. This system includes acquiring a raw image dataset of roads, labeling road information in the image dataset to obtain a training image dataset containing first and second ground truth bounding boxes for each road; inputting the training image dataset into a road detection model to obtain first and second road detection boxes; and training the road detection model based on the loss values ​​between the first and second road detection boxes and their corresponding ground truth bounding boxes. This enables road detection from the image data to be detected. This invention trains the road detection model by calculating loss values ​​between the detection boxes corresponding to lane lines and stop lines, and the detection boxes corresponding to road signs, and their corresponding ground truth bounding boxes, simultaneously distinguishing drivable roads and improving road detection accuracy.

[0005] This patented deep learning-based road detection system uses an anchor-based algorithm for the first road detection and an anchor-free algorithm for the second road detection. While the anchor-free algorithm is more flexible and has fewer parameters, it is prone to problems such as extreme imbalance between positive and negative samples and semantic ambiguity. This invention, however, chooses an anchor-based deep learning algorithm, which involves two rounds of positive and negative sample selection, thus alleviating the problem of imbalanced positive and negative samples. Furthermore, through improvements to FPN, the transmission and fusion of semantic information are greatly enhanced. Summary of the Invention

[0006] This invention aims to solve the problems of the prior art mentioned above. It proposes a road target detection system based on deep learning. The technical solution of this invention is as follows:

[0007] A deep learning-based road target detection system includes: a backbone network, a neck network, a head network, a region of interest pooling layer, and a classification and regression module, wherein:

[0008] The backbone network is used to extract features from road images captured by vehicle-mounted cameras to obtain road image feature maps.

[0009] The neck network is used to enhance the feature maps of road images and connects the backbone network and the head network;

[0010] The head network uses a Region Proposal Network (RPN) to generate candidate bounding boxes for the target region;

[0011] The region-of-interest pooling layer is used to pool the candidate feature maps into regions of interest, outputting a fixed-size feature map that is then fed into the fully connected layer.

[0012] The classification and regression module is used to perform Softmax classification on the output targets of the fully connected layer and to perform regression correction on the target bounding boxes.

[0013] Furthermore, the backbone network adopts either a VGG network or a ResNet network, with ResNet network selected for image feature extraction.

[0014] Furthermore, the neck network employs a Feature Pyramid Network (FPN) for feature enhancement, and improves the upsampling of the nearest neighbor interpolation method in the FPN network by using an attention-based upsampling method, thereby enhancing the effective extraction of multi-scale target semantic information.

[0015] Furthermore, the improvement of the nearest neighbor interpolation method in the FPN network using an attention-based upsampling method specifically includes:

[0016] S1. Input the feature map into three 1×1 convolutions to obtain three feature maps Q, K, and V of the same scale;

[0017] S2. Apply bilinear interpolation to feature map Q and upsample it by a factor of 2. Apply transpose convolution to feature maps K and V and upsample them by a factor of 2. This will yield feature maps Q and V respectively. up ,K up V up ;

[0018] S3, for feature map Q up ,K up Perform dot product operation and then normalize using Softmax to obtain the weight matrix for each pixel in the feature map;

[0019] S4. Then combine the weight matrix with V up Perform a dot product operation to obtain a feature map with rich information.

[0020] Furthermore, the head network employs a Region Proposal Network (RPN) to generate candidate bounding boxes for the detected target, specifically including:

[0021] S1. RPN first generates 9 anchor boxes for each pixel of each feature map fused by FPN, that is, a total of h×w×9 initial anchor boxes are generated.

[0022] S2. Then determine whether each anchor box belongs to the target or the background. Specifically, positive anchor boxes are extracted using Softmax.

[0023] S3. Then, the bounding box regressor learns linear regression between the positive anchor boxes and the ground truth bounding boxes to generate the final candidate bounding boxes for the regions.

[0024] Furthermore, the classification and regression module uses the Softmax function to classify the target output of the fully connected layer, and uses a bounding box regressor to perform regression prediction and correction of the target detection box.

[0025] A road target detection method based on any one of the systems described above, comprising the following steps:

[0026] S1. The driving vehicle collects real-time video of the road scene through the camera. Then, Python and OpenCV are used to cut out each frame of the video and save it in sequence as images in a specified format. The target in the saved road scene image, including vehicles and pedestrians, is labeled with the category by the labeling software to generate a label file. The images and labels correspond one-to-one and are divided into training set, test set and validation set.

[0027] S2. Input the images and corresponding labels of the training set into the backbone network ResNet. Through multi-layer convolution, pooling, residual operations, multi-level multi-scale feature maps {C2,C3,C4,C5} are generated. C2,C3,C4,C5 represent the 2nd, 3rd, 4th, and 5th layer feature maps extracted from the road scene image during the multi-layer convolution, pooling, and activation process of the multi-ResNet network.

[0028] S3. Input the multi-level, multi-scale feature maps {C2, C3, C4, C5} into the FPN neck network based on attention-based upsampling, and perform corresponding lateral connection operations and attention-based upsampling operations. The lateral connection operation is a 1×1 convolution, which processes {C2, C3, C4, C5} into {F2, F3, F4, F5}. F2, F3, F4, and F5 are the multi-level, multi-scale feature maps after feature fusion through 1×1 convolution and attention-based upsampling, respectively, and they correspond one-to-one with {C2, C3, C4, C5}. The top-level feature map is F5 = Conv1×1(C5), and the remaining layers are represented as F... i =Conv1×1(C i )+Att_upsampe(F i+1 ), i = 4, 3, 2. Where Conv1×1 represents a 1×1 convolution, Att_upsampe represents attention-based upsampling, and i represents the corresponding layer number of the feature map, such as F i+1 →F i When i = 3, it means that the semantic information of the fourth layer feature map is passed to the third layer feature map;

[0029] S4. Pass {F2,F3,F4,F5} through Conv3×3 to obtain {P2,P3,P4,P5}. P2,P3,P4,P5 represent the fused feature maps of layers 2, 3, 4, and 5 generated after 3×3 convolution to alleviate the aliasing effect, respectively. Then, input {P2,P3,P4,P5} into the RPN head network to generate candidate bounding boxes for multi-scale targets.

[0030] S5. Perform Region of Interest Pooling (RoI Pooling) on ​​the candidate feature maps of the regions, convert them to the same size, and input them into the fully connected layer;

[0031] S6. After passing through the output of two fully connected layers, perform Softmax classification and bounding box regression correction on the detected target;

[0032] S7. The road target dataset processed in step S1 is trained iteratively from step S2 to S6 using a deep learning experimental platform to generate a road target detection model and verify the detection effect of the model.

[0033] S8. Deploy the road target detection model to a real vehicle and conduct a real-time road target detection experiment.

[0034] A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the road target detection method as described.

[0035] A computer program product includes a computer program that, when executed by a processor, implements the road target detection method as described.

[0036] The advantages and beneficial effects of this invention are as follows:

[0037] This invention provides a deep learning-based road target detection method. The overall network structure adopts Faster R-CNN, in which the backbone network uses ResNe residual network to improve classification accuracy; the neck network uses FPN based on attention mechanism upsampling, which can solve to some extent the problem that the nearest neighbor interpolation upsampling method of traditional FPN cannot effectively convey the deep semantic information of multi-scale targets; the head network uses RPN to generate region candidate boxes, which are then fed into the fully connected layer through region of interest pooling for Softmax target classification and target bounding box regression correction.

[0038] This invention mainly employs an FPN neck network based on attention mechanism upsampling in the Faster R-CNN network structure, which can effectively transmit deep semantic information of multi-scale targets, thereby improving the detection accuracy of small targets in road scenes.

[0039] FPN has been applied in many computer vision tasks, so the improvement of FPN in this invention is challenging. After replacing the upsampling of the nearest neighbor interpolation method with upsampling based on the attention mechanism, the road scene image dataset was trained and validated, and compared with the unimproved FPN. The average accuracy for small target detection was improved by 2%, and the overall accuracy was improved by 1.4%. Attached Figure Description

[0040] Figure 1 This is a flowchart of a preferred embodiment of the road target detection based on deep learning provided by the present invention;

[0041] Figure 2 This is a diagram of the object detection structure based on Faster R-CNN of the present invention;

[0042] Figure 3 This is the attention-based upsampling method of the present invention. Detailed Implementation

[0043] The technical solutions of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0044] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0045] This invention provides a road target detection method based on deep learning, the process of which is as follows: Figure 1 As shown, this invention aims to improve the detection accuracy of multi-scale targets (especially small targets) in road scenes within the field of autonomous driving environmental perception. Based on the Faster R-CNN neural network structure, the Feature Pyramid Network (FPN) is used to perform multi-scale feature fusion on acquired road images. However, the use of simple nearest neighbor interpolation as a top-down feature upsampling process cannot effectively convey deep semantic information, leading to poor detection accuracy for small targets. To address this issue, this invention proposes an FPN network based on attention mechanism upsampling, which can learn more deep semantic information and effectively perform multi-scale feature fusion, thereby improving the detection accuracy of multi-scale targets.

[0046] This invention provides a road target detection method based on deep learning. This method primarily uses a Faster R-CNN neural network structure, as shown in the following diagram. Figure 2 As shown, it mainly includes:

[0047] The backbone network is used for feature extraction from road images captured by vehicle-mounted cameras;

[0048] The neck network is used to enhance the feature maps and connect the backbone network and the head network.

[0049] The head network is used to generate candidate bounding boxes for the target.

[0050] The region of interest pooling layer is used to perform region of interest pooling (RoI Pooling) on ​​the candidate feature map, outputting a fixed-size feature map that is then fed into the fully connected layer.

[0051] The classification and regression module is used to perform Softmax classification of the target based on the output of the fully connected layer and to regress and correct the target bounding box.

[0052] Autonomous driving systems typically consist of three layers: perception, decision-making, and execution. Sensors acquire environmental information about the vehicle's surroundings, extracting richer and more accurate features, which are then fed into the decision-making layer for judgment and the formulation of corresponding control strategies. These strategies are then sent to the execution layer for motion control of the vehicle. The perception layer is particularly crucial, as the accuracy of its perception of the vehicle's surroundings determines subsequent decisions and motion control. Roads are generally complex environments, containing various vehicles, pedestrians, traffic signs, and more. Therefore, this invention provides a deep learning-based road target detection method, which effectively conveys deep semantic information about multi-scale targets, thereby improving the detection accuracy of small targets.

[0053] This invention provides a road target detection method based on deep learning, the main steps of which include:

[0054] S1. Vehicles capture real-time video of road scenes via cameras. Python and OpenCV are then used to extract each frame of the video and save them sequentially as images in a specified format. Labeling software can be used to categorize vehicles, pedestrians, and other targets in the saved road scene images, generating label files. Images and labels are matched one-to-one, and the datasets are divided into training, testing, and validation sets.

[0055] S2. Input the images and corresponding labels of the training set into the backbone network ResNet, and generate multi-level, multi-scale feature maps, namely {C2,C3,C4,C5}, through multi-layer convolution, pooling, residual and other operations.

[0056] S3. Input the multi-level, multi-scale feature maps {C2, C3, C4, C5} into the FPN neck network based on attention mechanism upsampling, and perform corresponding lateral connection (1×1 convolution) operations and attention mechanism upsampling operations, that is, process {C2, C3, C4, C5} into {F2, F3, F4, F5}, that is, the top layer feature map is F5 = Conv1×1(C5), and the remaining layers can be represented as F i =Conv1×1(C i )+Att_upsampe(F i+1 ), i = 4, 3, 2, 1.

[0057] Lateral connections (1×1 convolutions) reduce the dimensionality of multi-level, multi-scale feature maps {C2, C3, C4, C5} to reduce computational overhead; while attention-based upsampling... Figure 3 As shown, a 1×1 convolution operation is performed on the top feature map P5 to generate three feature maps Q, K, and V, as shown in Equation (1).

[0058]

[0059] The weights are for the 1×1 convolution operation. Next, linear interpolation and transpose convolution are performed on Q, K, and V respectively, as shown in Equation (2).

[0060]

[0061] In equation (2), S = 2, meaning that Q, K, and V are upsampled by a factor of 2. Then, the correlation between pixels is learned through an attention mechanism.

[0062]

[0063] Y = A·V (4)

[0064] The dot product of Q and K is performed, and then the attention weights A are obtained by normalization using the Softmax function. A is then multiplied by V to obtain the final output Y. That is, the multi-scale feature fusion feature map {F2,F3,F4,F5} can be calculated by lateral connection (1×1 convolution) and attention-based upsampling (Att_upsample), as shown in equations (5) and (6).

[0065] F5=Conv1×1(C5) (5)

[0066] F i =Conv1×1(C i )+Att_upsampe(F i+1 ), i = 4, 3, 2, 1 (6)

[0067] S4. Obtain {P2,P3,P4,P5} from {F2,F3,F4,F5} through Conv3×3 to reduce aliasing. Then, input {P2,P3,P4,P5} into the RPN head network. First, generate 9 anchor boxes for each pixel, resulting in a total of h×w×9 anchor boxes. Then, select the anchor boxes that are close to the ground truth boxes through non-maximum suppression. These are called region candidate bounding boxes.

[0068] S5. Input the region candidate bounding box into the region of interest pooling layer, perform interest pooling on the region candidate feature map to generate a fixed-size feature map, and input it into the fully connected layer.

[0069] S6. The output of the two fully connected layers is used to perform Softmax classification and bounding box regression correction on the detected target.

[0070] S7. The road target dataset processed in step S1 is iteratively trained using steps S2-S6 through a deep learning experimental platform to reduce the category loss and regression loss of the detected targets, thereby generating a road target detection model and verifying the detection effect of the model. That is, the loss function of the Faster R-CNN network structure can be expressed as Equation (7).

[0071]

[0072] In the above formula, i represents the border index, and p i Represents the Softmax probability of the positive class. Let t represent the true predicted probability, and t represent the predicted bounding box. * λ represents the true bounding box corresponding to the positive class anchor box, and λ is the balance between the class loss and the regression loss. The regression loss uses smooth L1 Loss, and the calculation formulas are shown in (8) and (9).

[0073]

[0074]

[0075] S8. Deploy the road target detection model to a real vehicle and conduct real-time road target detection experiments.

[0076] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0077] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0078] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0079] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A deep learning-based road object detection system, characterized by, The method comprises the following steps: The backbone network is used for feature extraction of a road image collected by a vehicle-mounted camera, and a road image feature map is obtained. The neck network is used for feature enhancement of the road image feature map and connects the backbone network and the head network. The head network adopts a region proposal network (RPN) to generate a target region candidate frame. The region of interest pooling layer is used for region of interest pooling of the candidate feature map, and a fixed-size feature map is output and input into a fully connected layer. The neck network adopts a feature pyramid network (FPN) for feature enhancement, and improves the nearest neighbor interpolation upsampling in the FPN network based on an attention mechanism upsampling method, thereby improving the effective extraction of multi-scale target semantic information. The classification and regression module is used to perform classification and regression on the output target of the fully connected layer. Classification and regression correction of the target bounding box; The nearest neighbor interpolation upsampling in the FPN network is improved based on the attention mechanism upsampling method, and the improvement specifically comprises: The backbone network adopts a VGG network or a ResNet network, and the ResNet network is selected for image feature extraction. top-level feature maps performing convolution operations, generating three feature maps, as shown in equation (1); (1) For the weights of the convolution operation; second, the linear interpolation and transposed convolution are performed separately, as shown in equation (2); linear interpolation and transposed convolution are performed separately, as shown in equation (2); (2) in formula (2) i.e., 2-fold up-sampling is performed on Then, the correlation between the pixel points is learned through the attention mechanism. (3) (4) and as shown in equations (5) and (6):​​​​​​​ (5) (6)。 2.The road object detection system based on deep learning of claim 1, wherein, The head network adopts the RPN to generate a region candidate frame for a detected target, and the generation specifically comprises: 3.The road object detection system based on deep learning of claim 1, wherein, S3, the positive anchor frame and the real boundary frame are linearly regressed by the frame regressor, and a final region candidate frame is generated. S1, the RPN first generates 9 anchor boxes for each pixel point of each feature map fused by the FPN, that is, a total of initial anchor boxes are generated; S2, then determine whether each anchor frame belongs to the target or the background, specifically through extracting the positive anchor frame; The method comprises the following steps: 4.The deep learning based road object detection system of claim 1, wherein, The classification regression module adopts The function classifies the target by the output of the full connection layer, adopts the bounding box regressor to regress and predict the target detection box, and corrects.

5. A method of road object detection based on the system of any one of claims 1-4, characterized in that, S1, a vehicle in motion collects a road scene video in real time through a camera, then each frame of the video is cut out and saved as a picture in a specified format in sequence by using Python and OpenCV, and a target in the saved road scene image, such as a vehicle and a pedestrian, is labeled in class by using a labeling software, thereby generating a label file, the image and the label file correspond to each other, and the image is divided into a training set, a test set and a verification set; S5, the region candidate feature map is subjected to region of interest pooling (RoI Pooling) and converted into the same size, and then input into a fully connected layer; S2, input the images and corresponding labels of the training set into the backbone network ResNet, generate multi-level multi-scale feature maps through multi-layer convolution, pooling, residual operation, etc. , respectively represent the 2nd, 3rd, 4th and 5th layer feature maps extracted during the multi-ResNet network multi-layer convolution, pooling and excitation process of the road scene image. S3, Multi-level, multi-scale feature maps The input is fed into the attention-based upsampling FPN neck network, where corresponding lateral connection operations and attention-based upsampling operations are performed. The lateral connection operation is... Convolution, Processed as , Then they are respectively after The multi-level, multi-scale feature map resulting from the fusion of upsampled features obtained through convolution and attention mechanisms, and its relationship with... One-to-one correspondence; top-level feature map is The remaining layers are represented as ;in express convolution, This indicates attention mechanism upsampling. This indicates the corresponding layer number of the feature map, such as... , When this happens, it means that the semantic information of the fourth layer feature map is passed to the third layer feature map; S4, will pass get , They represent the process. The final fused feature maps generated after convolution to mitigate aliasing are then processed. The input is fed into the RPN head network to generate candidate bounding boxes for multi-scale targets. S7, the road target data set processed in step S1 is iteratively trained by steps S2-S6 through a deep learning experiment platform, so as to generate a road target detection model, and the detection effect of the model is verified; S6, the output of the two full connection layers is used to classify and correct the bounding box of the detection target ​ S8, the road target detection model is deployed to a real vehicle, and a real-time road target detection experiment is performed. The computer program is executed by the processor to realize the road target detection method of claim 5.

6. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to realize the road target detection method of claim 5.

7. A computer program product comprising a computer program, characterized in that, ​

Images