A new energy vehicle target detection method based on depth perception

By constructing a depth perception detection model in the target detection of new energy vehicles, using depth information to enhance detection accuracy and propose a deep guide loss function, the problems of insufficient detection accuracy and large resource consumption in the existing technology are solved, and efficient and accurate goals are achieved Test.

CN118397605BActive Publication Date: 2024-11-05GUANGDONG UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410866235.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-01
Publication Date
2024-11-05
Estimated Expiration
2044-07-01

Smart Images

  • Figure CN118397605B_ABST
    Figure CN118397605B_ABST
Patent Text Reader

Abstract

The present invention discloses a new energy vehicle target detection method based on depth perception, which belongs to the field of vehicle target detection technology, and includes the following steps: obtaining RGB images and depth images of new energy vehicles; constructing an efficient downsampling module, a feature extraction module and a feature fusion module, and constructing a backbone network based on an efficient downsampling module, three consecutive feature extraction modules and a feature fusion module; constructing a lightweight residual module, and constructing a neck network based on three consecutive lightweight residual modules and a feature pyramid network; constructing a detection head, and constructing a depth perception detection model based on the backbone network, the neck network and the detection head; inputting the RGB image and the depth image into the depth perception detection model, performing feature extraction, feature fusion, feature enhancement and feature classification operations, and realizing target detection. The new energy vehicle target detection method based on depth perception proposed by the present invention significantly improves the positioning capability of target detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of automobile target detection, and in particular relates to a new energy automobile target detection method based on depth perception. Background Art

[0002] At present, as new energy vehicles are developing, more and more intelligent devices are installed on the vehicle body due to the continuous improvement of endurance performance. Among them, safety equipment has attracted the attention of the new energy vehicle development industry. How to make safety equipment quickly and in real time handle road emergencies has become the mainstream of development. However, the safety protection of new energy vehicles is affected by many factors, mainly from complex road conditions, vehicles and pedestrians on the road, and special traffic scenes, which brings huge challenges to the development of new energy vehicle safety and the research of safety equipment.

[0003] In such a complex and ever-changing scenario, artificial intelligence technology, especially machine learning and deep learning technology, is now widely installed in the safety equipment system of new energy vehicles because of its powerful characteristics of complex data processing capabilities. Under the unique training data and training model, in-depth analysis of complex historical data is carried out to improve a set of training models to help the development industry develop more economical and effective solutions, which is increasingly attracting attention. Among them, target detection technology is the first barrier to safety protection. It can quickly and effectively identify sudden and complex scene information on the road, and reserve more time for subsequent safety plans, such as anti-collision detection and emergency avoidance. It is a key link in reducing traffic accidents and avoiding casualties.

[0004] In the prior art, a method and system for target detection in an autonomous driving scenario proposes collecting photos of faces and license plates in autonomous driving scenarios from various perspectives in different environments, annotating them, and generating a proprietary data set; building a yolov5 model structure, and then using the proprietary data set to train a pre-trained model, and fine-tuning the hyperparameters of the pre-trained model based on the training results to obtain a target detection model; finally, using the target detection model to identify and classify the images to be detected to obtain accurate detection targets.

[0005] A target detection method, device and equipment for autonomous driving are proposed, which include acquiring an image to be detected; inputting the image to be detected into a preset target detection model to obtain a detection result output by the preset target detection model; using a preset target classification model to classify the image area where a small target object in the detection result is located to obtain a classification result of the small target object; and using the classification result to correct the category information of the small target object in the detection result to obtain a detection result after the classification result correction.

[0006] An object detection system in an autonomous driving scenario based on image and laser data includes lidar data and camera data. It is proposed that the output end of the lidar data is connected to a traditional point cloud processing algorithm module and a point cloud processing neural network, and the output ends of the two types of data are connected to a fusion module. The output end of the fusion module is connected to a multi-sensor fusion module.

[0007] The above prior art has the following defects:

[0008] Three-dimensional object detection has achieved good detection performance in autonomous driving, however, it requires a lot of computing resources, which hinders its practical application. Two-dimensional object detection has a smaller computational burden, but lacks deeply embedded spatial and geometric information. In the case of complex and changeable real-world scenarios, the impact of detection accuracy on safety needs to be evaluated. The consumption of resources and time by general multi-stage object detection technology is unsustainable, and its application prospects in reducing traffic costs and improving vehicle efficiency should also become an important consideration in technology development.

[0009] In response to the above technical problems, it is urgent to propose a new energy vehicle target detection method based on depth perception. Summary of the invention

[0010] In order to solve the above technical problems, the present invention proposes a new energy vehicle target detection method based on depth perception, which is used to improve the accuracy of new energy vehicle target detection and achieve the purpose of improving the safety of new energy vehicles, so as to solve the problems existing in the above-mentioned prior art.

[0011] To achieve the above object, the present invention provides a new energy vehicle target detection method based on depth perception, comprising the following steps:

[0012] Get RGB images and depth images of new energy vehicles;

[0013] Construct an efficient downsampling module, a feature extraction module and a feature fusion module, and construct a backbone network based on an efficient downsampling module, three consecutive feature extraction modules and a feature fusion module;

[0014] Construct a lightweight residual module and build a neck network based on three consecutive lightweight residual modules and a feature pyramid network;

[0015] Build a detection head and build a depth perception detection model based on the backbone network, neck network and detection head;

[0016] The RGB image and the depth image are input into the depth perception detection model to perform feature extraction, feature fusion, feature enhancement and feature classification operations to achieve target detection.

[0017] Optionally, the process of obtaining RGB images and depth images includes: installing a lidar at any location on the top and around the car, obtaining 3D point cloud data of the car's surroundings based on the lidar, and processing the 3D point cloud data to obtain a corresponding depth image; installing cameras on the front, back and sides of the car, and obtaining RGB images of the car's surroundings based on the cameras.

[0018] Optionally, the efficient downsampling structure includes a 4×4 convolutional layer and two 3×3 convolutional layers.

[0019] Optionally, the process of extracting features based on three consecutive feature extraction modules includes: inputting image data after downsampling convolution operation, performing 1×1 standard convolution and 5×5 depth-aware convolution operations on the input image data based on the left branch, performing depth-aware average pooling and 1×1 standard convolution operations on the input image data based on the right branch, and then connecting the outputs of the two branches, performing channel shuffling operation, and finally performing 1×1 standard convolution operation on the output features after channel shuffling, so as to obtain features of different scales in the image data; the image data includes RGB images and depth images.

[0020] Optionally, the process of performing feature fusion based on the feature fusion module includes: taking the output of the feature extraction module as input, and using two fusion units to perform feature fusion in a paired manner.

[0021] Optionally, the process of performing feature enhancement based on the lightweight residual module includes: inputting the output features of the feature fusion module, performing 1×1 standard convolution and 5×5 depth-perceptual convolution operations based on the left branch, and adding and fusing the output features of the 1×1 standard convolution and the 5×5 depth-perceptual convolution, performing a 1×1 standard convolution operation based on the right branch, connecting the output features of the left branch and the right branch and shuffling the channels, and finally performing feature enhancement based on the 1×1 standard convolution.

[0022] Optionally, the process of performing feature classification based on the detection head includes: constructing an output channel based on the prediction result of whether the object exists, the predicted position of the bounding box, and the predicted probability of the object category, performing a standard convolution operation on the output channel based on the detection head, and outputting the target detection result.

[0023] Optionally, the depth-aware convolution is formulated as follows:

[0024] ,

[0025] ,

[0026] in, Indicates the center position of the current convolution window, represents a specific position within the current convolution window, w represents the weight of the convolution kernel, and x represents the convolution kernel. and Indicates location, represents the deep similarity, represents a constant, Represents a depth map.

[0027] Optionally, depth-aware average pooling is formulated as follows:

[0028] .

[0029] Optionally, the overall loss function used by the depth perception detection model is as follows:

[0030] ,

[0031] in, , , and They represent confidence loss, classification loss, bounding box regression loss, and depth-guided loss, respectively. , , and denote the hyperparameters of balanced confidence loss, classification loss, bounding box regression loss, and depth guided loss, respectively.

[0032] Compared with the prior art, the present invention has the following advantages and technical effects:

[0033] The target detection devices installed in existing new energy vehicles are often based on two-dimensional data or three-dimensional data alone. The method based on two-dimensional data will bring certain risks due to its detection accuracy, while the method based on three-dimensional data requires a large resource overhead. The present invention uses depth information to assist in two-dimensional target detection, and proposes an effective target detection method based on depth perception learning. A downsampling module is designed to downsample the input image to solve the resource overhead of feature extraction and achieve the purpose of shortening the processing time; at the same time, depth perception convolution and depth perception average pooling are used to enhance the perception of three-dimensional space, so that the designed detection head can learn spatial perception features under the framework; a lightweight residual module is further proposed, and then a feature pyramid network is used to enhance information, so that the lightweight residual block achieves high performance while using less calculations and parameters; finally, a new depth-guided loss function is proposed, which significantly improves the positioning ability of the depth perception detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0035] Figure 1 A depth perception detection framework diagram according to an embodiment of the present invention;

[0036] Figure 2 Schematic diagram of an efficient downsampling module according to an embodiment of the present invention;

[0037] Figure 3 Schematic diagram of a feature extraction module according to an embodiment of the present invention;

[0038] Figure 4 Schematic diagram of a feature fusion module according to an embodiment of the present invention;

[0039] Figure 5 Schematic diagram of a lightweight residual module according to an embodiment of the present invention;

[0040] Figure 6 Schematic diagram of a predicted bounding box according to an embodiment of the present invention, wherein (a) is a schematic diagram showing the intersection of a predicted bounding box and a true bounding box, (b) is a schematic diagram showing the inclusion of a predicted bounding box by a true bounding box, (c) is a schematic diagram showing the inclusion of a true bounding box by a predicted bounding box, and (d) is a schematic diagram showing the non-intersection of a predicted bounding box and a true bounding box. DETAILED DESCRIPTION

[0041] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] Embodiment 1

[0044] This embodiment provides a new energy vehicle target detection method based on depth perception, including the following steps:

[0045] Get RGB images and depth images of new energy vehicles;

[0046] Construct an efficient downsampling module, a feature extraction module and a feature fusion module, and construct a backbone network based on an efficient downsampling module, three consecutive feature extraction modules and a feature fusion module;

[0047] Construct a lightweight residual module and build a neck network based on three consecutive lightweight residual modules and a feature pyramid network;

[0048] Build a detection head and build a depth perception detection model based on the backbone network, neck network and detection head;

[0049] The RGB image and the depth image are input into the depth perception detection model to perform feature extraction, feature fusion, feature enhancement and feature classification operations to achieve target detection.

[0050] S1, data processing module:

[0051] The data collected in this embodiment uses a specially made electric car with various sensors installed on the body to obtain rich environmental information. These sensors are precisely installed in different positions of the car to provide all-round environmental perception capabilities. The main sensors include LiDAR, camera and GPS / IMU.

[0052] Laser radar (Li DAR): Li DAR is installed on the top or around the vehicle to capture 3D point cloud data of the vehicle's surroundings, project the points in the 3D point cloud onto a two-dimensional plane to calculate the depth value, store the calculated depth value in a two-dimensional image, and obtain the corresponding depth image. Li DAR works by emitting a laser beam and measuring the returned reflected signal, so that the distance and shape of the object can be accurately measured. This data is very important for 3D object detection and environmental modeling.

[0053] Cameras: High-resolution cameras are mounted on the front, back, and sides of the car to capture images of the vehicle’s surroundings. These images provide RGB information and are calibrated with the lidar data to align the 2D image information with the 3D point cloud data. This allows researchers to correlate and align between images and point clouds, enabling more comprehensive environmental perception.

[0054] GPS / IMU: The Global Positioning System (GPS) and Inertial Measurement Unit (IMU) are used to record the vehicle's position, direction, speed and other motion status. This information is very important for accurately tracking and locating the vehicle's motion trajectory.

[0055] These sensors collect data at a high frequency and are synchronized using precise timestamps to ensure consistency between them. The dataset contains data in a variety of different scenarios, such as urban roads, rural roads, highways, etc., as well as various weather conditions and traffic conditions. There are up to 15 vehicles and 30 pedestrians in each image, as well as various degrees of occlusion and truncation. The entire dataset consists of 389 pairs of stereo images and optical flow maps, 39.2km of visual ranging sequences, and more than 200k images of 3D annotated objects, sampled and synchronized at a frequency of 10Hz. The labels are subdivided into cars, vans, trucks, pedestrians, cyclists, and trams.

[0056] S2, build a depth perception detection model:

[0057] The depth perception detection framework for building a depth perception detection model is as follows Figure 1 As shown in the figure. The depth-aware detection framework accepts both RGB images and depth images as input. The backbone network of the depth-aware detection framework fully extracts and fuses RGB features and depth features. The neck network effectively integrates feature information of different scales. The head of the detector performs category classification and bounding box regression. By mapping the predicted bounding box to the original depth map, the detector can also output the distance information of the detected object.

[0058] The construction process of the backbone network includes: constructing an efficient downsampling module, a feature extraction module and a feature fusion module. The backbone network is constructed based on an efficient downsampling module, three consecutive feature extraction modules and a feature fusion module.

[0059] S3, efficient downsampling module:

[0060] The efficient downsampling module in the backbone network is used to efficiently downsample the input image, thereby producing a smaller feature map, which helps to reduce the computational load in subsequent modules. Figure 2 As shown, this embodiment uses a 4×4 convolution layer and two 3×3 convolution layers to construct an efficient downsampling module.

[0061] S4, feature extraction module:

[0062] The feature extraction module is used to effectively extract features. Multi-scale feature representation in convolutional neural networks plays a vital role in target detection. Therefore, in the backbone network, this embodiment uses three consecutive feature extraction modules to obtain features of different scales. The feature extraction modules are as follows: Figure 3As shown in the figure. At the initial stage of the feature extraction module, parallel downsampling convolution operations are applied to the input features. The left branch consists of a 1×1 standard convolution and a 5×5 depth-aware convolution. The right branch performs depth-aware average pooling and a 1×1 standard convolution on the input features. The outputs of the two branches are then concatenated, followed by a channel shuffle operation on the concatenated features. Finally, a 1×1 standard convolution is applied to the shuffled features to enhance the interaction between features. Depth-aware convolution and depth-aware average pooling incorporate geometric variance into the standard convolution and average pooling operations.

[0063] Depth-aware convolution adds a depth similarity term to the standard convolution, which makes pixels with the same depth as the center of the convolution kernel contribute more to the output than other pixels. It can be expressed as follows:

[0064] ,

[0065] ,

[0066] in, Refers to the center position of the current convolution window; refers to the specific position within the current convolution window, w refers to the weight of the convolution kernel, and x refers to the convolution kernel. and Indicates location. represents the deep similarity, is a constant, Represents a depth map.

[0067] like Figure 3 As shown in Figure 2, depth-aware average pooling ensures that pixels with higher depth correlation have a greater impact on the pooled output value, allowing the propagation of visual and geometric information. Its formula is as follows:

[0068] .

[0069] S5, feature fusion module:

[0070] The feature fusion module is responsible for merging and enhancing features of different scales in the backbone network. Figure 4 As shown in Figure 1, the feature fusion module performs a fusion operation on multi-scale features, enabling the model to simultaneously fuse the high-resolution information of low-level features and the high-level semantic information of high-level features. The feature fusion module receives the outputs of the three feature extraction modules as input and uses two fusion units to merge features in a pairwise manner.

[0071] The process of building the neck network includes: building a lightweight residual module, building a neck network based on three consecutive lightweight residual modules and a feature pyramid network.

[0072] S6, lightweight residual module:

[0073] The lightweight residual module is used to further extract feature information. Figure 5 As shown, this embodiment uses three groups of lightweight residual modules in the neck network to process the output of the feature fusion module, and then uses a feature pyramid network to enhance the information. The carefully designed lightweight residual module achieves high performance while using less calculations and parameters. Similar to the feature extraction module, the lightweight residual module also adopts a two-branch structure. In the left branch, a 1×1 standard convolution is applied first, followed by a 5×5 depth-aware convolution. The output features of the 1×1 standard convolution and the depth-aware convolution are then added to achieve feature fusion. The right branch consists of a 1×1 standard convolution that performs information interaction in the channel dimension. Similar to the feature extraction module, the outputs of the left and right branches are connected, the channels are shuffled, and finally enhanced by a 1×1 standard convolution.

[0074] S7, sensor head:

[0075] The detection head is responsible for predicting the detection results of the model. It adopts a multi-head structure, where each head takes features of different scales from the feature pyramid network as input. Each detection head uses a standard convolution operation with an output channel size of (1+4+Np)×3, where Np corresponds to the predicted probability of the object category. Here, 1 corresponds to the prediction of whether an object exists, and 4 corresponds to the predicted position of the bounding box. During the training phase, the above three predictions are compared with the ground truth labels to generate a confidence loss , classification loss , and the bounding box regression loss In addition, the predicted bounding boxes are mapped onto the depth map, where a depth-guided loss is generated. .

[0076] S8, loss optimization:

[0077] In order to improve the detection performance of the model, the overall loss function used by the optimization model in this embodiment is as follows:

[0078] ,

[0079] in, , , and They represent confidence loss, classification loss, bounding box regression loss, and depth-guided loss, respectively. , , and is a hyperparameter used to balance these losses. and Using cross entropy loss CIoU loss is used. This embodiment proposes a depth-guided loss function , effectively improving the positioning ability of the model. First, the minimum bounding rectangle of the predicted bounding box and the true bounding box is calculated. Then, the area corresponding to the minimum bounding rectangle is extracted on the depth map and represented as .Will The area corresponding to the true bounding box is set to 0, and the result is expressed as Similarly, The area corresponding to the predicted bounding box is set to 0, and the result is expressed as Assumptions The width is W and the height is H. This embodiment defines the following depth-guided loss function:

[0080] ,

[0081] in represent The value of row i and column j in represent The value in .

[0082] Figure 6 It shows all possible positional relationships between the predicted bounding box and the true bounding box. , on the right is In this embodiment, and It is divided into five regions: P1-P5 and G1-G5. Among them, regions P3, P4, G2 and G4 are all zero regions, while P1 is the same as G1, and P5 is the same as G5. Therefore, in the calculation , can be simplified as follows:

[0083] ,

[0084] in, represents the all-zero area. During the training phase, It will continue to optimize, driving the predicted bounding box and the true bounding box closer and closer, thereby improving the positioning ability.

[0085] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A new energy vehicle target detection method based on depth perception, characterized in that: The following steps are involved: Get RGB images and depth images of new energy vehicles; Construct an efficient downsampling module, a feature extraction module and a feature fusion module, and construct a backbone network based on an efficient downsampling module, three consecutive feature extraction modules and a feature fusion module; Construct a lightweight residual module and build a neck network based on three consecutive lightweight residual modules and a feature pyramid network; Build a detection head and build a depth perception detection model based on the backbone network, neck network and detection head; Inputting the RGB image and the depth image into the depth perception detection model, performing feature extraction, feature fusion, feature enhancement and feature classification operations to achieve target detection; The process of extracting features based on three consecutive feature extraction modules includes: inputting image data after downsampling convolution operation, performing 1×1 standard convolution and 5×5 depth-aware convolution operations on the input image data based on the left branch, performing depth-aware average pooling and 1×1 standard convolution operations on the input image data based on the right branch, then connecting the outputs of the two branches, performing channel shuffling operation, and finally performing 1×1 standard convolution operation on the output features after channel shuffling, so as to obtain features of different scales in the image data; the image data includes RGB images and depth images; The formula for depth-aware convolution is as follows: , , in, Indicates the center position of the current convolution window, represents a specific position within the current convolution window, w represents the weight of the convolution kernel, and x represents the convolution kernel. and Indicates location, represents the deep similarity, represents a constant, represents a depth map; The formula for depth-aware average pooling is as follows: 。 2. The new energy vehicle target detection method based on depth perception according to claim 1 is characterized in that: The process of obtaining RGB images and depth images includes: installing a laser radar at any position on the top and around the car, obtaining 3D point cloud data of the car's surroundings based on the laser radar, projecting the points in the 3D point cloud onto a two-dimensional plane to calculate the depth value, storing the calculated depth value in the two-dimensional image, and obtaining the corresponding depth image; installing cameras on the front, back and sides of the car, and obtaining RGB images of the car's surroundings based on the cameras.

3. The new energy vehicle target detection method based on depth perception according to claim 1 is characterized in that: The efficient downsampling structure includes a 4×4 convolution layer and two 3×3 convolution layers.

4. The new energy vehicle target detection method based on depth perception according to claim 1 is characterized in that: The process of performing feature fusion based on the feature fusion module includes: taking the output of the feature extraction module as input, and using two fusion units to perform feature fusion in a paired manner.

5. The new energy vehicle target detection method based on depth perception according to claim 4 is characterized in that: The process of feature enhancement based on the lightweight residual module includes: inputting the output features of the feature fusion module, performing 1×1 standard convolution and 5×5 depth-aware convolution operations based on the left branch, and adding and fusing the output features of the 1×1 standard convolution and 5×5 depth-aware convolution, performing 1×1 standard convolution operation based on the right branch, connecting the output features of the left branch and the right branch and shuffling the channels, and finally performing feature enhancement based on the 1×1 standard convolution.

6. The new energy vehicle target detection method based on depth perception according to claim 1 is characterized in that: The process of feature classification based on the detection head includes: constructing an output channel based on the prediction result of whether the object exists, the predicted position of the bounding box, and the predicted probability of the object category, performing a standard convolution operation on the output channel based on the detection head, and outputting the target detection result.

7. The new energy vehicle target detection method based on depth perception according to claim 1 is characterized in that: The overall loss function used by the depth perception detection model is as follows: , in, , , and They represent confidence loss, classification loss, bounding box regression loss, and depth-guided loss, respectively. , , and denote the hyperparameters of balanced confidence loss, classification loss, bounding box regression loss, and depth guided loss, respectively.