A 3D target detection method based on vehicle-road cooperation

Abstract

The application discloses a 3D target detection method based on vehicle-road cooperation, obtains vehicle end image data and road end image data, unifies the vehicle end image data and the road end image data to the same time coordinate system and the same space coordinate system, then respectively inputs the vehicle end image data and the road end image data into a pre-trained feature extraction network to obtain a feature map F1 of the vehicle end image data and a feature map F2 of the vehicle end image data, then projects image features in the feature maps F1 and F2 into the same 3D voxel volume, carries out 3D feature acquisition on the projected 3D voxel volume, and maps the 3D voxel volume into a two-dimensional tensor, finally inputs the obtained two-dimensional tensor into a 3D target detection network to regress accurate 3D target frame information and category information. The application can realize good three-dimensional reconstruction, effectively solve the detection difficulty caused by the mutual shielding of vehicles at a complex traffic intersection, and improve the driving stability and safety of an autonomous vehicle.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a 3D target detection method based on vehicle-road cooperation. Background Technology

[0002] With the development of technology and the economy, and the widespread application of intelligent technologies, autonomous driving has become a hot research area. Among these, the field of autonomous driving perception has achieved rapid development in recent years, thanks to advancements in computer vision and laser point cloud technologies.

[0003] 3D object detection is an important branch of perception in autonomous driving. The main task of 3D object detection in autonomous driving is to process sensor input data and output the category and precise location of target objects, thereby helping vehicles avoid obstacles and plan routes. Therefore, accurate perception of the surrounding environment and the achievement of precise and reliable 3D object detection are crucial for autonomous driving.

[0004] However, current advanced 3D object detection in the field of autonomous driving often requires the assistance of onboard LiDAR, which significantly increases the actual production cost of autonomous vehicles, casting doubt on their market prospects. Furthermore, on congested urban roads, numerous onboard LiDAR systems are highly susceptible to mutual interference, thereby affecting the driving safety of autonomous vehicles.

[0005] Considering that in actual vehicle operation, the areas where LiDAR-assisted 3D target detection is most needed are often in complex traffic environments with high traffic volume and congestion, where vehicles often severely obstruct each other's view. Therefore, effectively solving the problem of 3D target detection in complex traffic environments without the aid of LiDAR is an urgent issue to be addressed.

[0006] Meanwhile, autonomous vehicles, especially driverless vehicles, require faster decision-making and control responses in complex traffic environments to improve their driving safety. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to address the deficiencies mentioned in the background technology by providing a 3D target detection method based on vehicle-road cooperation. The method aims to make full use of roadside camera information at traffic intersections and vehicle-side camera information of autonomous vehicles to achieve 3D target detection and, on this basis, to achieve rapid decision-making and control response.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] A 3D target detection method based on vehicle-road cooperation includes the following steps:

[0010] Step 1), acquire real-time image data of the road ahead from the vehicle-mounted camera as vehicle-side image data;

[0011] Step 2), acquire real-time image data of the intersection captured by the roadside camera at the traffic intersection as roadside image data;

[0012] Step 3) Unify the vehicle-side image data and roadside image data into the same time coordinate system and the same spatial coordinate system to obtain unified vehicle-side image data and roadside image data;

[0013] Step 4) Input the unified vehicle-side image data and road-side image data into the pre-trained feature extraction network to obtain feature map F1 and feature map F2 of the vehicle-side image data.

[0014] Step 5) Project the image features in feature maps F1 and F2 into the same 3D voxel volume. For each voxel, summarize the image features in feature maps F1 and F2 by element-wise averaging.

[0015] Step 6) Obtain 3D features from the projected 3D voxel volume and map it into a two-dimensional tensor;

[0016] Step 7) Input the obtained two-dimensional tensor into the 3D object detection network to regress accurate 3D object bounding box information and category information.

[0017] As a further optimization of the 3D target detection method based on vehicle-road cooperation of the present invention, in step 3), when unifying to the same time coordinate system, GPS time synchronization is used; when unifying to the same spatial coordinate system, the image coordinate system, pixel coordinate system and world coordinate system are used as intermediaries to realize the registration between vehicle-end image data and road-end image data.

[0018] As a further optimization of the 3D target detection method based on vehicle-road cooperation of the present invention, the feature extraction network in step 4) includes a ResNet-50 model and an FPN model, wherein the ResNet-50 model is used for preliminary feature extraction and the FPN model is used for multi-scale feature fusion.

[0019] As a further optimization of the 3D target detection method based on vehicle-road cooperation of the present invention, in step 6), the encoder part of the encoder-decoder architecture acquires 3D features of the projected 3D voxel volume and maps it into a two-dimensional tensor.

[0020] As a further optimization of the 3D target detection method based on vehicle-road cooperation of the present invention, the 3D target detection network in step 7) adopts a 3D target detection network based on the SSD model. The generated 3D target box is parameterized as (x, y, z, w, h, l, θ), where (x, y, z) are the coordinates of the center of the 3D target box, w, h, l are the width, height and length of the 3D target box respectively, and θ is the rotation angle around the z-axis of the 3D target box.

[0021] As a further optimization of the 3D target detection method based on vehicle-road cooperation of the present invention, the loss function L adopted by the 3D target detection network in step 6) is:

[0022]

[0023] In the formula, the position loss L loc To smooth the average absolute error, the classification loss L cls For focus loss, directional loss L dir For cross-entropy loss, n pos It is the number of positive anchors, λ loc =2.6, λ cls =0.8, λ dir =0.3.

[0024] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0025] 1. While making full use of the real-time detection information from vehicle-mounted cameras, we can also make reasonable use of the real-time detection information from roadside cameras. By combining the high angle and wide range characteristics of roadside cameras, we can fully explore the 3D information of objects and achieve good 3D reconstruction, thereby effectively solving the detection difficulties caused by the mutual occlusion of vehicles at complex traffic intersections.

[0026] 2. By fully leveraging the pure visual perception capabilities of autonomous driving, and eliminating the need for LiDAR sensors, the system can still effectively address the issue of vehicles occluding each other. This will significantly reduce the actual production cost of autonomous vehicles and lay a solid foundation for their future market promotion.

[0027] 3. It effectively avoids the problem of mutual signal interference caused by the convergence of a large number of vehicle-mounted LiDARs, thus increasing the driving safety of autonomous vehicles;

[0028] 4. A pre-trained end-to-end decision network model is adopted to realize end-to-end decision planning. At the same time, a matching drive-by-wire chassis is used to achieve rapid decision-making and control, which can better meet the autonomous driving needs of complex traffic environments and better deal with unexpected road conditions, thereby improving the driving stability and safety of autonomous vehicles. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0030] Figure 2 This is a diagram of the 3D target detection network framework of the present invention. Detailed Implementation

[0031] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0032] This invention can be implemented in many different forms and should not be considered limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully express the scope of the invention to those skilled in the art. In the drawings, components are enlarged for clarity.

[0033] Reference Figure 1 As shown, this invention discloses a 3D target detection method based on vehicle-road cooperation, characterized by comprising the following steps:

[0034] Step 1), acquire real-time image data of the road ahead from the vehicle-mounted camera as vehicle-side image data;

[0035] Step 2), acquire real-time image data of the intersection captured by the roadside camera at the traffic intersection as roadside image data;

[0036] Step 3) Unify the vehicle-side image data and roadside image data into the same time coordinate system and the same spatial coordinate system to obtain unified vehicle-side image data and roadside image data;

[0037] Step 4) Input the unified vehicle-side image data and road-side image data into the pre-trained feature extraction network to obtain feature map F1 and feature map F2 of the vehicle-side image data.

[0038] Step 5) Project the image features in feature maps F1 and F2 into the same 3D voxel volume. For each voxel, the projected features of the vehicle-end image data and the road-end image data are summarized by element-wise averaging.

[0039] Step 6) Obtain 3D features from the projected 3D voxel volume and map it into a two-dimensional tensor;

[0040] Step 7) Use the obtained two-dimensional tensor as input to regress accurate 3D target bounding box information and category information.

[0041] In step 3), when unifying to the same time coordinate system, GPS time synchronization is used; when unifying to the same spatial coordinate system, the image coordinate system, pixel coordinate system and world coordinate system are used as intermediaries to achieve registration between vehicle-end image data and road-end image data.

[0042] like Figure 2 As shown, the feature extraction network includes a ResNet-50 model and an FPN model, wherein the ResNet-50 model is used for preliminary feature extraction, and the FPN model is used for multi-scale feature fusion.

[0043] When projecting image features from feature maps F1 and F2 onto the same 3D voxel volume, let (x, y, z) be the voxel coordinates in world space and (u, v) be the pixel coordinates in image space. The correspondence is as follows:

[0044]

[0045] Among them, P t and K t These are the external and internal matrices of the image, respectively, and Π is the perspective mapping, using the slicing operator.

[0046] In step 6), the encoder part of the encoder-decoder architecture acquires 3D features of the projected 3D voxel volume and maps them into a two-dimensional tensor.

[0047] The 3D object detection network preferentially adopts the SSD model-based 3D object detection network. The generated 3D object box is parameterized as (x, y, z, w, h, l, θ), where (x, y, z) are the coordinates of the center of the 3D object box, w, h, l are the width, height and length of the 3D object box respectively, and θ is the rotation angle around the z-axis of the 3D object box.

[0048] The loss function L used in the 3D object detection network is:

[0049]

[0050] In the formula, the position loss L loc To smooth the average absolute error, the classification loss L cls For focus loss, directional loss L dir For cross-entropy loss, n pos It is the number of positive anchors, λ loc =2.6, λ cls =0.8, λ dir =0.3.

[0051] By inputting the results of 3D object detection, high-precision map information, and vehicle positioning and path planning information into a trained deep neural network, the control quantities required for vehicle control can be obtained, enabling rapid vehicle control.

[0052] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0053] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A 3D target detection method based on vehicle-road cooperation, characterized in that, Includes the following steps: Step 1), acquire real-time image data of the road ahead from the vehicle-mounted camera as vehicle-side image data; Step 2), acquire real-time image data of the intersection captured by the roadside camera at the traffic intersection as roadside image data; Step 3) Unify the vehicle-side image data and roadside image data into the same time coordinate system and the same spatial coordinate system to obtain unified vehicle-side image data and roadside image data; Step 4) Input the unified vehicle-side image data and road-side image data into the pre-trained feature extraction network to obtain feature map F1 and feature map F2 of the vehicle-side image data. Step 5) Project the image features in feature maps F1 and F2 into the same 3D voxel volume. For each voxel, summarize the image features in feature maps F1 and F2 by element-wise averaging. Step 6) Obtain 3D features from the projected 3D voxel volume and map it into a two-dimensional tensor; Step 7) Input the obtained two-dimensional tensor into the 3D object detection network to regress accurate 3D object bounding box information and category information.

2. The 3D target detection method based on vehicle-road cooperation according to claim 1, characterized in that, In step 3), when unifying to the same time coordinate system, GPS time synchronization is used; when unifying to the same spatial coordinate system, the image coordinate system, pixel coordinate system and world coordinate system are used as intermediaries to achieve registration between vehicle-end image data and road-end image data.

3. The 3D target detection method based on vehicle-road cooperation according to claim 1, characterized in that, The feature extraction network in step 4) includes a ResNet-50 model and an FPN model. The ResNet-50 model is used for preliminary feature extraction, and the FPN model is used for multi-scale feature fusion.

4. The 3D target detection method based on vehicle-road cooperation according to claim 1, characterized in that, In step 6), the encoder part of the encoder-decoder architecture acquires 3D features of the projected 3D voxel volume and maps them into a two-dimensional tensor.

5. The 3D target detection method based on vehicle-road cooperation according to claim 1, characterized in that, The 3D object detection network in step 7) adopts an SSD model-based 3D object detection network. The generated 3D object box is parameterized as (x, y, z, w, h, l, θ), where (x, y, z) are the coordinates of the center of the 3D object box, w, h, l are the width, height, and length of the 3D object box, respectively, and θ is the rotation angle around the z-axis of the 3D object box.

6. The 3D target detection method based on vehicle-road cooperation according to claim 1, characterized in that, The loss function L adopted by the 3D object detection network in step 6) is: In the formula, the position loss L loc To smooth the average absolute error, the classification loss L cls For focus loss, directional loss L dir For cross-entropy loss, n pos It is the number of positive anchors, λ loc =2.6, λ cls =0.8, λ dir =0.3.

Images