A vehicle-road cooperation end-to-end 3D target detection method, system, device and storage medium

Abstract

The application discloses a kind of vehicle-road cooperation end-to-end 3D target detection method, system, equipment, storage medium, wherein the method comprises: when detecting vehicle enters certain monitoring area, obtain current vehicle end monitoring image, current road end monitoring image and corresponding depth map;Current vehicle monitoring image and current road side monitoring image are input into depth alignment neural network, and implicit alignment feature map is obtained;Current road end monitoring image and corresponding depth map are input into depth feature extraction neural network, and corresponding road end perspective feature map is obtained;After implicit alignment feature map and road end perspective feature map are fused, input coordinate conversion network carries out coordinate conversion, and vehicle end perspective feature map is obtained;Vehicle end perspective feature map is input into 3D target detection network, and first 3D target detection information is obtained.The application saves the hardware cost of vehicle, also reduces unnecessary scene depth information processing process in vehicle end, effectively improves the stability of automatic driving system.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically, to a vehicle-road cooperative end-to-end 3D target detection method, system, device, and storage medium. Background Technology

[0002] Currently, existing vehicle-road cooperative 3D target detection methods either employ traditional algorithms for position alignment or require the installation and calibration of corresponding depth map acquisition devices on the vehicle. However, using traditional algorithms for position alignment results in the inability to achieve end-to-end output and suffers from the drawback of increasing position prediction errors over time. Installing and calibrating corresponding depth map acquisition devices on the vehicle not only increases the vehicle's hardware and software costs but also adds many unnecessary depth information processing steps.

[0003] In fact, during the autonomous driving process, it is only necessary to obtain 3D information of the target in some key driving areas to significantly improve the robustness of the autonomous driving system. Summary of the Invention

[0004] To address the issues raised by the prior art, which involves installing corresponding depth map acquisition devices on vehicles, increasing the vehicle's hardware and software costs and requiring unnecessary depth information processing, this invention provides a vehicle-road cooperative end-to-end 3D target detection method, system, device, and storage medium.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A vehicle-road cooperative end-to-end 3D target detection method, the method comprising the following steps:

[0007] When a vehicle is detected to have entered a certain monitoring area, the system requests the current roadside monitoring image and the depth map corresponding to the current roadside monitoring image from the roadside monitoring station, and also obtains the current vehicle-side monitoring image.

[0008] Input the current vehicle monitoring image and the current roadside monitoring image into the trained deep alignment neural network to obtain implicit alignment feature maps;

[0009] Input the current roadside monitoring image and the corresponding depth map into the trained depth feature extraction neural network to obtain the corresponding roadside view feature map;

[0010] After fusing the implicit alignment feature map with the roadside view feature map, the input is fed into the trained coordinate transformation network for coordinate transformation to obtain the vehicle-side view feature map.

[0011] The feature map from the vehicle's perspective is input into the trained 3D object detection network to obtain the first 3D object detection information based on the vehicle's perspective.

[0012] Preferably, the detection method detects whether the vehicle has entered a certain monitoring area. Specifically, when the distance between the vehicle's current position and the monitoring camera is less than a preset first distance threshold, it is determined that the vehicle has entered the monitoring area of ​​the corresponding monitoring camera.

[0013] Preferably, the implicit alignment feature map is the coordinate alignment relationship between the vehicle end and the road end.

[0014] Preferably, the first 3D target detection information includes the 3D position of the target from the vehicle's perspective, its distance from the vehicle, and its angle.

[0015] Preferably, the implicit alignment feature map and the roadside view feature map are the same size. The implicit alignment feature map and the roadside view feature map are spliced ​​on the channel and merged into a single feature map, which is then input into the coordinate transformation network for coordinate transformation.

[0016] Preferably, the deep alignment neural network, the deep feature extraction neural network, the coordinate transformation network, and the 3D target detection network are all deep neural networks.

[0017] Furthermore, based on the data flow, a 3D object detection model is constructed by combining a deep alignment neural network, a deep feature extraction neural network, a coordinate transformation network, and a 3D object detection network. The 3D object detection model is then pre-trained as follows:

[0018] The training vehicle-side monitoring images, roadside monitoring images, and the corresponding depth maps of the roadside monitoring images are input into the 3D object detection model to obtain the second 3D object detection information.

[0019] Calculate the loss between the second 3D object detection information and the true 3D object detection information of the label data, and then update the parameters of the 3D object detection model through gradient backpropagation.

[0020] A vehicle-road cooperative end-to-end 3D target detection system includes:

[0021] The detection module is used to detect whether a vehicle has entered a certain monitored area;

[0022] The acquisition module requests the current roadside monitoring image and the depth map corresponding to the current roadside monitoring image from the roadside monitoring module when the vehicle enters a certain monitoring area, and also acquires the current vehicle-side monitoring image.

[0023] The deep alignment neural network module is used to process the input current vehicle monitoring image and the current roadside monitoring image to obtain implicit alignment feature maps;

[0024] The deep feature extraction neural network module is used to extract the current roadside monitoring image and the corresponding depth map from the input current roadside monitoring image to obtain the corresponding roadside view feature map.

[0025] The fusion module is used to fuse the implicit alignment feature map and the roadside view feature map;

[0026] The coordinate transformation network module is used to transform the coordinates of the fused implicitly aligned feature map and the roadside view feature map to obtain the vehicle-side view feature map.

[0027] The 3D object detection network module is used to process the input vehicle-view feature map to obtain the first 3D object detection information based on the vehicle-view perspective.

[0028] A computer device includes a memory and a processor, the memory storing a computer program executable on the processor, the processor executing the computer program to implement the steps of the method described above.

[0029] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0030] The beneficial effects of this invention are as follows:

[0031] 1. Compared to the traditional method of installing related equipment on vehicles to obtain depth maps, the present invention does not require the installation of related equipment on vehicles to obtain depth maps. It only needs to request the current roadside monitoring image and the corresponding depth map from the roadside for processing. Therefore, it saves vehicle hardware costs and reduces unnecessary depth information processing on the vehicle, effectively improving the stability of the autonomous driving system.

[0032] 2. Compared to traditional algorithms, which suffer from the inability to achieve end-to-end result output and whose position prediction errors increase over time, this invention directly processes the corresponding data using a pre-trained deep alignment neural network, deep feature extraction neural network, deep alignment neural network, and coordinate transformation network. Only the relevant data needs to be input to directly output the 3D detection results from the vehicle's perspective, enabling rapid end-to-end result output and reducing position prediction errors. Attached Figure Description

[0033] Figure 1 This is a flowchart of the steps of an end-to-end 3D target detection method for vehicle-road cooperative systems according to the present invention.

[0034] Figure 2 This is the data flow diagram of each neural network in this invention.

[0035] Figure 3 This is a schematic diagram of the principle of a vehicle-road cooperative end-to-end 3D target detection system of the present invention. Detailed Implementation

[0036] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0037] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0038] To address the issue of existing methods that require installing depth map acquisition equipment on vehicles, which not only increases the vehicle's hardware costs but also adds unnecessary depth information processing steps, the robustness of the autonomous driving system can be significantly improved simply by acquiring 3D information of the target in a few key driving areas during actual autonomous driving.

[0039] In view of this, this embodiment proposes a vehicle-road cooperative end-to-end 3D target detection method for vehicle 3D target pose detection in the field of autonomous driving. It can effectively improve the stability of the autonomous driving system by utilizing the monitoring images and corresponding depth information of roadside cameras in key road sections during the autonomous driving process.

[0040] like Figure 1 , Figure 2 As shown, the vehicle-road cooperative end-to-end 3D target detection method includes the following steps:

[0041] When a vehicle is detected to have entered a certain monitoring area, the system requests the current roadside monitoring image and the depth map corresponding to the current roadside monitoring image from the roadside monitoring station, and also obtains the current vehicle-side monitoring image.

[0042] Input the current vehicle monitoring image and the current roadside monitoring image into the trained deep alignment neural network to obtain implicit alignment feature maps;

[0043] Input the current roadside monitoring image and the corresponding depth map into the trained depth feature extraction neural network to obtain the corresponding roadside view feature map;

[0044] After fusing the implicit alignment feature map with the roadside view feature map, the input is fed into the trained coordinate transformation network for coordinate transformation to obtain the vehicle-side view feature map.

[0045] The feature map from the vehicle's perspective is input into the trained 3D object detection network to obtain the first 3D object detection information based on the vehicle's perspective.

[0046] In this embodiment, surveillance cameras are pre-installed at the roadside and are networked. Since the surveillance cameras at the roadside are fixed and their positions do not change, the location coordinates of the surveillance cameras can be uploaded to the cloud.

[0047] When a vehicle is detected entering a monitored area, the system requests the current roadside monitoring image and the corresponding depth map from the roadside monitoring camera, and also acquires the current vehicle-side monitoring image. The current roadside monitoring image is a two-dimensional image of the road. The depth map is a depth map of the two-dimensional road image; a depth map is an image or image channel that contains information related to the distance from the surface of a scene object to the viewpoint.

[0048] Compared to the traditional method of installing related equipment on vehicles to obtain depth maps, the method described in this embodiment does not require the installation of such equipment on the vehicle. It only needs to request the current roadside monitoring image and the corresponding depth map from the roadside monitoring camera for processing. This saves on vehicle hardware costs. At the same time, by performing depth processing on the roadside monitoring image using the monitoring camera on the roadside, unnecessary depth information processing in the vehicle is reduced, effectively improving the stability of the autonomous driving system.

[0049] In this embodiment, the corresponding data is processed directly through the trained deep alignment neural network, deep feature extraction neural network, deep alignment neural network, and coordinate transformation network. Only the corresponding data needs to be input to directly output the 3D detection results from the vehicle's perspective, which can achieve fast end-to-end result output. Since this embodiment can output results quickly, it is beneficial to reduce position prediction errors.

[0050] In this embodiment, the detection of whether a vehicle has entered a certain monitoring area is specifically determined by the distance between the location coordinates of a monitoring camera pre-installed at the roadside and the vehicle's current position. When the distance between the vehicle's current position and the monitoring camera is less than a preset first distance threshold, it is determined that the vehicle has entered the monitoring area of ​​the corresponding monitoring camera.

[0051] Specifically, vehicle location information is obtained via GPS or other methods. The vehicle can also obtain the location coordinates of roadside surveillance cameras via the cloud. When the distance between the vehicle's current location and the surveillance camera is less than a preset first distance threshold, it indicates that the vehicle has entered the monitored area. At this time, the intelligent driving system requests the surveillance camera to obtain the current roadside monitoring image and the corresponding depth map. This method does not require timestamp information, so the requirement for request latency is not high. The request can be sent as soon as the system detects that the vehicle is about to enter the monitored area.

[0052] In this embodiment, the current vehicle monitoring image and the current roadside monitoring image are input into a trained deep alignment neural network to obtain an implicit alignment feature map, which represents the coordinate alignment relationship between the vehicle and the roadside. More specifically, after acquiring the current roadside monitoring image, both the current monitoring image and the current vehicle monitoring image are input into the deep alignment neural network. This deep alignment neural network will output an implicit coordinate alignment relationship between the vehicle and the roadside based on the input images.

[0053] The deep alignment neural network is a deep neural network. After training, the deep alignment neural network has the ability to align the viewpoints of the vehicle end and the road end.

[0054] In this embodiment, the current roadside monitoring image and its corresponding depth map are input into a trained deep feature extraction neural network to extract the corresponding roadside view feature map. The roadside view feature map includes information such as texture and depth of the roadside monitoring image. Generally, different network modules in the deep feature extraction neural network are used to extract features from the current roadside monitoring image and the current roadside monitoring image, respectively. Then, the same network module is used to fuse the feature information of the current roadside monitoring image and the current roadside monitoring image to obtain the final roadside view feature map.

[0055] In this embodiment, the feature map from the vehicle's perspective is input into a trained 3D object detection network to obtain first 3D object detection information based on the vehicle's perspective. This first 3D object detection information includes the 3D position of the target from the vehicle's perspective, its distance from the vehicle, and its angle.

[0056] Because this embodiment integrates the depth map corresponding to the current roadside monitoring image, the first 3D target detection information obtained in this embodiment will be more accurate than that obtained by simply using the vehicle monocular monitoring image. Furthermore, because a deep neural network is used as a coordinate transformation network to learn coordinate transformation information, and the implicit alignment feature map and the roadside view feature map are transformed, the vehicle-side view feature map can be directly output.

[0057] In this embodiment, the implicit alignment feature map and the roadside view feature map are the same size. The implicit alignment feature map and the roadside view feature map are spliced ​​on the channel and merged into a single feature map, which is then input into the coordinate transformation network for coordinate transformation.

[0058] In this embodiment, the deep alignment neural network, the deep feature extraction neural network, the coordinate transformation network, and the 3D object detection network are all deep neural networks.

[0059] In this embodiment, as Figure 2 As shown, a 3D object detection model is constructed by combining a deep alignment neural network, a deep feature extraction neural network, a coordinate transformation network, and a 3D object detection network according to the data flow direction. The 3D object detection model is pre-trained as follows:

[0060] The trained vehicle-side monitoring images, roadside monitoring images, and the corresponding depth maps of the roadside monitoring images are input into the 3D object detection model, and the second 3D object detection information is output.

[0061] Calculate the loss between the second 3D object detection information and the true 3D object detection information of the label data, and then update the parameters of the 3D object detection model through gradient backpropagation.

[0062] Due to errors in the samples during training, the 3D object detection model can learn to overcome time delay errors.

[0063] In one specific embodiment, a vehicle-road cooperative end-to-end 3D target detection system is also provided, such as... Figure 3 As shown, it includes:

[0064] The detection module is used to detect whether a vehicle has entered a certain monitored area;

[0065] The acquisition module requests the current roadside monitoring image and the depth map corresponding to the current roadside monitoring image from the roadside monitoring module when the vehicle enters a certain monitoring area, and also acquires the current vehicle-side monitoring image.

[0066] The deep alignment neural network module is used to process the input current vehicle monitoring image and the current roadside monitoring image to obtain implicit alignment feature maps;

[0067] The deep feature extraction neural network module is used to extract the current roadside monitoring image and the corresponding depth map from the input current roadside monitoring image to obtain the corresponding roadside view feature map.

[0068] The fusion module is used to fuse the implicit alignment feature map and the roadside view feature map;

[0069] The coordinate transformation network module is used to transform the coordinates of the fused implicitly aligned feature map and the roadside view feature map to obtain the vehicle-side view feature map.

[0070] The 3D object detection network module is used to process the input vehicle-view feature map to obtain the first 3D object detection information based on the vehicle-view perspective.

[0071] In one specific embodiment, a computer device is also provided, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the following steps of the vehicle-road cooperative end-to-end 3D target detection method:

[0072] When a vehicle is detected to have entered a certain monitoring area, the current vehicle-side monitoring image, the current roadside monitoring image, and the depth map corresponding to the current roadside monitoring image are obtained.

[0073] Input the current vehicle monitoring image and the current roadside monitoring image into the trained deep alignment neural network to obtain implicit alignment feature maps;

[0074] Input the current roadside monitoring image and the corresponding depth map into the trained depth feature extraction neural network to extract the corresponding roadside view feature map;

[0075] After fusing the implicit alignment feature map with the roadside view feature map, the result is input into the trained coordinate transformation network for coordinate transformation to obtain the vehicle-side view feature map.

[0076] The feature map from the vehicle's perspective is input into the trained 3D object detection network to obtain the first 3D object detection information based on the vehicle's perspective.

[0077] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0078] In one specific embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the following steps of the vehicle-road cooperative end-to-end 3D target detection method:

[0079] When a vehicle is detected to have entered a certain monitoring area, the current vehicle-side monitoring image, the current roadside monitoring image, and the depth map corresponding to the current roadside monitoring image are obtained.

[0080] Input the current vehicle monitoring image and the current roadside monitoring image into the trained deep alignment neural network to obtain implicit alignment feature maps;

[0081] Input the current roadside monitoring image and the corresponding depth map into the trained depth feature extraction neural network to obtain the corresponding roadside view feature map;

[0082] After fusing the implicit alignment feature map with the roadside view feature map, the result is input into the trained coordinate transformation network for coordinate transformation to obtain the vehicle-side view feature map.

[0083] The feature map from the vehicle's perspective is input into the trained 3D object detection network to obtain the first 3D object detection information based on the vehicle's perspective.

[0084] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0085] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the claims of the present invention.

[0086] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A vehicle-road cooperative end-to-end 3D target detection method, characterized in that: The method includes the following steps: When a vehicle is detected to have entered a certain monitoring area, the system requests the current roadside monitoring image and the depth map corresponding to the current roadside monitoring image from the roadside monitoring station, and also obtains the current vehicle-side monitoring image. Input the current vehicle monitoring image and the current roadside monitoring image into the trained deep alignment neural network to obtain implicit alignment feature maps; Input the current roadside monitoring image and the corresponding depth map into the trained depth feature extraction neural network to obtain the corresponding roadside view feature map; After fusing the implicit alignment feature map with the roadside view feature map, the input is fed into the trained coordinate transformation network for coordinate transformation to obtain the vehicle-side view feature map. The feature map from the vehicle's perspective is input into the trained 3D object detection network to obtain the first 3D object detection information based on the vehicle's perspective.

2. The vehicle-road cooperative end-to-end 3D target detection method according to claim 1, characterized in that: The system detects whether a vehicle has entered a certain monitoring area. Specifically, when the distance between the vehicle's current position and the monitoring camera is less than a preset first distance threshold, it determines that the vehicle has entered the monitoring area of ​​the corresponding monitoring camera.

3. The vehicle-road cooperative end-to-end 3D target detection method according to claim 1, characterized in that: The implicit alignment feature map represents the coordinate alignment relationship between the vehicle end and the road end.

4. The vehicle-road cooperative end-to-end 3D target detection method according to claim 1, characterized in that: The first 3D target detection information includes the target's 3D position, distance from the vehicle, and angle from the vehicle's perspective.

5. The vehicle-road cooperative end-to-end 3D target detection method according to claim 1, characterized in that: The implicit alignment feature map and the roadside view feature map are the same size. The implicit alignment feature map and the roadside view feature map are concatenated on the channel and merged into a single feature map, which is then input into the coordinate transformation network for coordinate transformation.

6. The vehicle-road cooperative end-to-end 3D target detection method according to claim 1, characterized in that: The deep alignment neural network, deep feature extraction neural network, coordinate transformation network, and 3D object detection network are all deep neural networks.

7. The vehicle-road cooperative end-to-end 3D target detection method according to claim 6, characterized in that: Based on the data flow, a 3D object detection model is constructed by combining a deep alignment neural network, a deep feature extraction neural network, a coordinate transformation network, and a 3D object detection network. The 3D object detection model is then pre-trained as follows: The vehicle-side monitoring images, roadside monitoring images, and the corresponding depth maps of the roadside monitoring images used for training are input into the 3D object detection model for training to obtain the second 3D object detection information. Calculate the loss between the second 3D object detection information and the true 3D object detection information of the label data, and then update the parameters of the 3D object detection model through gradient backpropagation.

8. A vehicle-road cooperative end-to-end 3D target detection system, characterized in that: include: The detection module is used to detect whether a vehicle has entered a certain monitored area; The acquisition module requests the current roadside monitoring image and the depth map corresponding to the current roadside monitoring image from the roadside monitoring module when the vehicle enters a certain monitoring area, and also acquires the current vehicle-side monitoring image. The deep alignment neural network module is used to process the input current vehicle monitoring image and the current roadside monitoring image to obtain implicit alignment feature maps; The deep feature extraction neural network module is used to extract the current roadside monitoring image and the corresponding depth map from the input current roadside monitoring image to obtain the corresponding roadside view feature map. The fusion module is used to fuse the implicit alignment feature map and the roadside view feature map; The coordinate transformation network module is used to transform the coordinates of the fused implicitly aligned feature map and the roadside view feature map to obtain the vehicle-side view feature map. The 3D object detection network module is used to process the input vehicle-view feature map to obtain the first 3D object detection information based on the vehicle-view perspective.

9. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images