A multi-level 3D target expression method, system and medium applied to an automatic driving scene

Abstract

The application relates to a multi-level 3D target expression method, system and medium applied to an automatic driving scene, the method comprising the following steps: B1. A vehicle travels on a road, real-time road image data information is acquired based on a vehicle-mounted camera, real-time road point cloud data information is acquired based on a vehicle-mounted laser radar, and distance data information of an obstacle is acquired in real time based on a vehicle-mounted ultrasonic radar; B2. The road image data information is input into a convolutional neural network for identification, and is classified according to highway signs, and highway scene classification data information and non-highway scene classification data information are output; and B3. The road point cloud data information is projected to the road image data information based on the highway scene classification data information. The application not only facilitates selection of an optimal expression scheme for decision planning under suitable working conditions, but also solves the problem of perception and planning cooperation in different scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a multi-level 3D target expression method, system and medium applied to an automatic driving scene. BACKGROUND

[0002] In the field of automatic driving, a multi-level 3D target expression method is used to accurately and detailedly describe various objects on the road, so that the automatic driving system can understand and interact with these objects. The following are some key technologies related to this technical background: Point cloud processing (geometric method): Automatic driving systems usually use lidar or depth cameras to generate point cloud data, where each point represents a surface point of an object in the scene. Point cloud processing technology is used to segment, filter and feature extract point cloud data to obtain the geometric shape, position and motion information of the target object.

[0003] Object detection and recognition (deep learning method): In automatic driving, it is necessary to accurately detect and recognize various objects on the road, such as vehicles, pedestrians, bicycles, etc. Object detection and recognition technology uses deep learning models and computer vision algorithms to analyze input images or point cloud data to determine the category and location of the object.

[0004] 3D object modeling: In order to better understand and predict the motion and behavior of objects, the automatic driving system needs to model the detected objects in three dimensions. This includes estimating the size, pose, speed, etc. of the object and representing it as a 3D model with a hierarchical structure to better capture the geometric shape and motion characteristics of the object.

[0005] These technologies and methods together constitute the technical background of the multi-level 3D target expression method in automatic driving, providing the automatic driving system with a comprehensive understanding and perception of objects in the road scene. However, none of these methods provides a good 3D target expression. SUMMARY

[0006] In view of the above problems, the present application provides a multi-level 3D target expression method, system and medium applied to an automatic driving scene, which not only facilitates the selection of the optimal expression scheme for decision planning under suitable working conditions, but also solves the problem of perception and planning coordination in different scenarios.

[0007] To achieve the above objects and other related objects, the technical solutions provided by the present application are as follows:

[0008] A multi-level 3D target expression method applied to an automatic driving scene, the method comprising:

[0009] B1. The vehicle travels on the road, real-time obtains road image data information based on the vehicle-mounted camera, real-time obtains road point cloud data information based on the vehicle-mounted laser radar, and real-time obtains distance data information of the obstacle based on the vehicle-mounted ultrasonic radar;

[0010] B2. The road image data information is input into a convolutional neural network for recognition, and is classified according to highway signs, and highway scene classification data information and non-highway scene classification data information are output;

[0011] B3. Based on the highway scene classification data information, the road point cloud data information is projected onto the road image data information to obtain 2D semantic point cloud with a point direction, the 2D semantic point cloud with the point direction is fused with the distance data information of the obstacle, and 2D BEV data information of the road is output;

[0012] B4. Based on the non-highway scene classification data information, a 3D-CVF algorithm is used to fuse the road image data information and the road point cloud data information to obtain fused road data information, and according to the fused road data information and the distance data information of the obstacle, 3D frame data information of the vehicle, OccupancyCylinder data information of the pedestrian and Occupancy Grid data information of the narrow road passage are output.

[0013] Further, the convolutional neural network comprises a backbone network and a fully connected layer, and the backbone network comprises a convolutional calculation layer, an excitation layer and at least one pooling layer.

[0014] Further, the excitation function J(x) of the excitation layer is

[0015]

[0016]

[0017] wherein α and β are constant coefficients, x is an independent variable, and e is a natural constant.

[0018] Further, in step B3, the projection of the road point cloud data information onto the road image data information comprises:

[0019] B311. The road image data information is 2D segmented to output an ROI region of the road image;

[0020] B312. The road point cloud data information is projected onto the ROI region of the road image to perform image fusion to obtain 2D semantic point cloud with a point direction.

[0021] ​​Further, in step B3, the fusing of the 2D semantic point cloud with the distance data information of the obstacles includes:

[0022] B321. The distance data information of the obstacles is labeled, including the distance data information of the road boundary and the ego vehicle and the distance data information of other vehicles and the ego vehicle, and the labeled obstacle data information is outputted;

[0023] B322. Based on the labeled obstacle data information, the 2D semantic point cloud is labeled with an obstacle bounding box, and is optimized and fused, and the 2D BEV data information of the road is outputted.

[0024] Further, in step B4, the fusing of the road image data information and the road point cloud data information by using the 3D-CVF algorithm includes:

[0025] B41. Based on the road point cloud data information, a point cloud overhead view feature mapping is performed, and the overhead view mapping data information of the road point cloud is outputted;

[0026] B42. The overhead view mapping data information of the road point cloud and the road image data information are inputted into a gated feature fusion network model for feature fusion, and the road multi-feature fusion data information is outputted;

[0027] B43. Based on the road multi-feature fusion data information, vehicle-mounted high-precision map data information is coupled, and the fused road data information is outputted.

[0028] In order to achieve the above-mentioned purpose and other related purposes, the application further provides a multi-level 3D target expression system applied to an automatic driving scene, which comprises:

[0029] A data acquisition module is configured to acquire image data information, road point cloud data information and obstacle distance data information of a road;

[0030] A road scene classification module is connected with the data acquisition module and is configured to classify a road scene;

[0031] A road obstacle expression module is connected with the road scene classification module and is configured to perform 3D multi-level expression of obstacles.

[0032] Further, the system further comprises a data display module connected with the road obstacle expression module and configured to display image data information of obstacles in real time.

[0033] Further, the system further comprises a voice prompt module connected with the road obstacle expression module, for real-time broadcasting of the road surrounding obstacle data information of the vehicle.

[0034] To achieve the above object and other related objects, the present application further provides a computer readable storage medium, which stores a computer program programmed or configured to execute any one of the multi-level 3D target expression methods applied to the automatic driving scene.

[0035] The present application has the following positive effects:

[0036] 1. The present application forms a multi-level expression scheme according to different expression methods of 3D targets, which facilitates selection of the optimal expression scheme for decision planning under suitable working conditions.

[0037] 2. The present application adds scene classification according to the method of 3D target perception, selects a suitable 3D target expression method, and solves the problem of perception and planning coordination under different scenes.

[0038] 3. The present application defines a discrimination space for the driving problem of a real vehicle, selects a suitable 3D target expression method according to different obstacle intrusion working conditions, and solves the problem of perception expression under narrow roads and dense pedestrian / vehicle flow. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The figure is a flowchart of the method of the present application;

[0040] Figure 2 The figure is a 2D BEV table diagram of the present application;

[0041] Figure 3 The figure is a 3D box table diagram of the present application;

[0042] Figure 4 The figure is an Occupancy Cylinder expression diagram of the present application;

[0043] Figure 5 The figure is an Occupancy Grid table diagram of the present application. DETAILED DESCRIPTION

[0044] The exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0045] Embodiment 1

[0046] As shown in Figure 1 or Figure 2 or Figure 3 or Figure 4 or Figure 5 A multi-level 3D target expression method applied to an automatic driving scene, the method comprising:

[0047] B1. The vehicle travels on the road, real-time road image data information is obtained based on the vehicle-mounted camera, real-time road point cloud data information is obtained based on the vehicle-mounted laser radar, and distance data information of the obstacle is obtained based on the vehicle-mounted ultrasonic radar;

[0048] B2. The road image data information is input into a convolutional neural network for recognition, and is classified according to highway signs, and highway scene classification data information and non-highway scene classification data information are output;

[0049] B3. Based on the highway scene classification data information, the road point cloud data information is projected onto the road image data information to obtain 2D semantic point cloud with a point direction, and the 2D semantic point cloud with a point direction is fused with the distance data information of the obstacle to output 2D BEV data information of the road;

[0050] B4. Based on the non-highway scene classification data information, a 3D-CVF algorithm is used to fuse the road image data information and the road point cloud data information to obtain fused road data information, and based on the fused road data information and the distance data information of the obstacle, 3D frame data information of the vehicle, OccupancyCylinder data information of the pedestrian and Occupancy Grid data information of the narrow road passage are output.

[0051] In this embodiment, the convolutional neural network comprises a backbone network and a fully connected layer, and the backbone network comprises a convolutional calculation layer, an excitation layer and at least one pooling layer.

[0052] In this embodiment, the excitation function J(x) of the excitation layer is

[0053] ,

[0054] ,

[0055] Wherein, α and β are constant coefficients, x is the independent variable, and e is the natural constant.

[0056] In this embodiment, in step B3, the projection of the road point cloud data information onto the road image data information comprises:

[0057] B311.2D segment the road image data information, output the ROI region of the road image;

[0058] B312.project the road point cloud data information to the ROI region of the road image, perform image fusion, and obtain a 2D semantic point cloud with a point direction.

[0059] In the embodiment, in step B3, the fusion of the 2D semantic point cloud with a point direction and the distance data information of the obstacle includes:

[0060] B321. label the distance data information of the obstacle, the distance data information of the obstacle including distance data information of a road boundary and the vehicle itself and distance data information of other vehicles and the vehicle itself, and output labeled obstacle data information;

[0061] B322. based on the labeled obstacle data information, perform obstacle bounding box labeling on the 2D semantic point cloud with a point direction, and perform optimization and image fusion, and output 2D BEV data information of the road.

[0062] Embodiment 2:

[0063] Based on the multi-level 3D target expression method applied to the automatic driving scene in embodiment 1, the application will be further described and explained.

[0064] As shown in Figure 1 or Figure 2 or Figure 3 or Figure 4 or Figure 5 A multi-level 3D target expression method applied to an automatic driving scene, the method comprising:

[0065] B1. The vehicle travels on the road, real-time road image data information is obtained based on the vehicle-mounted camera, real-time road point cloud data information is obtained based on the vehicle-mounted laser radar, and distance data information of the obstacle is obtained based on the vehicle-mounted ultrasonic radar;

[0066] B2. input the road image data information into a convolutional neural network for recognition, and classify according to highway signs, output highway scene classification data information and non-highway scene classification data information;

[0067] B3. based on the highway scene classification data information, project the road point cloud data information to the road image data information, obtain a 2D semantic point cloud with a point direction, and fuse the 2D semantic point cloud with a point direction and the distance data information of the obstacle, and output 2D BEV data information of the road.

[0068] B4. Based on the non-high-speed scene classification data information, the 3D-CVF algorithm is used to fuse the road image data information and the road point cloud data information, to obtain fused road data information, and according to the fused road data information and the distance data information of the obstacle, the 3D frame data information of the vehicle, the OccupancyCylinder data information of the pedestrian and the Occupancy Grid data information of the narrow road passing are output.

[0069] In the embodiment, in step B4, the 3D-CVF algorithm is used to fuse the road image data information and the road point cloud data information, including:

[0070] B41. Based on the road point cloud data information, a point cloud overhead view feature mapping is performed, and overhead view mapping data information of the road point cloud is output.

[0071] B42. The overhead view mapping data information of the road point cloud and the road image data information are input into a gated feature fusion network model for feature fusion, and road multi-feature fusion data information is output.

[0072] B43. Based on the road multi-feature fusion data information, vehicle-mounted high-precision map data information is coupled, and fused road data information is output.

[0073] The application provides a multi-level 3D target expression system applied to an automatic driving scene, the system comprising:

[0074] A data acquisition module is configured to acquire image data information of a road, road point cloud data information and obstacle distance data information.

[0075] A road scene classification module is connected with the data acquisition module and is configured to classify a road scene.

[0076] A road obstacle expression module is connected with the road scene classification module and is configured to perform 3D multi-level expression of an obstacle.

[0077] In the embodiment, the system further comprises a data display module connected with the road obstacle expression module and configured to display image data information of the obstacle in real time.

[0078] In the embodiment, the system further comprises a voice prompt module connected with the road obstacle expression module and configured to broadcast road obstacle data information of a vehicle in real time.

[0079] The application provides a computer readable storage medium, which stores a computer program programmed or configured to perform any one of the multi-level 3D target expression methods applied to an automatic driving scene.

[0080] Any reference to storage, memory, database or other medium herein can include non-volatile and / or volatile storage. Non-volatile storage can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile storage can include random-access memory (RAM), or external cache memory. By way of illustration, and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). The disclosure should make it manifestly clear that the scope of the disclosure is made not subject to the RAM types recited herein.

[0081] In summary, the present application not only facilitates selecting the optimal expression scheme under suitable working conditions for decision planning, but also solves the problem of perception and planning coordination under different scenarios.

[0082] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multi-level 3D object representation method applied to an autonomous driving scenario, characterized in that, The method comprises: B1. The vehicle travels on the road, real-time road image data information is obtained based on the vehicle-mounted camera, real-time road point cloud data information is obtained based on the vehicle-mounted laser radar, and distance data information of obstacles is obtained in real time based on the vehicle-mounted ultrasonic radar; B2. The road image data information is input into a convolutional neural network for recognition, and classified according to highway signs, and highway scene classification data information and non-highway scene classification data information are output; B3. Based on the highway scene classification data information, the road point cloud data information is projected onto the road image data information to obtain 2D semantic point cloud with a point direction, and the 2D semantic point cloud with a point direction is fused with the distance data information of the obstacles to output 2D BEV data information of the road; B4. Based on the non-highway scene classification data information, the 3D-CVF algorithm is used to fuse the road image data information and the road point cloud data information to obtain fused road data information, and based on the fused road data information and the distance data information of the obstacles, 3D frame data information of the vehicle, OccupancyCylinder data information of pedestrians and Occupancy Grid data information of narrow road passage are output; The convolutional neural network comprises a backbone network and a fully connected layer, the backbone network comprises a convolutional calculation layer, an excitation layer and at least one pooling layer; the excitation function J(x) of the excitation layer is ， ， Wherein, α and β are constant coefficients, x is an independent variable, and e is a natural constant; In step B3, the projection of the road point cloud data information onto the road image data information comprises: B311. The road image data information is 2D segmented to output the ROI region of the road image; B312. The road point cloud data information is projected onto the ROI region of the road image to perform image fusion and obtain 2D semantic point cloud with a point direction; In step B4, the fusion of the road image data information and the road point cloud data information by the 3D-CVF algorithm comprises: B41. Based on the road point cloud data information, a point cloud overhead view feature mapping is performed to output overhead view mapping data information of the road point cloud; B42. The overhead view mapping data information of the road point cloud and the road image data information are input into a gated feature fusion network model for feature fusion to output road multi-feature fusion data information; B43. Based on the road multi-feature fusion data information, vehicle-mounted high-precision map data information is coupled to output fused road data information. 2.The multi-level 3D object representation method for autonomous driving scene of claim 1, wherein, In step B3, the fusion of the 2D semantic point cloud with a point direction and the distance data information of the obstacles comprises: B321. The distance data information of the obstacles is labeled, the distance data information of the obstacles comprises distance data information of the road boundary and the vehicle itself and distance data information of other vehicles and the vehicle itself, and labeled obstacle data information is output; B322. Based on the annotated obstacle data information, the 2D semantic point cloud with point direction is annotated with an obstacle graph frame, and is optimized and image fused to output 2D BEV data information of the road.

3. A multi-level 3D object representation system applied to an autonomous driving scenario, characterized in that, The system for implementing the multi-level 3D target expression method applied to the automatic driving scene according to any one of claims 1-2, the system comprises: a data acquisition module configured to acquire image data information, road point cloud data information, and obstacle distance data information of a road; a road scene classification module connected to the data acquisition module and configured to classify a road scene; a road obstacle expression module connected to the road scene classification module and configured to perform 3D multi-level expression of an obstacle.

4. The multi-level 3D object representation system for autonomous driving scenarios of claim 3, wherein: The system further comprises a data display module connected to the road obstacle expression module and configured to display image data information of the obstacle in real time.

5. The multi-level 3D object representation system for autonomous driving scenarios of claim 3, wherein: The system further comprises a voice prompt module connected to the road obstacle expression module and configured to broadcast obstacle data information around the road of the vehicle in real time.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program programmed or configured to perform the multi-level 3D target expression method applied to the automatic driving scene according to any one of claims 1-2.

Images