Driving environment sensing method, system and device and computer readable storage medium

Abstract

The invention discloses a driving environment sensing method, system and device and a computer readable storage medium. The method comprises the steps of obtaining radar point cloud data and camera image data; after the radar data features are converted into a voxel space, vertical coding is carried out to obtain radar BEV features; converting the camera data features into a voxel space to obtain camera voxel features, and filling the camera voxel features into a preset 3D container in a BEV view space to obtain camera BEV features; carrying out dimension reduction on the camera BEV features, and splicing the camera BEV features with the radar BEV features to obtain enhanced BEV features; sampling position offset is obtained according to the enhanced radar BEV features; performing autocorrelation feature fusion on the camera BEV feature and the enhanced BEV feature according to the sampling position offset to obtain a fused BEV feature; and performing driving environment perception according to the fused BEV features to obtain a driving environment perception result. The technical problems that in the prior art, a BEV sensing scheme based on multiple types of sensors is high in cost, large in computing power consumption and not ideal in sensing effect can be solved.

Description

Technical Field

[0001] The present application relates to the field of multi-type sensor fusion perception, and specifically to a driving environment perception method, system, device and computer-readable storage medium. Background Art

[0002] With the rapid improvement of hardware computing power and the widespread popularization of intelligent network technology, multi-modal vehicle sensors, such as cameras, millimeter-wave radars, and corresponding multi-modal perception data processing technologies, are gradually becoming one of the key technologies for the development of automatic intelligent driving. In this context, multi-sensor BEV (Bird's Eye View) perception technology is gradually becoming the mainstream perception solution in the field of automatic intelligent driving due to its accurate perception capabilities across fields of vision and modality.

[0003] Chinese patent (application number: CN202310332750.8) A target detection method and device for the fusion of millimeter-wave radar and camera data discloses that based on the relative position relationship between the millimeter-wave radar and the camera and the internal parameters of the camera, the millimeter-wave radar data points are projected to the pixel position in the image plane, and a corresponding relationship is established based on the relationship between the pixel and the pixel area range of the target. Finally, the millimeter-wave radar data points are filtered and clustered using the target category information based on the image, and the position and speed information of the millimeter-wave radar data points are used to define the position and speed of the target, and the complete three-dimensional detection information of the target is output.

[0004] The shortcomings of this solution are:

[0005] 1. Strong dependence on relative position and internal parameters: This method depends on the relative position relationship between the millimeter-wave radar and the camera and the internal parameters of the camera. If these parameters are inaccurate or change, it may lead to inaccurate data projection and target detection.

[0006] 2. Pixel relationship assumptions about the target area: This method assumes that there is a corresponding relationship between the pixel position of the target in the image plane and its position in the millimeter-wave radar data point. However, in some cases, this relationship may be unstable or inaccurate due to factors such as target posture changes or occlusion.

[0007] 3. Limitations of image data: Although image-based target category information is used to filter and cluster millimeter-wave radar data points, the image data itself may be affected by factors such as lighting, shadows, and occlusions, resulting in reduced accuracy in target detection and classification.

[0008] The Chinese patent (application number: CN202311582081.6) is based on a 3D target detection method based on the fusion of image and 4D millimeter wave radar. It discloses that a neural network is constructed by simultaneously reading image and point cloud data, preprocessing and voxelizing. The loss function is determined using real label data, and the final loss function is obtained by weighted summation. The neural network is back-propagated and parameter updated to train a neural network that can be used for target detection. The defects of this solution are:

[0009] 1. Data fusion effect: How to ensure the effective fusion of image and millimeter-wave radar data, avoid information redundancy or loss, and ensure the accuracy and robustness of target detection.

[0010] 2. Diversity of training samples: whether it covers targets in various complex scenarios and data samples under different environmental conditions to ensure that the trained neural network has wide adaptability.

[0011] 3. Real-time and efficiency: During the target detection process, whether the real-time requirements can be met, and whether the computing efficiency and resource consumption when processing large-scale data are controllable.

[0012] The Chinese patent (application number: CN202110955241.1) discloses a 4D millimeter wave three-dimensional target detection method and system based on the self-attention mechanism, which first collects and preprocesses 4D millimeter wave radar point cloud data in real time, and then inputs the data into a pre-trained three-dimensional target detection model to obtain the target detection result. The model includes a bird's-eye view voxelization module, a column self-attention feature extraction module, a CNN backbone network, and a PRN detection head. The bird's-eye view voxelization module is used to extract spatial feature information, and the column self-attention feature extraction module uses the self-attention mechanism to obtain the global features of the point cloud and generate a BEV pseudo image. The defects of this solution are:

[0013] 1. High computing resource requirements: Using technologies such as self-attention mechanism and deep neural network may require a lot of computing resources and computing power support, which may limit the feasibility of this method in practical applications.

[0014] 2. Real-time performance: The data processing and model reasoning processes involved in the method may affect the real-time performance of the system, especially in scenarios such as autonomous driving that require rapid response, where real-time requirements are high.

[0015] In addition, since BEV perception technology is oriented to multi-view and multi-modal perception and related research is relatively limited, it still faces many challenges. The factors that most restrict the implementation of BEV perception technology are:

[0016] 1. The BEV perception model consumes a lot of computing power and has high requirements for the computing power platform, which restricts the practicality of the model.

[0017] 2. Current mainstream BEV perception solutions rely on a large number of sensor configurations, which greatly increases the implementation cost of these solutions. Summary of the invention

[0018] The present application provides a driving environment perception method, system, device and computer-readable storage medium, which can solve the technical problems existing in the prior art of high cost, high computing power consumption and unsatisfactory perception effect of BEV perception solutions based on multiple types of sensors.

[0019] In a first aspect, an embodiment of the present application provides a driving environment perception method, characterized in that the method comprises:

[0020] After converting the radar data features into the voxel space corresponding to the BEV view space, vertical encoding is performed to obtain the radar BEV features;

[0021] After converting the camera data features into the voxel space, the camera voxel features are obtained, and the camera voxel features are filled into a preset 3D container in the BEV view space to obtain the camera BEV features;

[0022] The camera BEV feature is reduced in dimension and then spliced ​​with the radar BEV feature to obtain an enhanced BEV feature; the enhanced radar BEV feature is encoded and generated using the sampling area to obtain a sampling position offset; the camera BEV feature and the enhanced BEV feature are autocorrelatedly fused according to the sampling position offset to obtain a fused BEV feature;

[0023] The driving environment perception is performed based on the fused BEV characteristics to obtain the driving environment perception results.

[0024] In combination with the first aspect, in one implementation, extracting corresponding radar data features and camera data features specifically includes the following steps:

[0025] Extracting point feature tensors from radar point cloud data, normalizing the point feature tensors, and generating radar data features based on the BEV view space;

[0026] The image feature tensor is extracted from the camera image data, and after normalizing the image feature tensor, the camera data feature is generated based on the voxel space.

[0027] In combination with the first aspect, in one implementation, when filling the camera voxel features into a preset 3D container in the BEV view space, data exceeding a preset range of the 3D container is filtered to obtain the camera BEV features.

[0028] In combination with the first aspect, in one implementation, when the camera BEV features are reduced in dimension, the dimensions of the camera BEV features after the reduction in dimension are the same as those of the radar BEV features.

[0029] In combination with the first aspect, in one embodiment, when the reduced-dimensional camera BEV features are spliced ​​with the radar BEV features, the splicing is performed according to the geometric distribution of a preset grid in the BEV view space to fill the reduced-dimensional camera BEV features into the radar BEV features to obtain the enhanced BEV features.

[0030] In combination with the first aspect, in one implementation, when encoding the enhanced radar BEV features, the enhanced radar BEV features are encoded using a convolutional layer, a batch normalization layer, and an activation function layer in sequence to obtain encoded data.

[0031] In a second aspect, an embodiment of the present application provides a driving environment perception system, the system comprising:

[0032] A data acquisition module, which is used to obtain radar point cloud data and camera image data;

[0033] A data processing module is used to extract corresponding radar data features and camera data features; after converting the radar data features into a voxel space corresponding to the BEV view space, vertical encoding is performed to obtain radar BEV features; after converting the camera data features into the voxel space, camera voxel features are obtained, and the camera voxel features are filled into a preset 3D container in the BEV view space to obtain camera BEV features; after reducing the dimension of the camera BEV features, the camera BEV features are spliced ​​with the radar BEV features to obtain enhanced BEV features; after encoding the enhanced radar BEV features, the sampling area is used to generate them to obtain a sampling position offset; after performing feature fusion on the camera BEV features and the enhanced BEV features according to the sampling position offset, a fused BEV feature is obtained;

[0034] The data analysis module is used to perceive the driving environment according to the integrated BEV characteristics and obtain the driving environment perception results.

[0035] In combination with the first aspect, in one embodiment, the data acquisition module includes multiple radar devices and multiple camera devices, and the multiple radar devices and the multiple camera devices cover a preset area around the vehicle to collect radar point cloud data and camera image data of all points in the preset area.

[0036] In a third aspect, an embodiment of the present application provides a driving environment perception device, which includes a processor, a memory, and a driving environment perception program stored in the memory and executable by the processor, wherein when the driving environment perception program is executed by the processor, the steps of the driving environment perception method are implemented.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a driving environment perception program is stored, wherein when the driving environment perception program is executed by a processor, the steps of the driving environment perception method are implemented.

[0038] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:

[0039] For each point in the preset area, the radar device collects radar point cloud data and the camera device collects camera image data, and the view spaces of both are converted into the same BEV view space, thereby achieving geometric alignment of the radar point cloud data and the camera image data. The camera BEV features are obtained with the help of the preset 3D container in the BEV view space, and the strong geometric alignment of the radar point cloud data and the camera image data is further achieved to ensure the consistency of the radar perception target and the camera perception target, effectively alleviating the error recognition problem caused by the differences between different sensors in the perception of the autonomous driving environment, thereby improving the accuracy of the subsequent driving environment perception.

[0040] When converting radar data features to the voxel space corresponding to the BEV view space, vertical encoding is used for encoding, which can alleviate the problem of sparse encoding information caused by the data sparsity of the radar point cloud data itself and avoid data loss.

[0041] The enhanced BEV features are obtained by concatenating the camera BEV features with the radar BEV features after dimensionality reduction. The sampling position offset is obtained according to the enhanced radar BEV features.

[0042] Obtain radar point cloud data and camera image data, and extract corresponding radar data features and camera data features;

[0043] The offset is used to perform autocorrelation feature fusion on the camera BEV features and the enhanced BEV features, without introducing the attention mechanism, thus reducing the computational cost. In addition, the attention mechanism itself is not limited to its dependence on the network model, and the change value of the offset can be automatically adjusted through network model training, which can achieve better adaptability to driving scenarios. The implemented model has better robustness, and therefore performs better in more complex scenarios.

[0044] For the preset area around the vehicle, the number of radar devices and camera devices arranged on the vehicle only needs to meet the collection range covering the preset area to achieve driving environment perception, reducing the cost of installing sensors on the vehicle side. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of an embodiment of a driving environment perception method of the present application;

[0046] Figure 2 This is a flow chart of an embodiment of a driving environment perception method of the present application;

[0047] Figure 3 This is a schematic diagram of the functional modules of an embodiment of the driving environment perception system of the present application;

[0048] Figure 4 A schematic diagram of the sensor layout of an embodiment of the driving environment perception system of the present application;

[0049] Figure 5 This is a schematic diagram of the hardware structure of the driving environment perception device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0052] In a first aspect, an embodiment of the present application provides a driving environment perception method.

[0053] In one embodiment, if Figure 1 and Figure 2 As shown, the driving environment perception method includes:

[0054] Step S1, obtaining radar point cloud data and camera image data, and extracting corresponding radar data features and camera data features;

[0055] Step S2, converting the radar data features into a voxel space corresponding to the BEV view space, and performing vertical encoding to obtain radar BEV features;

[0056] Step S3, after converting the camera data features into voxel space, obtaining the camera voxel features, filling the camera voxel features into a preset 3D container in the BEV view space, obtaining the camera BEV features;

[0057] Step S4, reducing the dimension of the camera BEV feature and concatenating it with the radar BEV feature to obtain an enhanced BEV feature;

[0058] Step S5, after encoding the enhanced radar BEV feature, use the sampling area to generate and obtain the sampling position offset;

[0059] Step S6, after performing autocorrelation feature fusion on the camera BEV feature and the enhanced BEV feature according to the sampling position offset, a fused BEV feature is obtained;

[0060] Step S7: Perform driving environment perception based on the integrated BEV characteristics to obtain a driving environment perception result.

[0061] Step S2 and step S3 can be performed simultaneously.

[0062] In this embodiment, for each point in the preset area, the radar device collects radar point cloud data and the camera device collects camera image data, and the view spaces of both are converted into the same BEV view space, thereby achieving geometric alignment of the radar point cloud data and the camera image data. With the help of the preset 3D container in the BEV view space, features are filled into the container to obtain the camera BEV features, and further achieve strong geometric alignment of the radar point cloud data and the camera image data, ensuring the consistency of the radar perception target and the camera perception target, and effectively alleviating the error recognition problem caused by the differences between different sensors in the perception of the autonomous driving environment, thereby improving the accuracy of the subsequent driving environment perception.

[0063] When converting radar data features to the voxel space corresponding to the BEV view space, using a vertical encoding method, such as pillar encoding, can alleviate the problem of sparse encoding information caused by the data sparsity of the radar point cloud data itself and avoid data loss.

[0064] The enhanced BEV feature is obtained by reducing the dimension of the camera BEV feature and concatenating it with the radar BEV feature. The sampling position offset is obtained according to the enhanced radar BEV feature. The camera BEV feature and the enhanced BEV feature are autocorrelatedly fused according to the sampling position offset. There is no need to introduce an attention mechanism, which reduces the computational cost. In addition, the attention mechanism itself is not limited to its dependence on the network model. The change value of the offset can be automatically adjusted through network model training, which can achieve better adaptability to driving scenarios. The implemented model has better robustness and therefore performs better in more complex scenarios.

[0065] For the preset area around the vehicle, the number of radar devices and camera devices arranged on the vehicle only needs to meet the collection range covering the preset area to achieve driving environment perception, reducing the cost of installing sensors on the vehicle side.

[0066] Furthermore, in one embodiment, the extracting of corresponding radar data features and camera data features specifically includes the following steps:

[0067] Extracting point feature tensors from radar point cloud data, normalizing the point feature tensors, and generating radar data features based on the BEV view space;

[0068] The image feature tensor is extracted from the camera image data, and after normalizing the image feature tensor, the camera data feature is generated based on the voxel space.

[0069] In this embodiment, the vehicle-side layout and sensing range of different types of sensors are very different. For the problem of data inconsistency, normalization can be used to process the data level to alleviate the data deviation caused by this problem. The radar data features and camera data features are obtained by using the following formulas (1) and (2):

[0070]

[0071] Among them, f radar Represents radar data characteristics. camera Represents camera data characteristics. Represents the grid points of the BEV view space. v<> represents the voxel space container, representing the intermediate mode before aligning the camera data to the BEV space. Φ radar () represents the feature extraction of radar point cloud data, Φ camera () represents the feature extraction of camera image data, Represents the normalization operation, which is used to normalize the data value to the range of 0 to 1 to alleviate the error of data distribution. P represents the point feature tensor, and I represents the image feature tensor. Represents all the physical information of the millimeter-wave radar, such as the coordinates of the millimeter-wave radar point, the reflection value and speed corresponding to the millimeter-wave radar point, etc. d represents the amount of information contained in each radar point data.

[0072] Formula (1) and formula (2) can be used to obtain the radar feature data and camera data features in the BEV view space. Since the BEV view space is a spatial representation based on the vehicle itself, the BEV perception architecture can maximize the spatial geometric feature alignment of the radar point cloud data and the camera image data.

[0073] Furthermore, in one embodiment, when filling the camera voxel features into a preset 3D container in the BEV view space, data beyond the preset range of the 3D container is filtered to obtain the camera BEV features.

[0074] In this embodiment, after the geometric alignment process, the features of different types of sensors need to be processed separately into the corresponding BEV view space.

[0075] Therefore, considering that different types of sensor data have different representation forms, radar data is represented in the form of 3D points, that is, Among them, P and A represent the x, y, z three-dimensional spatial point information of the radar, as well as the corresponding speed, reflection value and other attribute information. The pixel information of the image is composed of two points and one pixel value. If the BEV bird's-eye view is generated directly by coordinate mapping and data sampling, it will bring huge differences and interference, so the radar data features and camera data need to be processed separately.

[0076] For radar data features, the point encoding method is used in the vertical direction, that is, the vertical encoding method. This method is used to uniformly encode the information of the current point to alleviate the problem of in-density encoding information caused by sparse radar points. After encoding, the radar BEV feature can be obtained. The feature dimension of the radar BEV feature can be expressed as Among them C radar , H and W represent the number of channels for extracting features, and the height and width of the features, respectively.

[0077] For the processing of camera data features, a predefined container filling method is designed to generate camera BEV features, which can be completed in three steps:

[0078] First, we use the image feature extraction backbone network Resnet50 to extract image features. The extracted camera data features can be expressed as 3 represents the different viewing angles of the three cameras, C cams , H cams and W cams Respectively represent the number of extracted image feature channels and the height and width of the features.

[0079] Define a format The three-dimensional visual data, that is, the 3D container, is initialized The data is all 0, and then the calibration mapping matrix from the driving vehicle inertial navigation coordinate system (global coordinate system) to the camera (camera coordinate system) is used to map the X, Y, and Z data to the image coordinate system, that is, from three-dimensional space to image space, set to Of course, the mapping matrices of the three cameras are used in the mapping process, which will generate three different sets of X, Y, and Z mapping coordinate values. To filter and record effective values, set

[0080] Finally, use Image features to Filling the container with data. After filling, you can get a container with data that is not all zero. Then By transforming the tensor dimension from dimension Transformed into After mapping the Z-axis data to the XY plane, the resulting camera BEV feature is expressed as Where W and H correspond to X and Y respectively.

[0081] Completed the geometric alignment analysis in the unified BEV view space, completed the BEV view space feature conversion of different types of sensors, and generated radar BEV features and camera BEV features Compared with the method of mapping millimeter-wave radar data to image coordinates in the prior art, this solution adopts feature-level processing. Since the feature level belongs to a higher-dimensional data representation, this processing method can significantly reduce the errors that may occur in the early coordinate conversion.

[0082] Compared with the method of directly processing millimeter-wave radar data into voxels in the second prior art, this solution adopts the method of regional vertical encoding, which can better capture the coding features within the region. Compared with the relatively sparse voxel processing method, this method can provide richer feature information, thereby improving the network model's ability to understand driving scenes.

[0083] Furthermore, in one embodiment, when the camera BEV features are reduced in dimension, the camera BEV features after the reduction in dimension have the same dimension as the radar BEV features.

[0084] When splicing the camera BEV features after dimensionality reduction with the radar BEV features, the splicing is performed according to the geometric distribution of the preset grid in the BEV view space, so as to fill the camera BEV features after dimensionality reduction into the radar BEV features to obtain the enhanced BEV features.

[0085] In this embodiment, even if the data alignment of different types of sensor features is completed geometrically, the obtained feature data will still have differences in data distribution, and this difference will cause inaccurate perception detection results, ultimately affecting the performance of the autonomous driving function.

[0086] Will Features and The features are spliced ​​at the channel level according to the geometric distribution of the BEV view space grid. In this way, the sparse radar BEV features are supplemented with the camera BEV features, and this step generates enhanced BEV features with higher dimensions.

[0087] Furthermore, in one embodiment, when encoding the enhanced radar BEV features, the enhanced radar BEV features are encoded using a convolutional layer, a batch normalization layer, and an activation function layer in sequence to obtain encoded data.

[0088] In this embodiment, in order to further explore and To improve the relationship between the objects and the detection accuracy of the objects under the BEV view, and to ensure that the BEV view can more fully integrate the scene data, autocorrelation feature fusion is also required.

[0089] The fusion of autocorrelation features needs to solve two key problems: Bird's-eye view features and The first is the alignment problem between bird’s-eye view features, and the other is to achieve efficient fusion.

[0090] The autocorrelation feature fusion is divided into two branches. After the convolution layer, batch normalization, and activation function calculation encoding, the sampling area is used to generate the sampling position of the normalized result. Compared with the method of directly using deformable convolution, this method can better reduce the computing power requirements of network calculation. The specific sampling area generation calculation adopts the following formula (3):

[0091]

[0092] in, represents the sampling features obtained by sampling point offset, and H represents The characteristic height of the feature, Represents the use of CBR to generate sampling position offset features for and Characteristics are distinguished, h means according to The continuous sampling points set by the height of the feature, w represents the The width of the feature sets the number of consecutive sampling points, Δ 1hw Indicates the offset (height direction) of the sampling point generated using CBR, Δ 2hw Indicates the offset (width direction) of the sampling points generated by CBR.

[0093] F BiLrepresents the bilinear interpolation sampling function, h′, w′ and 1hw, 2hw represent the size of the feature map and the offset size of the sampling position respectively, <*> indicates how to calculate the sampling position of the offset point, and the aggregation of features in a wider range is achieved by setting the offset. This step can not only better align the two sets of BEV features through network training by calculating the offset, but also aggregate more related features from different features for the same target, thereby improving the recognition ability of the target.

[0094] Finally, the obtained samples Characteristics (left side of equation 3) and original camera BEV characteristics Then splice and fuse them to achieve the enhancement and fusion of autocorrelation features at a time. and All included By using this relationship, the correlation features can be enhanced and fused. The fused features will be used in the final detection and decoding part to obtain the detection results under the BEV view. For the convenience of introduction, this part of the features is called the fused BEV features that integrate richer features.

[0095] Compared with the existing technology 3, this solution does not introduce a more computationally intensive attention mechanism. Instead, it uses sampling position offsets to achieve autocorrelation enhancement and fusion of network features, which results in lower computational costs. In addition, it is not limited to the dependence of attention itself on the network model. The change value of the offset can be automatically adjusted through network model training, which can achieve better adaptability to driving scenarios. The implemented model has better robustness, and therefore performs better in more complex scenarios.

[0096] In summary, the data geometric alignment problem between millimeter-wave radar data and camera data is fully considered. With the help of the defined BEV grid data, the pillar bird's-eye view features generated by the millimeter-wave radar are associated with the designed predefined container filling image bird's-eye view features. The radar coordinate system and the camera image coordinate system are associated through a predefined BEV grid coordinate to achieve stronger geometric alignment, ensure the consistency of radar perception targets and camera perception targets, and effectively alleviate the misrecognition problem caused by the differences between different sensors in the perception of autonomous driving environment.

[0097] By utilizing the aligned millimeter-wave radar BEV features and camera BEV features, the millimeter-wave radar and image BEV features are spliced ​​through the radar image BEV enhancement module, and then the autocorrelation feature fusion method is adopted. There is no need to introduce the attention mechanism, which reduces the computational cost and improves the model robustness.

[0098] In a second aspect, an embodiment of the present application also provides a driving environment perception device.

[0099] In one embodiment, referring to Figure 3 , Figure 3 This is a functional module diagram of an embodiment of the driving environment perception device of the present application. Figure 3 As shown, the driving environment perception device includes:

[0100] A data acquisition module, which is used to obtain radar point cloud data and camera image data;

[0101] A data processing module is used to extract corresponding radar data features and camera data features; after converting the radar data features into a voxel space corresponding to the BEV view space, vertical encoding is performed to obtain radar BEV features; after converting the camera data features into the voxel space, camera voxel features are obtained, and the camera voxel features are filled into a preset 3D container in the BEV view space to obtain camera BEV features; after reducing the dimension of the camera BEV features, the camera BEV features are spliced ​​with the radar BEV features to obtain enhanced BEV features; after encoding the enhanced radar BEV features, the sampling area is used to generate them to obtain a sampling position offset; after performing feature fusion on the camera BEV features and the enhanced BEV features according to the sampling position offset, a fused BEV feature is obtained;

[0102] The data analysis module is used to perceive the driving environment according to the integrated BEV characteristics and obtain the driving environment perception results.

[0103] Furthermore, in one embodiment, the data acquisition module includes multiple radar devices and multiple camera devices, and the multiple radar devices and the multiple camera devices cover a preset area around the vehicle to collect radar point cloud data and camera image data of all points in the preset area.

[0104] At present, there are many sensor layout schemes for autonomous driving. The current mainstream perception scheme relies heavily on multiple camera sensors, such as the traditional 8-camera (8V) scheme, 11-camera + 5 millimeter-wave radar (11V5R) scheme, etc. The computing power requirements will account for 50% or even more of the entire vehicle computing power chip, and require higher hardware costs. Due to the extremely high integration of autonomous driving vehicles, the allocation of chip computing power is very strict. The solution under the current "perception-decision-control" framework design requires a lot of computing power coordination and support. While more sensors bring better perception performance, they will also greatly increase computing power consumption and hardware costs, resulting in a significant increase in the cost of the entire vehicle. Although fewer sensor configurations can achieve basic perception functions, they cannot meet the perception requirements in complex scenarios.

[0105] In a specific embodiment, a solution with both computing power and perception performance is provided through extensive research on current sensor layouts and solutions. Figure 4, design a 360° (5R3V) full-scene sensor layout solution based on 5 millimeter-wave radars + 3 surround-view cameras. This solution only uses 3 surround-view cameras and 5 millimeter-wave radars to make up for the blind spots that may exist when collecting 360° full-scene data for the preset area, reduce hardware and computing power costs, and achieve better environmental perception performance. Each millimeter-wave radar can divide the 360° scene into 72°, and each surround-view camera can divide the 360° scene into 120°. By cross-laying out millimeter-wave radars and surround-view cameras, a full coverage layout of the 360° scene can be achieved.

[0106] The use of three surround-view cameras and five millimeter-wave radars to complement the perception area information can more accurately perceive various targets in the preset area. At the same time, the surround-view camera can efficiently obtain dense pixel data to make up for the limitations of insufficient perception accuracy caused by the sparseness of millimeter-wave radar data. The millimeter-wave radar can more accurately collect information such as the speed and depth of objects in the scene, making up for the shortcomings of the camera image data in the inaccurate three-dimensional space geometry perception. The sensor layout solution of 5R3V uses a very simple hardware configuration and layout, which improves the perception ability of the final vehicle sensor equipment for autonomous driving scenes under the premise of extremely low hardware and computing power costs.

[0107] Using 5 millimeter-wave radars to cover the 360° full scene and 3 surround-view cameras as a sensor supplement can make good use of the characteristics of millimeter-wave radar. By combining the BEV generation method of camera filling with predefined containers and the BEV characteristics of millimeter-wave radar, the vehicle's recognition and detection capabilities for close-range targets can be improved, alleviating the target splitting and false detection problems caused by the fusion of millimeter-wave radar perception targets and camera perception targets.

[0108] The sensor layout of 5R3V obtains BEV features with richer perception information by filling and complementing the radar point cloud data and camera image data, and using autocorrelation feature enhancement and fusion. These rich perception features can meet the realization of more and more advanced autonomous driving functions at a lower cost.

[0109] Among them, the functional implementation of each module in the above-mentioned driving environment perception device corresponds to the various steps in the above-mentioned driving environment perception method embodiment, and its functions and implementation processes will not be repeated here one by one.

[0110] On the third aspect, an embodiment of the present application provides a driving environment perception device, which may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0111] Reference Figure 5 , Figure 5 Schematic diagram of the hardware structure of the driving environment perception device involved in the embodiment of the present application. In the embodiment of the present application, the driving environment perception device may include a processor, a memory, a communication interface and a communication bus.

[0112] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0113] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the driving environment perception device, and the interface used to realize the interconnection between the driving environment perception device and other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.

[0114] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0115] The processor may be a general-purpose processor, and the general-purpose processor may call the driving environment perception program stored in the memory and execute the driving environment perception method provided by the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the driving environment perception program is called may refer to the various embodiments of the driving environment perception method of the present application, which will not be repeated here.

[0116] Those skilled in the art will understand that Figure 5 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0117] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0118] A driving environment perception program is stored on the computer-readable storage medium of the present application, and when the driving environment perception program is executed by a processor, the steps of the driving environment perception method as described above are implemented.

[0119] Among them, the method implemented when the driving environment perception program is executed can refer to the various embodiments of the driving environment perception method of the present application, and will not be repeated here.

[0120] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0121] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0122] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0123] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or. For example, A / B can mean A or B. The “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0124] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0125] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0126] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A driving environment perception method, characterized in that: The method comprises: Obtain radar point cloud data and camera image data, and extract corresponding radar data features and camera data features; After converting the radar data features into the voxel space corresponding to the BEV view space, vertical encoding is performed to obtain the radar BEV features; After converting the camera data features into the voxel space, the camera voxel features are obtained, and the camera voxel features are filled into a preset 3D container in the BEV view space to obtain the camera BEV features; The camera BEV feature is reduced in dimension and then spliced ​​with the radar BEV feature to obtain an enhanced BEV feature; the enhanced radar BEV feature is encoded and generated using the sampling area to obtain a sampling position offset; the camera BEV feature and the enhanced BEV feature are autocorrelatedly fused according to the sampling position offset to obtain a fused BEV feature; The driving environment perception is performed based on the fused BEV characteristics to obtain the driving environment perception results.

2. The driving environment perception method according to claim 1, characterized in that: The extracting of corresponding radar data features and camera data features specifically includes the following steps: Extracting point feature tensors from radar point cloud data, normalizing the point feature tensors, and generating radar data features based on the BEV view space; The image feature tensor is extracted from the camera image data, and after normalizing the image feature tensor, the camera data feature is generated based on the voxel space.

3. The driving environment perception method according to claim 1, characterized in that: When filling the camera voxel features into a preset 3D container in the BEV view space, data beyond the preset range of the 3D container is filtered to obtain the camera BEV features.

4. The driving environment perception method according to claim 1, characterized in that: When the camera BEV features are reduced in dimension, the dimensions of the camera BEV features after the reduction in dimension are the same as those of the radar BEV features.

5. The driving environment perception method according to claim 1, characterized in that: When splicing the camera BEV features after dimensionality reduction with the radar BEV features, the splicing is performed according to the geometric distribution of the preset grid in the BEV view space, so as to fill the camera BEV features after dimensionality reduction into the radar BEV features to obtain the enhanced BEV features.

6. The driving environment perception method according to claim 1, characterized in that: When encoding the enhanced radar BEV features, the enhanced radar BEV features are encoded using a convolution layer, a batch normalization layer, and an activation function layer in sequence to obtain encoded data.

7. A driving environment perception system, characterized in that: The system comprises: A data acquisition module, which is used to obtain radar point cloud data and camera image data; A data processing module is used to extract corresponding radar data features and camera data features; after converting the radar data features into a voxel space corresponding to the BEV view space, vertical encoding is performed to obtain radar BEV features; after converting the camera data features into the voxel space, camera voxel features are obtained, and the camera voxel features are filled into a preset 3D container in the BEV view space to obtain camera BEV features; after reducing the dimension of the camera BEV features, the camera BEV features are spliced ​​with the radar BEV features to obtain enhanced BEV features; after encoding the enhanced radar BEV features, the sampling area is used to generate them to obtain a sampling position offset; after performing feature fusion on the camera BEV features and the enhanced BEV features according to the sampling position offset, a fused BEV feature is obtained; The data analysis module is used to perceive the driving environment according to the integrated BEV characteristics and obtain the driving environment perception results.

8. The driving environment perception method according to claim 7, characterized in that: The data acquisition module includes multiple radar devices and multiple camera devices, and the multiple radar devices and the multiple camera devices cover a preset area around the vehicle to collect radar point cloud data and camera image data of all points in the preset area.

9. A driving environment perception device, characterized in that: The driving environment perception device includes a processor, a memory, and a driving environment perception program stored in the memory and executable by the processor, wherein when the driving environment perception program is executed by the processor, the steps of the driving environment perception method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a driving environment perception program, wherein when the driving environment perception program is executed by a processor, the steps of the driving environment perception method according to any one of claims 1 to 7 are implemented.

Images