Three-dimensional feature determination method, apparatus, device, and storage medium

Abstract

The application relates to a three-dimensional feature determination method, device, equipment and storage medium, and relates to the technical field of automobiles. The method comprises the following steps: a three-dimensional feature determination device acquires image data collected by a target camera at a first time and a first radar heat map collected by a target 3D millimeter wave radar at a time closest to the first time. The three-dimensional feature determination device determines a second radar heat map according to an image space corresponding to the target camera and the first radar heat map. Image features are determined according to the image data, and radar heat map features are determined according to the second radar heat map. Further, the three-dimensional feature determination device determines three-dimensional features corresponding to each first voxel in the image space according to the image features and the radar heat map features, and the three-dimensional features are used to represent object information in a space indicated by the first voxel. Therefore, data loss in the process of determining the three-dimensional features is reduced, and the three-dimensional feature perception accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of automotive technology, and more particularly to the field of sensor perception technology, specifically to a method, apparatus, device, and storage medium for determining three-dimensional features. Background Technology

[0002] Radar can detect the distance, velocity, and azimuth of objects within its field of view. Radar systems detect these object properties by emitting electromagnetic waves and analyzing the electromagnetic waves reflected from objects within the field of view. Millimeter-wave radar uses short waves with wavelengths on the order of millimeters for detection, and it can detect object motion at the millimeter level. Therefore, millimeter-wave radar has become a commonly used sensor in autonomous driving systems.

[0003] However, millimeter-wave radar cannot directly generate point clouds. Instead, it generates a heat map on the azimuth-range plane that expresses reflection intensity and velocity, and then uses post-processing to generate point clouds or the detected target. However, the post-processing process can easily lead to the loss of some information, which in turn affects the accuracy of the data in subsequent applications. Summary of the Invention

[0004] One of the purposes of this application is to provide a method, apparatus, device and storage medium for determining three-dimensional features, so as to reduce data loss in the process of determining three-dimensional features and improve the accuracy of three-dimensional feature perception.

[0005] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0006] According to a first aspect of this application, a method for determining three-dimensional features is provided. A vehicle is equipped with multiple cameras and multiple 3D millimeter-wave radars. The method includes: a three-dimensional feature determining device acquiring image data collected by a target camera at a first moment, and a first radar heatmap collected by the target 3D millimeter-wave radar at the moment closest to the first moment. The target camera is any one of the multiple cameras, and the target 3D millimeter-wave radar is a 3D millimeter-wave radar whose field of view overlaps with that of the target camera. Further, the three-dimensional feature determining device determines a second radar heatmap based on the image space corresponding to the target camera and the first radar heatmap. The size of the second radar heatmap is the same as the size of the width and depth planes of the image space, and the image space includes multiple first voxels. Further, the three-dimensional feature determining device determines image features based on the image data and determines radar heatmap features based on the second radar heatmap. Further, the three-dimensional feature determining device determines a three-dimensional feature corresponding to each first voxel in the image space based on the image features and the radar heatmap features. The three-dimensional features are used to characterize object information within the space indicated by the first voxel.

[0007] Based on the aforementioned technical means, the three-dimensional feature determination method provided in this application, based on the perception data collected by multiple cameras and multiple 3D millimeter-wave radars deployed on the vehicle, firstly aligns the perception data collected by the target camera and the target 3D millimeter-wave radar in time, and further determines the radar heat map on the width and depth plane perpendicular to the imaging plane according to the image space corresponding to the target camera. Thus, after determining the image features and radar heat map features, the three-dimensional features corresponding to each voxel in the image space are fused and determined. Compared with the point cloud data after radar data processing in related technologies, directly determining the three-dimensional features based on radar heat map features can reduce data loss and thus improve perception accuracy.

[0008] In one possible implementation, the aforementioned three-dimensional feature determination device determines a second radar heatmap based on the image space corresponding to the target camera and a first radar heatmap, including: for any radar heatmap point in the first radar heatmap, determining the projection position of the radar heatmap point on the width-depth plane of the image space corresponding to the target camera based on the direction angle and distance of the radar heatmap point; for each region corresponding to a first voxel on the width-depth plane of the image space, determining the radar data of the region corresponding to the first voxel based on the values ​​of multiple radar heatmap points included in the region corresponding to the first voxel; and determining the second radar heatmap based on the radar data of the region corresponding to each first voxel on the width-depth plane of the image space.

[0009] Based on the above technical means, this application realizes the transformation of radar heat map in the coordinate system of 3D millimeter-wave radar to radar heat map on the width-depth plane of camera image space.

[0010] In one possible implementation, the aforementioned three-dimensional feature determination device determines the projection position of the radar thermal map point on the width-depth plane of the image space corresponding to the target camera based on the direction angle and distance of the radar thermal map point. This includes: determining the coordinates of the radar thermal map point in the vehicle coordinate system based on the direction angle and distance of the radar thermal map point and a first transformation matrix, wherein the first transformation matrix is ​​used to characterize the transformation relationship from the vehicle coordinate system to the coordinate system of the target 3D millimeter-wave radar; and determining the projection position of the radar thermal map point on the width-depth plane of the image space corresponding to the target camera based on the coordinates of the radar thermal map point in the vehicle coordinate system, the intrinsic parameter matrix of the target camera, and a second transformation matrix, wherein the second transformation matrix is ​​used to characterize the transformation relationship from the vehicle coordinate system to the coordinate system of the target camera.

[0011] Based on the above technical means, this application realizes the projection position of radar thermal map points in the coordinate system of 3D millimeter-wave radar on the width and depth plane of the camera image space.

[0012] In one possible implementation, the three-dimensional feature determination device determines the radar data of the region corresponding to the first voxel based on the values ​​of multiple radar heatmap points included in the region corresponding to the first voxel, including: determining the average or maximum value of the multiple radar heatmap points as the radar data of the region corresponding to the first voxel.

[0013] In one possible implementation, the above-mentioned three-dimensional feature determination device determines image features based on image data and determines radar heatmap features based on a second radar heatmap, including: inputting image data into a preset image feature extraction network to determine image features; and inputting the second radar heatmap into a preset radar heatmap feature extraction network to determine radar heatmap features.

[0014] In one possible implementation, the vehicle is also equipped with a 4D millimeter-wave radar, and the aforementioned three-dimensional feature determination method further includes: a three-dimensional feature determination device determining a third voxel within the coverage area of ​​the 4D millimeter-wave radar; generating a pseudo-4D radar heatmap based on the three-dimensional features corresponding to the third voxel and a preset pseudo-4D radar heatmap generation network; and adjusting a preset image feature extraction network, a preset radar heatmap feature extraction network, and a preset pseudo-4D radar heatmap generation network when the similarity between the 4D radar heatmap output by the 4D millimeter-wave radar and the pseudo-4D radar heatmap is less than a preset similarity.

[0015] Based on the aforementioned technical means, the three-dimensional feature determination method provided in this application, when a 4D millimeter-wave radar is deployed on a vehicle, determines the third voxel within the coverage area of ​​the 4D millimeter-wave radar. Based on the previously determined three-dimensional features corresponding to the third voxel, a pseudo-4D radar heatmap is generated. This heatmap is then compared with the 4D millimeter-wave radar heatmap generated by the deployed 4D millimeter-wave radar to determine whether the three-dimensional features corresponding to the third voxel meet the accuracy requirements. If the accuracy requirements are not met, the aforementioned multiple neural networks are adjusted to improve the accuracy of the three-dimensional features. This improves the accuracy of subsequent tasks based on three-dimensional features (target recognition, autonomous driving, etc.) while ensuring the accuracy of the three-dimensional features. Furthermore, the three-dimensional features generated based on image data and the 3D radar heatmap can be further fused with the three-dimensional features generated by the 4D millimeter-wave radar to improve the fusion efficiency of multi-sensor data in the vehicle coordinate system.

[0016] In one possible implementation, the size of the pseudo 4D radar heatmap is a preset size corresponding to the preset pseudo 4D radar heatmap generation network. After generating the pseudo 4D radar heatmap, the three-dimensional feature determination method further includes: when the size of the 4D radar heatmap output by the 4D millimeter-wave radar is not a preset size, the three-dimensional feature determination device performs sampling or interpolation operations on the 4D radar heatmap output by the 4D millimeter-wave radar.

[0017] In one possible implementation, the above-described three-dimensional feature determination method further includes: the three-dimensional feature determination device determining the vehicle's perception range based on the perception range of multiple cameras deployed on the vehicle; determining multiple second voxels included within the vehicle's perception range; for any one of the multiple second voxels, determining the three-dimensional feature corresponding to the second voxel based on the three-dimensional feature of the first voxel corresponding to the second voxel; and generating environmental features within the vehicle's perception range based on the three-dimensional feature corresponding to each of the multiple second voxels.

[0018] Based on the above technical means, this application realizes that the three-dimensional features of the first voxel in the image space corresponding to each camera are spatially transformed to obtain the three-dimensional features of the second voxel within the vehicle's perception range, thereby realizing the generation of environmental features within the vehicle's perception range based on the position of the second voxel.

[0019] In one possible implementation, the three-dimensional feature determination device determines the three-dimensional feature corresponding to the second voxel based on the three-dimensional feature of the first voxel corresponding to the second voxel, including: when there is only one first voxel corresponding to the second voxel, determining the three-dimensional feature corresponding to the first voxel as the three-dimensional feature corresponding to the second voxel; when there are at least two first voxels corresponding to the second voxel, determining the three-dimensional feature corresponding to the second voxel based on the three-dimensional feature corresponding to each of the at least two first voxels.

[0020] According to a second aspect of this application, a three-dimensional feature determination device is provided. A vehicle is equipped with multiple cameras and multiple 3D millimeter-wave radars. The device includes an acquisition unit, a determination unit, and a processing unit. The acquisition unit acquires image data collected by a target camera at a first moment and a first radar heatmap collected by the target 3D millimeter-wave radar at the moment closest to the first moment. The target camera is any one of the multiple cameras, and the target 3D millimeter-wave radar is the 3D millimeter-wave radar whose field of view overlaps with that of the target camera. The determination unit determines a second radar heatmap based on the image space corresponding to the target camera and the first radar heatmap. The size of the second radar heatmap is the same as the width and depth plane of the image space, which includes multiple first voxels. The processing unit determines image features based on the image data. The processing unit further determines radar heatmap features based on the second radar heatmap. The determination unit further determines three-dimensional features corresponding to each first voxel in the image space based on the image features and the radar heatmap features. The three-dimensional features characterize object information within the space indicated by the first voxel.

[0021] In one possible implementation, the determining unit is specifically configured to, for any radar heatmap point in the first radar heatmap, determine the projection position of the radar heatmap point on the width-depth plane of the image space corresponding to the target camera based on the direction angle and distance of the radar heatmap point; for each region corresponding to a first voxel on the width-depth plane of the image space, determine the radar data of the region corresponding to the first voxel based on the values ​​of multiple radar heatmap points included in the region corresponding to the first voxel; and determine the second radar heatmap based on the radar data of the region corresponding to each first voxel on the width-depth plane of the image space.

[0022] In one possible implementation, the determining unit is specifically used to determine the coordinates of the radar thermal map points in the vehicle coordinate system based on the direction angle and distance of the radar thermal map points and a first transformation matrix. The first transformation matrix is ​​used to characterize the transformation relationship from the vehicle coordinate system to the coordinate system of the target 3D millimeter-wave radar. Based on the coordinates of the radar thermal map points in the vehicle coordinate system, the intrinsic parameter matrix of the target camera, and a second transformation matrix, the unit determines the projection position of the radar thermal map points on the width-depth plane of the image space corresponding to the target camera. The second transformation matrix is ​​used to characterize the transformation relationship from the vehicle coordinate system to the coordinate system of the target camera.

[0023] In one possible implementation, the determining unit is specifically used to determine the average or maximum value of multiple radar heatmap points as the radar data of the region corresponding to the first voxel.

[0024] In one possible implementation, the processing unit is specifically configured to input image data into a preset image feature extraction network to determine image features; and input the second radar heatmap into a preset radar heatmap feature extraction network to determine radar heatmap features.

[0025] In one possible implementation, the vehicle is also equipped with a 4D millimeter-wave radar. The determining unit is further configured to determine a third voxel within the coverage area of ​​the 4D millimeter-wave radar. The processing unit is further configured to generate a pseudo-4D radar heatmap based on the three-dimensional features corresponding to the third voxel and a preset pseudo-4D radar heatmap generation network. The processing unit is further configured to adjust the preset image feature extraction network, the preset radar heatmap feature extraction network, and the preset pseudo-4D radar heatmap generation network if the similarity between the 4D radar heatmap output by the 4D millimeter-wave radar and the pseudo-4D radar heatmap is less than a preset similarity.

[0026] In one possible implementation, the size of the pseudo 4D radar heatmap is a preset size corresponding to the preset pseudo 4D radar heatmap generation network. After generating the pseudo 4D radar heatmap, the processing unit is further configured to perform sampling or interpolation operations on the 4D radar heatmap output by the 4D millimeter-wave radar if the size of the 4D radar heatmap output by the 4D millimeter-wave radar is not a preset size.

[0027] In one possible implementation, the determining unit is further configured to determine the vehicle's perception range based on the perception range of the multiple cameras deployed on the vehicle. The determining unit is also configured to determine a plurality of second voxels included within the vehicle's perception range. The determining unit is further configured to, for any one of the plurality of second voxels, determine a three-dimensional feature corresponding to the second voxel based on the three-dimensional feature of a first voxel corresponding to the second voxel. The processing unit is further configured to generate environmental features within the vehicle's perception range based on the three-dimensional feature corresponding to each of the plurality of second voxels.

[0028] In one possible implementation, the determining unit is specifically configured to, when the number of first voxels corresponding to the second voxel is one, determine the three-dimensional feature corresponding to the first voxel as the three-dimensional feature corresponding to the second voxel; and when the number of first voxels corresponding to the second voxel is at least two, determine the three-dimensional feature corresponding to the second voxel based on the three-dimensional feature corresponding to each of the at least two first voxels.

[0029] According to a third aspect of this application, a three-dimensional feature determination device is provided, deployed in a vehicle. The three-dimensional feature determination device includes a memory and a processor, coupled together; the memory stores computer program code, which includes computer instructions; when the processor executes the computer instructions, the three-dimensional feature determination device performs the three-dimensional feature determination method provided by the first aspect and any possible implementation thereof.

[0030] According to the fourth aspect provided in this application, a computer-readable storage medium is provided, which stores instructions that, when executed on a three-dimensional feature determination device, cause the three-dimensional feature determination device to perform the three-dimensional feature determination method provided in the first aspect and any possible implementation thereof.

[0031] According to the fifth aspect provided in this application, a vehicle is provided, including the three-dimensional feature determination device provided in the third aspect above.

[0032] According to the sixth aspect provided in this application, a computer program product is provided, the computer program product including computer instructions, which, when executed on a three-dimensional feature determination device, cause the three-dimensional feature determination device to perform the three-dimensional feature determination method provided in the first aspect and any possible implementation thereof.

[0033] Therefore, the above-mentioned technical features of this application have the following beneficial effects:

[0034] (1) In the three-dimensional feature determination method provided in this application, based on the perception data collected by multiple cameras and multiple 3D millimeter-wave radars deployed on the vehicle, the perception data collected by the target camera and the target 3D millimeter-wave radar are first aligned in time. Then, according to the image space corresponding to the target camera, the radar heat map on the width and depth plane perpendicular to the imaging plane is determined. Thus, after determining the image features and radar heat map features, the three-dimensional features corresponding to each voxel in the image space are fused and determined. Compared with the point cloud data after radar data processing in related technologies, the three-dimensional features are directly determined based on the radar heat map features, which can reduce data loss and improve perception accuracy.

[0035] (2) This application also realizes that the three-dimensional features of the first voxel in the image space corresponding to each camera are spatially transformed to obtain the three-dimensional features of the second voxel within the vehicle's perception range, thereby realizing the generation of environmental features within the vehicle's perception range based on the position of the second voxel.

[0036] (3) In the three-dimensional feature determination method provided in this application, when a 4D millimeter-wave radar is deployed on the vehicle, the third voxel within the coverage area of ​​the 4D millimeter-wave radar is determined. Based on the three-dimensional features corresponding to the previously determined third voxel, a pseudo-4D radar heatmap is generated. This heatmap is then compared with the 4D millimeter-wave radar heatmap generated by the deployed 4D millimeter-wave radar to determine whether the three-dimensional features corresponding to the third voxel meet the accuracy requirements. If the accuracy requirements are not met, the aforementioned multiple neural networks are adjusted to improve the accuracy of the three-dimensional features. This improves the accuracy of subsequent tasks based on three-dimensional features (target recognition, autonomous driving, etc.) while ensuring the accuracy of the three-dimensional features. Furthermore, the three-dimensional features generated based on image data and the 3D radar heatmap can be further fused with the three-dimensional features generated by the 4D millimeter-wave radar to improve the fusion efficiency of multi-sensor data in the vehicle coordinate system.

[0037] It should be noted that the technical effects of any of the implementation methods in aspects two through six can be found in the technical effects of the corresponding implementation methods in aspect one, and will not be repeated here.

[0038] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0039] Figure 1This is a schematic diagram illustrating the structure of a three-dimensional feature determination system according to an exemplary embodiment;

[0040] Figure 2 This is a schematic diagram illustrating a sensor deployment according to an exemplary embodiment;

[0041] Figure 3 This is a flowchart illustrating a three-dimensional feature determination method according to an exemplary embodiment;

[0042] Figure 4 This is a schematic diagram illustrating a feature transformation according to an exemplary embodiment;

[0043] Figure 5 This is a flowchart illustrating yet another method for determining three-dimensional features according to an exemplary embodiment;

[0044] Figure 6 This is a flowchart illustrating yet another method for determining three-dimensional features according to an exemplary embodiment;

[0045] Figure 7 This is a flowchart illustrating yet another method for determining three-dimensional features according to an exemplary embodiment;

[0046] Figure 8 This is a block diagram illustrating a three-dimensional feature determining device according to an exemplary embodiment;

[0047] Figure 9 This is a block diagram illustrating a three-dimensional feature determination device according to an exemplary embodiment. Detailed Implementation

[0048] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0049] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0050] Radar can detect the distance, velocity, and azimuth of objects within its field of view. Radar systems detect these object properties by emitting electromagnetic waves and analyzing the electromagnetic waves reflected from objects within the field of view. Millimeter-wave radar uses short waves with wavelengths on the order of millimeters for detection and can detect object motion at the millimeter level. Therefore, millimeter-wave radar has become a commonly used sensor in autonomous driving systems. However, millimeter-wave radar cannot directly generate point clouds; instead, it generates a heatmap on the azimuth-range plane expressing reflection intensity and velocity, and then uses post-processing to generate the point cloud or the detected target. However, this post-processing process can easily lead to the loss of some information, thus affecting the data accuracy in subsequent applications.

[0051] Furthermore, cameras are an essential sensor in autonomous driving systems because they need to perceive a large amount of visual semantic information in the environment. The visual images output by cameras can be seen as projections of environmental objects onto the image plane; therefore, these images do not contain distance information. This complements the information provided by millimeter-wave radar.

[0052] To address the aforementioned problems, this application proposes a three-dimensional feature determination method, apparatus, device, and storage medium. A vehicle is equipped with multiple cameras and multiple 3D millimeter-wave radars. The three-dimensional feature determination apparatus acquires image data collected by the target camera at a first moment, and a first radar heatmap collected by the target 3D millimeter-wave radar at the moment closest to the first moment. The target camera is any one of the multiple cameras, and the target 3D millimeter-wave radar is the 3D millimeter-wave radar whose field of view overlaps with that of the target camera. Further, the three-dimensional feature determination apparatus determines a second radar heatmap based on the image space corresponding to the target camera and the first radar heatmap. The size of the second radar heatmap is the same as the width and depth planes of the image space, which contains multiple first voxels. Further, the three-dimensional feature determination apparatus determines image features based on the image data and radar heatmap features based on the second radar heatmap. Further, the three-dimensional feature determination apparatus determines the three-dimensional features corresponding to each first voxel in the image space based on the image features and radar heatmap features. The three-dimensional features are used to characterize object information within the space indicated by the first voxel.

[0053] In this way, the three-dimensional feature determination method provided in this application first aligns the perception data collected by multiple cameras and multiple 3D millimeter-wave radars deployed on the vehicle in time, based on the perception data collected by the target camera and the target 3D millimeter-wave radar. Then, according to the image space corresponding to the target camera, a radar heat map is determined on the width and depth plane perpendicular to the imaging plane. This allows for the fusion determination of the three-dimensional features corresponding to each voxel in the image space after determining the image features and radar heat map features. Compared with the point cloud data after radar data processing in related technologies, directly determining the three-dimensional features based on radar heat map features can reduce data loss and thus improve perception accuracy.

[0054] Figure 1 This application illustrates a three-dimensional feature determination system. The three-dimensional feature determination method provided in this embodiment can be applied to, for example, […]. Figure 1 The illustrated 3D feature determination system is designed to reduce data loss during the 3D feature determination process and improve the accuracy of 3D feature perception. For example... Figure 1 As shown, the three-dimensional feature determination system 10 includes a three-dimensional feature determination device 11, multiple cameras 12, and multiple 3D millimeter-wave radars 13.

[0055] The three-dimensional feature determination device 11, multiple cameras 12, and multiple 3D millimeter-wave radars 13 are all deployed on the same vehicle. The three-dimensional feature determination device 11 is directly connected to the multiple cameras 12 and the multiple 3D millimeter-wave radars 13, or indirectly connected through intermediate devices (domain controllers, vehicle terminals, etc.). It is used to acquire the perception data (image data) collected by each of the multiple cameras 12, and to acquire the perception data (first radar heat map) collected by each of the multiple 3D millimeter-wave radars 13.

[0056] It should be noted that after multiple cameras 12 and multiple 3D millimeter-wave radars 13 are deployed on the same vehicle, the maintenance personnel of the three-dimensional feature determination system calibrate the external parameters of each sensor in order to convert the data of each sensor to the same coordinate system.

[0057] In some embodiments, the deployment of multiple cameras 12 and multiple 3D millimeter-wave radars 13 on the vehicle is as follows: Figure 2 As shown, the field of view of multiple cameras 12 and the field of view of multiple 3D millimeter-wave radars 13 can cover a 360° range around the vehicle body, and the corresponding fan-shaped area is the field of view of each sensor.

[0058] Multiple cameras 12 can be used to collect image data around the vehicle body.

[0059] Multiple 3D millimeter-wave radars 13 can be used to acquire radar thermal maps around the vehicle body.

[0060] The radar heat map includes a reflection intensity heat map and a velocity heat map. In the reflection intensity heat map, the data at each point is the reflection intensity information at the corresponding location. In the velocity heat map, the data at each point is the velocity information at the corresponding location.

[0061] The three-dimensional feature determination device 11 can be used to acquire image data collected by the target camera among multiple cameras 12 at the first moment, and the first radar thermal map collected by the target 3D millimeter-wave radar among multiple 3D millimeter-wave radars 13 at the moment closest to the first moment.

[0062] Among them, the target camera is any one of the multiple cameras 12, and the target 3D millimeter-wave radar is the 3D millimeter-wave radar among the multiple 3D millimeter-wave radars 13 whose field of view overlaps with the field of view of the target camera.

[0063] In some embodiments, the three-dimensional feature determination device 11 pre-stores a mapping relationship between the target camera and the target 3D millimeter-wave radar, which is used to characterize the overlap between the field of view of the camera and the 3D millimeter-wave radar.

[0064] The three-dimensional feature determination device 11 can also be used to determine a second radar heat map based on the image space corresponding to the target camera and the first radar heat map.

[0065] The second radar heatmap has the same dimensions as the width and depth planes of the image space, which includes multiple first voxels. The width of the image space is w (the number of pixels horizontally), the height is h (the number of pixels vertically), and the depth is n (a preset number corresponding to the target camera). d The image space includes w*h*n d The first element.

[0066] The three-dimensional feature determination device 11 can also be used to determine image features based on image data.

[0067] The three-dimensional feature determination device 11 can also be used to determine radar heat map features based on the second radar heat map.

[0068] The three-dimensional feature determination device 11 can also be used to determine the three-dimensional features corresponding to each first voxel in the image space based on image features and radar heat map features.

[0069] Figure 3 This is a flowchart illustrating a three-dimensional feature determination method according to some exemplary embodiments. In some embodiments, the above-described three-dimensional feature determination method can be applied to, for example... Figure 1The three-dimensional feature determination system 10 shown includes a three-dimensional feature determination device 11. Hereinafter, this application will describe the three-dimensional feature determination method by taking the application of the three-dimensional feature determination method to the three-dimensional feature determination device 11 as an example.

[0070] like Figure 3 As shown, the three-dimensional feature determination method provided in this application includes the following steps S201-S206.

[0071] S201, The three-dimensional feature determination device acquires the image data collected by the target camera at the first moment.

[0072] The target camera can be any one of multiple cameras.

[0073] As one possible implementation, after the camera acquires image data, it sends the acquisition time of the image data and the acquired image data to the three-dimensional feature determination device.

[0074] Accordingly, the three-dimensional feature determination device determines the acquisition time carried by the target camera when sending image data as the first moment, and determines the received image data as the image data acquired by the target camera at the first moment.

[0075] S202, The three-dimensional feature determination device acquires the first radar thermal map collected by the target 3D millimeter-wave radar at the moment closest to the first moment.

[0076] The target 3D millimeter-wave radar is one of multiple 3D millimeter-wave radars whose field of view overlaps with that of the target camera. The radar heatmap includes a reflection intensity heatmap and a velocity heatmap. In the reflection intensity heatmap, the data at each point represents the reflection intensity information at that location. In the velocity heatmap, the data at each point represents the velocity information at that location. The first radar heatmap can be a reflection intensity heatmap and / or a velocity heatmap.

[0077] It should be noted that due to the inherent characteristics of cameras and 3D millimeter-wave radars, their sampling frequencies differ. Generally, 3D millimeter-wave radars have a higher sampling frequency. Therefore, in this embodiment, the sampling time of the camera is used as a reference to match the radar heatmap closest to the camera's sampling time. In some embodiments, when the camera's sampling frequency is higher than the 3D millimeter-wave radar's sampling frequency, the sampling time of the 3D millimeter-wave radar can also be used as a reference to match the image data closest to the 3D millimeter-wave radar's sampling time.

[0078] As one possible implementation, after acquiring the radar heat map, the 3D millimeter-wave radar sends the acquisition time of the radar heat map and the acquired radar heat map to the three-dimensional feature determination device.

[0079] Accordingly, based on the first moment determined in step S201 above, the three-dimensional feature determination device determines the acquisition moment closest to the first moment among the multiple acquisition moments sent by the target 3D millimeter-wave radar, and determines the radar heat map acquired at that acquisition moment as the first radar heat map.

[0080] Understandably, by aligning the image data and radar heatmaps over time, it is possible to ensure that the external environment does not change significantly within the data acquisition time intervals of each sensor, thus guaranteeing the accuracy of the generated 3D features.

[0081] S203, the three-dimensional feature determination device determines the second radar heat map based on the image space corresponding to the target camera and the first radar heat map.

[0082] The second radar heatmap has the same dimensions as the width and depth planes of the image space. The image space includes multiple first voxels, with the width being the number of pixels w in the horizontal direction, the height being the number of pixels h in the vertical direction, and the depth being a preset number n corresponding to the target camera. d The image space includes w*h*n d The first element.

[0083] As one possible implementation, the three-dimensional feature determination device determines the position of each point in the width-depth plane of the image space of the target camera based on the orientation angle and distance of each point in the first radar heat map, and further processes the first radar heat map in the image space based on the size of the width-depth plane of the image space corresponding to the target camera to obtain a second radar heat map.

[0084] It should be noted that the image space corresponding to the target camera can be pre-set in the 3D feature determination device by the maintenance personnel of the 3D feature determination system. Here, the width of the image space is w (the number of pixels horizontally), the height is h (the number of pixels vertically), and the depth is n (the preset number of pixels corresponding to the target camera). d Wherein, the unit distance in depth is the farthest detection distance d defined by the target camera. max and the nearest detection distance d min The unit distance is determined to be (d). max -d min ) / n d Thus, we can obtain w*h*n in the image space corresponding to the target camera. d The first element.

[0085] In some embodiments, the three-dimensional feature determination device determines a second radar heat map based on the image space corresponding to the target camera and the first radar heat map, which may include the following steps S2031-S2033.

[0086] S2031. The three-dimensional feature determination device determines the projection position of any radar heat map point in the first radar heat map on the width and depth plane of the image space corresponding to the target camera, based on the direction angle and distance of the radar heat map point.

[0087] As one possible implementation, the 3D feature determination device determines the coordinates of the radar thermal map points in the vehicle coordinate system based on the direction angle and distance of the radar thermal map points, as well as the first transformation matrix. Further, the 3D feature determination device determines the projection position of the radar thermal map points on the width-depth plane of the image space corresponding to the target camera based on the coordinates of the radar thermal map points in the vehicle coordinate system, the intrinsic parameter matrix of the target camera, and the second transformation matrix.

[0088] It should be noted that the first transformation matrix is ​​used to characterize the transformation relationship from the vehicle coordinate system to the target 3D millimeter-wave radar coordinate system; the second transformation matrix is ​​used to characterize the transformation relationship from the vehicle coordinate system to the target camera coordinate system. The first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the target camera can be preset in the 3D feature determination device by the maintenance personnel of the 3D feature determination system, and this application embodiment does not specifically limit this.

[0089] In some embodiments, the first transformation matrix The second transformation matrix is: The intrinsic parameter matrix of the target camera is a 3x3 matrix K. Where R... radar For the target 3D millimeter-wave radar, t is the 3*3 rotation matrix. radar R is the 3*1 translation matrix corresponding to the target's 3D millimeter-wave radar; cam Let t be the 3x3 rotation matrix corresponding to the target camera. cam This is the 3*1 translation matrix corresponding to the target camera.

[0090] For example, for any point p in the first radar heatmap, let point p be (α, r) in the direction angle-range representation, then the corresponding point coordinates p transformed into the vehicle's whole coordinate system are... ego As shown in the formula below.

[0091]

[0092] Furthermore, the point p is transformed from the vehicle coordinate system to the image space, as shown in the following formula.

[0093]

[0094] Based on the above formula, the projection position of point p on the width-depth plane (WD plane) of the image space is determined as (u,d), where the D-axis is the depth axis, perpendicular to the image plane.

[0095] S2032, The three-dimensional feature determination device determines the radar data of the region corresponding to each first voxel on the width-depth plane of the image space based on the values ​​of multiple radar heat map points included in the region corresponding to the first voxel.

[0096] As one possible implementation, the 3D feature determination device obtains the regions corresponding to multiple first voxels on the width-depth plane of the image space based on the projection of each first voxel. Then, based on the projection positions of the radar heatmap points determined in step S2031, it determines the multiple radar heatmap points included in the region corresponding to each first voxel. Further, the 3D feature determination device determines the radar data for the region corresponding to the first voxel based on the values ​​of the multiple radar heatmap points included in the region corresponding to the first voxel.

[0097] It should be noted that when the value of a radar heatmap point is reflection intensity, the radar data for the region corresponding to the first voxel is reflection intensity; when the value of a radar heatmap point is velocity, the radar data for the region corresponding to the first voxel is velocity; and when the value of a radar heatmap point includes both reflection intensity and velocity, the radar data for the region corresponding to the first voxel includes both reflection intensity and velocity.

[0098] In some embodiments, the three-dimensional feature determining device determines the radar data corresponding to the first voxel based on the values ​​of multiple radar heat map points included in the region corresponding to the first voxel. Alternatively, the three-dimensional feature determining device may determine the radar data of the region corresponding to the first voxel as the average value of the multiple radar heat map points included in the region corresponding to the first voxel.

[0099] To reduce the computational complexity of determining the radar data corresponding to the first voxel, the three-dimensional feature determination device determines the value of the radar heat map point corresponding to the center point of the region corresponding to the first voxel as the radar data of the region corresponding to the first voxel.

[0100] Alternatively, the three-dimensional feature determination device may determine the average value of radar heat map points within a small range near the center point of the region corresponding to the first voxel as the radar data of the region corresponding to the first voxel.

[0101] Alternatively, the three-dimensional feature determination device can determine the average or maximum value of multiple radar heatmap points as the radar data of the region corresponding to the first voxel.

[0102] In some embodiments, there is a situation where the fields of view of multiple 3D millimeter-wave radars overlap with the field of view of a camera. When the first voxel corresponds to the first radar heat map of multiple target 3D millimeter-wave radars, the three-dimensional feature determination device first merges the multiple first radar heat maps, and the radar heat map of the overlapping part takes the maximum value or average value when merging.

[0103] In some embodiments, if the region corresponding to the first voxel does not include radar heatmap points, then the radar data corresponding to the first voxel is determined to be 0, or the mean value of the surrounding first voxels.

[0104] S2033, the three-dimensional feature determination device determines the second radar heat map based on the radar data of the region corresponding to each first voxel on the width-depth plane of the image space.

[0105] The second radar heatmap has a size of w*n. d .

[0106] As one possible implementation, the three-dimensional feature determination device obtains a second radar thermal map with the same size as the broadband depth plane based on the radar data of the region corresponding to each first voxel determined in step S2032 above, and based on the position of each first voxel.

[0107] S204. The three-dimensional feature determination device determines image features based on image data.

[0108] As one possible implementation, the three-dimensional feature determination device inputs the w*h-sized image data obtained in step S201 above into a preset image feature extraction network to obtain image features corresponding to the image data.

[0109] It should be noted that the preset image feature extraction network can be pre-set in the 3D feature determination device by the operators of the 3D feature determination system. For example, the preset image feature extraction network can be a 2D backbone convolutional neural network, such as ResNet, EfficientNet, SwinTransformer, VoVNetV2 and other network structures with a certain depth.

[0110] S205, The three-dimensional feature determination device determines the radar heat map features based on the second radar heat map.

[0111] As one possible implementation, the three-dimensional feature determination device is based on the w*n obtained in step S203 above. d The second radar heatmap of the specified size is input into a preset radar heatmap feature extraction network to obtain radar heatmap features corresponding to the second radar heatmap.

[0112] It should be noted that the preset radar heatmap feature extraction network can be pre-set in the 3D feature determination device by the maintenance personnel of the 3D feature determination system. For example, the preset radar heatmap feature extraction network can be a 2D backbone convolutional neural network, such as ResNet, EfficientNet, SwinTransformer, VoVNetV2 and other network structures with a certain depth.

[0113] S206. The three-dimensional feature determination device determines the three-dimensional features corresponding to each first voxel in the image space based on image features and radar heat map features.

[0114] Among them, the three-dimensional features are used to characterize the object information in the space indicated by the first voxel. The object information can be the shape, position, velocity, height, orientation, etc. of the detected target.

[0115] As one possible implementation, the three-dimensional feature determination device, based on the image features determined in step S204 and the radar heat map features determined in step S205, directly adds the image features and radar features corresponding to each first voxel in space or splices them in the channel dimension to achieve the fusion of image features and radar heat map features, thereby obtaining the three-dimensional features corresponding to each first voxel.

[0116] In some embodiments, the three-dimensional feature determination device predicts the distribution of image features in the depth direction, and then, based on the distribution of image features in the depth direction corresponding to each first voxel, directly adds or stitches the corresponding radar heat map features in the channel dimension to achieve the fusion of image features and radar heat map features, thereby obtaining the three-dimensional features corresponding to each first voxel.

[0117] In some embodiments, the three-dimensional feature determination device inputs the image features and radar heat map features corresponding to each first voxel into a preset cross-attention algorithm, and fuses the image features and radar heat map features to obtain the three-dimensional features corresponding to each first voxel.

[0118] It should be noted that the preset cross-attention algorithm can be pre-set in the 3D feature determination device by the operators of the 3D feature determination system, and this application embodiment does not specifically limit it.

[0119] In some embodiments, after determining the three-dimensional features corresponding to each first voxel in the image space of the target camera, the three-dimensional feature determination device can also be used to perform spatial transformation to obtain the three-dimensional features corresponding to each first voxel in the vehicle coordinate system.

[0120] For example, such as Figure 4As shown, the first part consists of image features in the width-height plane (WH) determined in step S204 and radar heatmap features in the width-depth plane (WD) determined in step S205. After feature fusion, the second part obtains the three-dimensional features of each voxel in the target camera's image space. After spatial transformation, the third part obtains the three-dimensional features of each voxel in the vehicle coordinate system.

[0121] Understandably, in the three-dimensional feature determination method provided in the above embodiments of this application, based on the perception data collected by multiple cameras and multiple 3D millimeter-wave radars deployed on the vehicle, the perception data collected by the target camera and the target 3D millimeter-wave radar are first aligned in time. Then, according to the image space corresponding to the target camera, a radar heat map on the width and depth plane perpendicular to the imaging plane is determined. This achieves the fusion determination of the three-dimensional features corresponding to each voxel in the image space after determining the image features and the radar heat map features. Compared with the point cloud data after radar data processing in related technologies, directly determining the three-dimensional features based on the radar heat map features can reduce data loss and thus improve the perception accuracy.

[0122] In one design, to further determine and present scene features in the environment surrounding the vehicle, the three-dimensional feature determination method provided in this application embodiment, such as... Figure 5 As shown, it also includes S301-S304.

[0123] S301, The three-dimensional feature determination device determines the vehicle's perception range based on the perception range of multiple cameras deployed on the vehicle.

[0124] As one possible implementation, the 3D feature determination device acquires the perception range of each of multiple cameras deployed on the vehicle based on the vehicle coordinate system, and determines the vehicle's perception range based on the perception ranges of the multiple cameras in the vehicle coordinate system. For example, the perception range in the x-axis direction of the vehicle coordinate system is -x. PECP1 ~x PECP2 The sensing range in the y-axis direction is -y. PECP1 ~y PECP2 The sensing range in the z-axis direction is -z. PECP1 ~z PECP2 .

[0125] S302, The three-dimensional feature determination device determines multiple second voxels included within the vehicle's perception range.

[0126] As one possible implementation, the 3D feature determination device is based on the resolution Δx of multiple cameras along the x, y, and z axes of the vehicle coordinate system. PECP Δy PECP ΔzPECP And the vehicle's perception range determined in step S301 above, determining the area included within the vehicle's perception range. A second element.

[0127] It should be noted that the vehicle's sensing range on the same axis may or may not be symmetrical about the origin, and this application does not specifically limit this.

[0128] S303, the three-dimensional feature determination device determines the three-dimensional feature corresponding to any one of the multiple second voxels based on the three-dimensional feature of the first voxel corresponding to the second voxel.

[0129] As one possible implementation, the three-dimensional feature determination device determines the first voxel corresponding to any one of the multiple second voxels, and then performs a spatial transformation on the three-dimensional feature corresponding to the first voxel to obtain the three-dimensional feature corresponding to the second voxel.

[0130] In some embodiments, the three-dimensional feature determining device determines the three-dimensional feature corresponding to the second voxel based on the three-dimensional feature of the first voxel corresponding to the second voxel, including: when the number of the first voxels corresponding to the second voxel is one, the three-dimensional feature determining device performs a spatial transformation on the three-dimensional feature corresponding to the first voxel and determines it as the three-dimensional feature corresponding to the second voxel.

[0131] When the number of first voxels corresponding to the second voxel is at least two, the three-dimensional feature determination device determines the three-dimensional feature corresponding to the second voxel based on the three-dimensional feature corresponding to each of the at least two first voxels.

[0132] The three-dimensional feature determination device merges the three-dimensional features corresponding to each of the at least two first voxels, performs spatial transformation, and determines them as the three-dimensional features corresponding to the second voxel.

[0133] S304. The three-dimensional feature determination device generates environmental features within the vehicle's perception range based on the three-dimensional features corresponding to each of the multiple second voxels.

[0134] As one possible implementation, the three-dimensional feature determination device generates environmental features within the vehicle's perception range based on the three-dimensional features corresponding to each of the multiple second voxels and the position of each second voxel within the vehicle's perception range.

[0135] In some embodiments, after generating environmental features within the vehicle's perception range, these features can be used to input different perception task networks such as target detection and occupancy grid prediction. These networks supervise the environmental features based on the corresponding annotation information.

[0136] It is understood that in the three-dimensional feature determination method provided in the above embodiments of this application, the three-dimensional features of the first voxel in the image space corresponding to each camera are spatially transformed to obtain the three-dimensional features corresponding to the second voxel within the vehicle's perception range, thereby realizing the generation of environmental features within the vehicle's perception range based on the position of the second voxel.

[0137] In one design, to ensure the accuracy of the generated 3D features, a 4D millimeter-wave radar is also deployed on the vehicle. The 3D feature determination method provided in this application also implements supervision of the generated 3D features, such as... Figure 6 As shown, it includes S401-S403.

[0138] S401, the three-dimensional feature determination device determines the third voxel within the coverage area of ​​the 4D millimeter-wave radar.

[0139] As one possible implementation, the three-dimensional feature determination device determines the coverage area of ​​the 4D millimeter-wave radar, and further determines the voxels within the 4D millimeter-wave radar coverage area within the vehicle's perception range as third voxels.

[0140] S402, the three-dimensional feature determination device generates a pseudo-4D radar heat map based on the three-dimensional features corresponding to the third voxel and a preset pseudo-4D radar heat map generation network.

[0141] As one possible implementation, the three-dimensional feature determination device inputs the third voxel determined in step S401 above, and the three-dimensional features corresponding to each third voxel, into a preset pseudo-4D radar heat map generation network, and outputs a pseudo-4D radar heat map.

[0142] It should be noted that the preset pseudo 4D millimeter-wave radar heat map generation network can be pre-set in the 3D feature determination device by the operation and maintenance personnel of the 3D feature determination system. For example, it can be a 3D convolution, 3D sparse convolution or 3D transformer network. This application embodiment does not specifically limit it.

[0143] In some embodiments, the preset pseudo 4D millimeter-wave radar heat map generation network can be determined by the system's maintenance personnel based on three-dimensional features. The three-dimensional features identified by the 4D millimeter-wave radar are used as network training features, and the 4D radar heat map output by the 4D millimeter-wave radar is used as the network training label to perform supervised training on the preset neural network.

[0144] In some embodiments, the size of the pseudo-4D radar heatmap is a preset size corresponding to a preset pseudo-4D radar heatmap generation network.

[0145] When the size of the 4D radar heat map output by the 4D millimeter-wave radar is not the preset size, the three-dimensional feature determination device performs sampling or interpolation operations on the 4D radar heat map output by the 4D millimeter-wave radar.

[0146] S403. When the similarity between the 4D radar heatmap output by the 4D millimeter-wave radar and the pseudo-4D radar heatmap is less than the preset similarity, the three-dimensional feature determination device adjusts the preset image feature extraction network, the preset radar heatmap feature extraction network, and the preset pseudo-4D radar heatmap generation network.

[0147] In some embodiments, by supervising the pseudo 4D radar heatmap using a 4D radar heatmap, the three-dimensional features generated based on image data and 3D radar heatmap can be made closer to the three-dimensional features generated by 4D millimeter-wave radar. This improves the fusion efficiency of multi-sensor data in the vehicle coordinate system when the three-dimensional features generated based on image data and 3D radar heatmap are fused with the three-dimensional features generated by 4D millimeter-wave radar.

[0148] It is understood that in the three-dimensional feature determination method provided in the above embodiments of this application, when a 4D millimeter-wave radar is deployed on the vehicle, the generated three-dimensional features can be supervised and detected by the 4D radar heat map output by the 4D millimeter-wave radar to ensure the accuracy of the three-dimensional features.

[0149] In one design, in conjunction with the above embodiments of this application, such as... Figure 7 As shown, a flowchart of a three-dimensional feature determination method is proposed, including steps S501-S507.

[0150] S501. Calibration and time synchronization of internal and external parameters of various sensors (cameras, 3D millimeter-wave radar) on the vehicle.

[0151] S502, the three-dimensional feature determination device acquires the radar heat map of the 3D millimeter-wave radar in each camera.

[0152] S503, the three-dimensional feature determination device generates image features and radar heat map features, and fuses them to generate three-dimensional features.

[0153] S504, the three-dimensional feature determination device performs supervised detection of three-dimensional features based on 4D millimeter-wave radar.

[0154] S505, the three-dimensional feature determination device fuses the three-dimensional features corresponding to each camera to obtain the environmental features within the vehicle's perception range.

[0155] S506, a three-dimensional feature determination device constructs a perception task network.

[0156] Optionally, the three-dimensional feature determination device is constructed with the multi-task loss function corresponding to each feature extraction network model.

[0157] S507, a three-dimensional feature determination device is used to train and upgrade the network model, and to obtain the optimal parameter weights in the network model.

[0158] Optionally, the 3D feature determination device constructs loss functions based on different perception tasks. For example, a cross-entropy loss function is used for target and grid occupancy classification, and an L1 loss function is used for target size regression. Furthermore, the weights of each loss function are set to ensure balanced training across tasks. An initial weight can be set empirically, and then fine-tuned based on the accuracy metrics obtained from training each task. Further, data augmentation methods are used, including random flipping, rotation, scaling, and cropping of images, and overall flipping, rotation, scaling, and cropping of millimeter-wave radar data and annotations. Finally, the network model is trained, and network parameters are adjusted to obtain optimal parameter weights.

[0159] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the three-dimensional feature determination device or apparatus includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0160] This application embodiment can, according to the above method, exemplarily divide a 3D feature determination device or 3D feature determination equipment into functional modules. For example, the 3D feature determination device or 3D feature determination equipment may include functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0161] Figure 8 This is a schematic diagram of a three-dimensional feature determination device provided in an embodiment of this application. This three-dimensional feature determination device is used to perform the above-described three-dimensional feature determination method. Figure 8 As shown, the three-dimensional feature determination device 60 includes an acquisition unit 601, a determination unit 602, and a processing unit 603.

[0162] The acquisition unit 601 is used to acquire image data collected by the target camera at the first moment and the first radar heat map collected by the target 3D millimeter-wave radar at the moment closest to the first moment. The target camera is any one of multiple cameras, and the target 3D millimeter-wave radar is a 3D millimeter-wave radar whose field of view overlaps with that of the target camera among the multiple 3D millimeter-wave radars.

[0163] The determining unit 602 is used to determine a second radar heat map based on the image space corresponding to the target camera and the first radar heat map. The size of the second radar heat map is the same as the size of the width and depth plane of the image space. The image space includes multiple first voxels.

[0164] The processing unit 603 is used to determine image features based on image data.

[0165] The processing unit 603 is also used to determine radar heat map features based on the second radar heat map.

[0166] The determining unit 602 is also used to determine the three-dimensional features corresponding to each first voxel in the image space based on the image features and the radar heat map features. The three-dimensional features are used to characterize the object information in the space indicated by the first voxel.

[0167] Optionally, the determining unit 602 is specifically used to determine, for any radar heatmap point in the first radar heatmap, the projection position of the radar heatmap point on the width-depth plane of the image space corresponding to the target camera based on the direction angle and distance of the radar heatmap point; for each region corresponding to a first voxel on the width-depth plane of the image space, the radar data of the region corresponding to the first voxel is determined based on the values ​​of multiple radar heatmap points included in the region corresponding to the first voxel; and the second radar heatmap is determined based on the radar data of the region corresponding to each first voxel on the width-depth plane of the image space.

[0168] Optionally, the determining unit 602 is specifically used to determine the coordinates of the radar thermal map points in the vehicle coordinate system based on the direction angle and distance of the radar thermal map points and the first transformation matrix. The first transformation matrix is ​​used to characterize the transformation relationship from the vehicle coordinate system to the coordinate system of the target 3D millimeter-wave radar. Based on the coordinates of the radar thermal map points in the vehicle coordinate system, the intrinsic parameter matrix of the target camera, and the second transformation matrix, the projected position of the radar thermal map points on the width and depth plane of the image space corresponding to the target camera is determined. The second transformation matrix is ​​used to characterize the transformation relationship from the vehicle coordinate system to the coordinate system of the target camera.

[0169] Optionally, the determining unit 602 is specifically used to determine the average or maximum value of multiple radar heat map points as the radar data of the region corresponding to the first voxel.

[0170] Optionally, the processing unit 603 is specifically used to input image data into a preset image feature extraction network to determine image features; and to input the second radar heat map into a preset radar heat map feature extraction network to determine radar heat map features.

[0171] Optionally, the vehicle is also equipped with 4D millimeter-wave radar.

[0172] The determination unit 602 is also used to determine a third voxel within the coverage area of ​​the 4D millimeter-wave radar.

[0173] The processing unit 603 is also used to generate a pseudo-4D radar heatmap based on the three-dimensional features corresponding to the third voxel and a preset pseudo-4D radar heatmap generation network.

[0174] The processing unit 603 is also used to adjust the preset image feature extraction network, the preset radar heat map feature extraction network, and the preset pseudo-4D radar heat map generation network when the similarity between the 4D radar heat map output by the 4D millimeter-wave radar and the pseudo-4D radar heat map is less than the preset similarity.

[0175] Optionally, the size of the pseudo 4D radar heatmap is a preset size corresponding to the preset pseudo 4D radar heatmap generation network. After generating the pseudo 4D radar heatmap, the processing unit 603 is also used to perform sampling or interpolation operations on the 4D radar heatmap output by the 4D millimeter-wave radar if the size of the 4D radar heatmap output by the 4D millimeter-wave radar is not a preset size.

[0176] Optionally, the determining unit 602 is also configured to determine the perception range of the vehicle based on the perception range of the multiple cameras deployed on the vehicle.

[0177] The determining unit 602 is also used to determine multiple second voxels included within the vehicle's perception range.

[0178] The determining unit 602 is also used to determine the three-dimensional features corresponding to the second voxel based on the three-dimensional features of the first voxel corresponding to the second voxel for any one of the multiple second voxels.

[0179] The processing unit 603 is also configured to generate environmental features within the vehicle's perception range based on the three-dimensional features corresponding to each of the multiple second voxels.

[0180] Optionally, the determining unit 602 is specifically used to determine the three-dimensional feature corresponding to the first voxel as the three-dimensional feature corresponding to the second voxel when there is only one first voxel corresponding to the second voxel; and to determine the three-dimensional feature corresponding to the second voxel based on the three-dimensional feature corresponding to each of the at least two first voxels when there are at least two first voxels corresponding to the second voxel.

[0181] Figure 9 This is a block diagram illustrating a three-dimensional feature determination device according to an exemplary embodiment. For example... Figure 9 As shown, the three-dimensional feature determination device 70 includes, but is not limited to, a processor 701 and a memory 702.

[0182] The memory 702 described above is used to store the executable instructions of the processor 701. It is understood that the processor 701 is configured to execute instructions to implement the three-dimensional feature determination method in the above embodiments.

[0183] It should be noted that those skilled in the art will understand that Figure 9 The structure of the 3D feature determination device shown does not constitute a limitation on the 3D feature determination device. The 3D feature determination device may include, but is not limited to, a device that can be compared ... Figure 9 This may indicate more or fewer components, or combinations of certain components, or different component arrangements.

[0184] The processor 701 is the control center of the 3D feature determination device. It connects various parts of the device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 702, and by calling data stored in the memory 702, thereby providing overall monitoring of the 3D feature determination device. The processor 701 may include one or more processing units. Optionally, the processor 701 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor 701.

[0185] The memory 702 can be used to store software programs and various data. The memory 702 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required by at least one functional module (such as a determination unit, processing unit, etc.), etc. Furthermore, the memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0186] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 702 including instructions, which can be executed by a processor 701 of a three-dimensional feature determination device 70 to implement the three-dimensional feature determination method in the above embodiments.

[0187] In actual implementation, Figure 8 The functions of the acquisition unit 601, the determination unit 602, and the processing unit 603 can all be provided by... Figure 9 The processor 701 calls the computer program stored in the memory 702 to implement the process. The specific execution process can be found in the description of the three-dimensional feature determination method in the previous embodiment, and will not be repeated here.

[0188] Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0189] In an exemplary embodiment, this application also provides a vehicle including the above-described three-dimensional feature determination device.

[0190] In an exemplary embodiment, this application also provides a computer program product including one or more instructions, which can be executed by the processor 701 of the three-dimensional feature determination device to complete the three-dimensional feature determination method in the above embodiments.

[0191] It should be noted that when one or more instructions in the computer-readable storage medium or computer program product are executed by the processor of the three-dimensional feature determination device, they implement the various processes of the above-described three-dimensional feature determination method embodiments and achieve the same technical effect as the above-described three-dimensional feature determination method. To avoid repetition, these will not be elaborated here.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0193] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0194] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the classified units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0195] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0196] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, essentially, or the part that contributes to the prior art, or a complete or partial classification of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0197] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining three-dimensional features, characterized in that, The vehicle is equipped with multiple cameras and multiple 3D millimeter-wave radars, and the method includes: The image data collected by the target camera at the first moment and the first radar heat map collected by the target 3D millimeter-wave radar at the moment closest to the first moment are obtained. The target camera is any one of the multiple cameras, and the target 3D millimeter-wave radar is the 3D millimeter-wave radar whose field of view overlaps with that of the target camera among the multiple 3D millimeter-wave radars. Based on the image space corresponding to the target camera and the first radar heat map, a second radar heat map is determined. The size of the second radar heat map is the same as the size of the width and depth plane of the image space. The image space includes multiple first voxels. Image features are determined based on the image data, and radar heatmap features are determined based on the second radar heatmap; Based on the image features and the radar heatmap features, the three-dimensional features corresponding to each first voxel in the image space are determined, and the three-dimensional features are used to characterize the object information in the space indicated by the first voxel.

2. The three-dimensional feature determination method according to claim 1, characterized in that, The step of determining the second radar heatmap based on the image space corresponding to the target camera and the first radar heatmap includes: For any radar heat map point in the first radar heat map, the projection position of the radar heat map point on the width-depth plane of the image space corresponding to the target camera is determined based on the direction angle and distance of the radar heat map point. For each region corresponding to a first voxel on the width-depth plane of the image space, the radar data of the region corresponding to the first voxel is determined based on the values ​​of multiple radar heatmap points included in the region corresponding to the first voxel. The second radar heatmap is determined based on the radar data of the region corresponding to each first voxel on the width-depth plane of the image space.

3. The three-dimensional feature determination method according to claim 2, characterized in that, Determining the projection position of the radar thermal map point on the width-depth plane of the image space corresponding to the target camera based on the direction angle and distance of the radar thermal map point includes: Based on the azimuth angle and distance of the radar heat map point, and the first transformation matrix, the coordinates of the radar heat map point in the vehicle coordinate system are determined. The first transformation matrix is ​​used to characterize the transformation relationship from the vehicle coordinate system to the coordinate system of the target 3D millimeter-wave radar. Based on the coordinates of the radar thermal map points in the vehicle coordinate system, the intrinsic parameter matrix of the target camera, and the second transformation matrix, the projection position of the radar thermal map points on the width-depth plane of the image space corresponding to the target camera is determined. The second transformation matrix is ​​used to characterize the transformation relationship from the vehicle coordinate system to the coordinate system of the target camera.

4. The three-dimensional feature determination method according to claim 2, characterized in that, The step of determining the radar data of the region corresponding to the first voxel based on the values ​​of multiple radar heatmap points included in the region corresponding to the first voxel includes: The mean or maximum value of the multiple radar heatmap points is determined as the radar data of the region corresponding to the first voxel.

5. The three-dimensional feature determination method according to any one of claims 1-4, characterized in that, The step of determining image features based on image data and determining radar heatmap features based on the second radar heatmap includes: The image data is input into a preset image feature extraction network to determine the image features; The second radar heatmap is input into a preset radar heatmap feature extraction network to determine the radar heatmap features.

6. The three-dimensional feature determination method according to claim 5, characterized in that, The vehicle is also equipped with a 4D millimeter-wave radar, and the method further includes: Determine the third voxel within the coverage area of ​​the 4D millimeter-wave radar; Based on the three-dimensional features corresponding to the third voxel and the preset pseudo-4D radar heat map generation network, a pseudo-4D radar heat map is generated. If the similarity between the 4D radar heatmap output by the 4D millimeter-wave radar and the pseudo-4D radar heatmap is less than a preset similarity, the preset image feature extraction network, the preset radar heatmap feature extraction network, and the preset pseudo-4D radar heatmap generation network are adjusted.

7. The three-dimensional feature determination method according to claim 6, characterized in that, The size of the pseudo-4D radar heatmap is the preset size corresponding to the preset pseudo-4D radar heatmap generation network. After generating the pseudo-4D radar heatmap, the method further includes: If the size of the 4D radar heat map output by the 4D millimeter-wave radar is not the preset size, a sampling operation or an interpolation operation is performed on the 4D radar heat map output by the 4D millimeter-wave radar.

8. The three-dimensional feature determination method according to any one of claims 1-4, characterized in that, The method further includes: The perception range of the vehicle is determined based on the perception range of the multiple cameras deployed on the vehicle. Determine the multiple second voxels included within the perception range of the vehicle; For any one of the plurality of second voxels, the three-dimensional features corresponding to the second voxel are determined based on the three-dimensional features of the first voxel corresponding to the second voxel. Based on the three-dimensional features corresponding to each of the plurality of second voxels, environmental features within the vehicle's perception range are generated.

9. The three-dimensional feature determination method according to claim 8, characterized in that, The step of determining the three-dimensional features corresponding to the second voxel based on the three-dimensional features of the first voxel corresponding to the second voxel includes: When there is only one first voxel corresponding to the second voxel, the three-dimensional feature corresponding to the first voxel is determined as the three-dimensional feature corresponding to the second voxel. When there are at least two first voxels corresponding to the second voxel, the three-dimensional features corresponding to the second voxel are determined based on the three-dimensional features corresponding to each of the at least two first voxels.

10. A three-dimensional feature determination device, characterized in that, The vehicle is equipped with multiple cameras and multiple 3D millimeter-wave radars. The three-dimensional feature determination device includes an acquisition unit, a determination unit, and a processing unit. The acquisition unit is used to acquire image data collected by the target camera at the first moment and the first radar heat map collected by the target 3D millimeter-wave radar at the moment closest to the first moment. The target camera is any one of the plurality of cameras, and the target 3D millimeter-wave radar is the 3D millimeter-wave radar whose field of view overlaps with that of the target camera among the plurality of 3D millimeter-wave radars. The determining unit is used to determine a second radar heat map based on the image space corresponding to the target camera and the first radar heat map. The size of the second radar heat map is the same as the size of the width and depth plane of the image space. The image space includes a plurality of first voxels. The processing unit is used to determine image features based on the image data; The processing unit is further configured to determine radar heat map features based on the second radar heat map; The determining unit is further configured to determine the three-dimensional features corresponding to each first voxel in the image space based on the image features and the radar heat map features, wherein the three-dimensional features are used to characterize the object information in the space indicated by the first voxel.

11. A three-dimensional feature determination device, characterized in that, Deployed in vehicles, including memory and processor; The memory and the processor are coupled; The memory is used to store computer program code, which includes computer instructions; When the processor executes the computer instructions, the three-dimensional feature determination device performs the three-dimensional feature determination method as described in any one of claims 1-9.

12. A computer-readable storage medium storing instructions, characterized in that, When the instructions are executed on the three-dimensional feature determination device, the three-dimensional feature determination device performs the three-dimensional feature determination method as described in any one of claims 1-9.

13. A vehicle, characterized in that, Includes the three-dimensional feature determination device as described in claim 11.

Images