Radar Data Processing Method and Device

Abstract

Embodiments of the present invention disclose a radar data processing method and apparatus. Among them, the method includes: obtaining radar data; corresponding radar points in the radar data to pixel points in an image, where each pixel point includes: 0, 1, or multiple radar points; calculating multiple attributes of each pixel point based on the radar points included in each pixel point to generate multiple attribute maps; respectively performing a continuous operation on the multiple attribute maps; and generating an image result of the radar data based on the multiple continuous attribute maps. The image data output in this embodiment can improve the stability and certainty of the model when used in a deep learning model.

Description

Technical Field

[0001] The present invention relates to the processing of radar data, and in particular, to a method and apparatus for processing radar data. Background Art

[0002] In vehicles, it has become increasingly common to use radars, such as millimeter-wave radars, to detect objects such as people and vehicles existing around the vehicle. Moreover, with the development of deep learning technology, more and more solutions choose to process radar data based on deep learning solutions to achieve functions such as target detection and tracking. In current deep learning models, the target detection and tracking models based on image data are relatively mature. Therefore, visualizing radar data and performing target detection and tracking in the relatively mature field of image algorithm models is a mainstream method. Summary of the Invention

[0003] Embodiments of the present invention provide a method and apparatus for processing radar data, which are used to output image data and can improve the certainty and stability of the model when the image data is used in a deep learning model.

[0004] A method for processing radar data according to an embodiment of the present invention includes: obtaining radar data; corresponding radar points in the radar data to pixel points in an image, where each pixel point includes: 0, 1, or multiple radar points; calculating multiple attributes of each pixel point based on the radar points included in each pixel point to generate multiple attribute maps; respectively performing a continuous operation on the multiple attribute maps; and generating an image result of the radar data based on the continuously processed multiple attribute maps.

[0005] Wherein, the continuous operation is performed based on a Gaussian kernel or a radial basis function.

[0006] Wherein, the variance σ of the Gaussian kernel (h,w) is variable, and the variance of the Gaussian kernel corresponding to each pixel point is the mean of the distances between the pixel point and its k nearest non-empty pixel points, where a non-empty pixel point is a pixel point that includes at least 1 radar point.

[0007] Wherein, the multiple attribute maps include: a radar density map and a radar reflection intensity map; generating the image result of the radar data based on the continuously processed multiple attribute maps includes: weighting the radar reflection intensity map by using the radar density map, and processing the weighted result into data of a first image channel.

[0008] Among them, the multiple attribute maps further include: a radial relative velocity map and a height map; generating the visualized result of the radar data based on the continuously processed multiple attribute maps further includes: processing the radial relative velocity map and the height map into data of a second and a third image channel respectively; performing normalization processing on the data of the first to third image channels to generate the visualized result of the radar data.

[0009] Among them, the multiple attribute maps include: a radar density map, a radar reflection intensity map, a radial relative velocity map, and a height map.

[0010] Among them, the density of each pixel is the number of radar points included in the pixel; the reflection intensity, radial relative velocity, and height of each pixel are the mean value, maximum value, or weighted average of the reflection intensity, radial relative velocity, and height of the radar points included in the pixel respectively.

[0011] A radar data processing device according to an embodiment of the present invention includes: an acquisition module for acquiring radar data; a conversion module for corresponding the radar points in the radar data to pixel points in an image, where each pixel point includes: 0, 1, or multiple radar points; a calculation module for calculating multiple attributes of each pixel point based on the radar points included in each pixel point to generate multiple attribute maps; a continuous processing module for performing continuous processing operations on the multiple attribute maps respectively; and a generation module for generating the visualized result of the radar data based on the continuously processed multiple attribute maps.

[0012] A computer device according to an embodiment of the present invention includes: a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the steps of the method according to the embodiment of the present invention.

[0013] A computer-readable storage medium according to an embodiment of the present invention has a computer program stored thereon, and when the computer program / instructions are executed by a processor, the steps of the method according to the embodiment of the present invention are implemented.

[0014] A computer program product according to an embodiment of the present invention includes computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the method according to the embodiment of the present invention are implemented.

[0015] Advantageous effects of the embodiments of the present invention:

[0016] In this embodiment, image data that can be used in a deep learning model is generated based on radar data, and during the process of generating the image data, by performing continuous processing operations on the attribute maps to reduce 0 values, the generated image data can improve the certainty and stability of the model when used in the deep learning model. Description of the Drawings

[0017] Other details and advantages of the present invention will become apparent from the following detailed description. It should be understood that the following drawings are merely illustrative and thus should not be considered as limiting the present invention. The following will be described in detail with reference to the drawings, where:

[0018] Figure 1 is a schematic flowchart of an embodiment of the radar data processing method of the present invention;

[0019] Figure 2 is a schematic diagram of the installation of the radar and the data coordinate system;

[0020] Figure 3 is a schematic diagram of a radar image;

[0021] Figure 4 is a schematic structural diagram of an embodiment of the radar data processing apparatus of the present invention. Detailed Embodiments

[0022] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not used to limit the present invention.

[0023] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. Moreover, the terms "first", "second", etc. are applicable to distinguish similar objects and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such terms may be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0024] As Figure 1 shown, it is a schematic flowchart of an embodiment of the radar data processing method of the present invention. This method flow can be used in radar products, such as millimeter-wave radars, such as 4D (4-dimensional) millimeter-wave radars; or, this method flow can also be used in other electronic devices. This method flow includes the following steps:

[0025] Step S10: Obtain radar data.

[0026] Among them, the radar data is collected by the radar. As Figure 2As shown in the figure, the radar can be installed at the front end of the vehicle, for example, to detect the situation in front of the vehicle. Specifically, taking a millimeter-wave radar as an example, during operation, the millimeter-wave radar emits millimeter waves forward and receives the electromagnetic waves reflected by targets (such as other vehicles, pedestrians, etc.) to generate radar data. This radar data is, for example, a point cloud data set in a rectangular coordinate system with the center of the radar surface as the origin. Refer to Figure 2 , the x direction is the forward direction of the radar, the y direction is the direction perpendicular to the x direction on the horizontal plane, and the z direction is the direction perpendicular to the xy plane.

[0027] Among them, each point (radar point) in the radar data (such as point cloud data) has multiple attributes. For example, the longitudinal distance R in the x, y, and z directions x , the lateral distance R y , the height R z , the radial relative velocity V relRad , and the radar reflection intensity RCS, etc.

[0028] Step S12: Correlate the radar points in the radar data to the pixel points in the image, where each pixel point includes: 0, 1, or multiple radar points.

[0029] In this step, the size of the image can be determined first. For example, the number of pixel points in the length and width of the image can be represented by h and w respectively. Among them, the size of the image is related to the effective detection distance of the radar. To improve the data processing performance, the points outside the effective detection distance and at the boundary can be filtered out.

[0030] Next, each radar point is divided into each pixel point in the image. For example, each point P in the radar data can be traversed i , and discretized in the x and y directions for each point, and its corresponding pixel point X is calculated (h,w) , and this point is added to the point set S included in this pixel point (h,ω) .

[0031] Based on the characteristics of the radar data, after correlating the radar points to the pixel points in the image, some pixel points may include one or more radar points, and some pixel points may not include radar points.

[0032] Step S14: Calculate multiple attributes of each pixel point based on the radar points included in each pixel point to generate multiple attribute maps.

[0033] Among them, the attribute of each pixel point is determined by the attributes of the radar points included in this pixel point. For the pixel points that do not include radar points, that is, the number of radar points is 0, its attribute value is 0. The attributes of each pixel point together constitute the attribute map.

[0034] In this embodiment, the properties related to each pixel point may include, for example: four parameters, namely (radar) density, reflection intensity, radial relative velocity, and height. Correspondingly, the property maps generated in step S14 may include, for example: a density map, a reflection intensity map, a radial relative velocity map, and a height map.

[0035] The following is an example to illustrate the calculation methods of the above parameters.

[0036] For the density of each pixel point, it can be set as the number of radar points included in this pixel point.

[0037] For the reflection intensity, radial relative velocity, and height of each pixel point, they can be respectively the mean values of the reflection intensity, radial relative velocity, and height of the radar points included in this pixel point.

[0038] For example, let the pixel point be X (h,w) and its point set be S (h,ω) which contains m radar points, and the radial relative velocity of any one of them is v i , then the radial relative velocity φ(X (h,w) ) of the pixel point X is defined as the mean value of the radial relative velocities of the m points: (h,ω) The same applies to other properties and will not be elaborated here. In addition to using the mean value, the maximum value or the weighted average value can also be selected as the property value of the pixel point.

[0039] In this step S14, the properties of the pixel points are determined based on the point set S

[0040] instead of a single point, and the properties of the pixel points include four parameters: density, reflection intensity, radial relative velocity, and height. Therefore, the original information of the radar data can be retained to a great extent, providing a high-quality data source for subsequent image processing, thereby improving the effect of subsequent processing. (h,ω)

[0041] Step S16: Perform a continuous operation on each of the multiple property maps obtained in step S14.

[0042] Among them, based on the characteristics of the radar data, the multiple property maps obtained in step S14 are discrete. For example, when a certain pixel point does not include radar points, its corresponding property value is 0. However, in the training and inference processes of the deep learning model of the image, excessive 0 values will affect the performance of the model. Therefore, in this embodiment, a Gaussian kernel (such as a Gaussian kernel with an integral of 1) or other radial basis functions are used to perform a continuous operation on each property map to eliminate excessive 0 values.

[0043] ​For example, assume that D(h, w), V(h, w), H(h, w), and I(h, w) represent the density map, radial relative velocity map, height map, and reflection intensity map respectively, and the Gaussian kernel is G σ , where σ is the variance of the Gaussian kernel. Then the continuous operation of this step can be:

[0044]

[0045]

[0046]

[0047]

[0048] where and represent the density map, radial relative velocity map, height map, and reflection intensity map after continuousization respectively.

[0049] In the above method, the variance σ of the Gaussian kernel is a fixed variance. In some other embodiments, the variance σ of the Gaussian kernel can vary adaptively based on the pixel points. For example, a Gaussian kernel with variance σ (h,w) can be selected to perform the above continuousization operation, where where γ is a hyperparameter that can be adjusted according to the actual application. will vary based on the pixel points, taking Figure 3 as an example. Assume that x(0, 0), x(1, 2), x(2, 4), and x(5, 2) are pixel points containing radar points, and the remaining points are pixel points not containing radar points. For x(0, 0), when k = 3, its is the mean of the distances (actual distances detected by the radar) between x(0, 0) and x(1, 2), x(2, 4), and x(5, 2) respectively, where x(1, 2), x(2, 4), and x(5, 2) are the three non-empty pixel points closest to x(0, 0).

[0050] That is to say, when performing the continuousization operation (such as convolution operation) on each pixel point (regardless of whether the point contains a radar point) using the Gaussian kernel, the variance σ of the Gaussian kernel used (h,w) is the mean of the distances between the pixel point and its k nearest non-empty pixel points, where the non-empty pixel points are pixel points containing at least 1 radar point, and the value of k can be adjusted according to actual needs and is not limited to k = 3 above.

[0051] Through the continuousization operation of this step, the 0 values in the data can be reduced. Therefore, in the process of training and inference of the deep learning model, the certainty and stability of the model can be improved.

[0052] Step S18: Generate an imaging result of the radar data based on the multiple continuous attribute maps.

[0053] In this step, each attribute map is organized as data in the "R", "G", and "B" channels in the image, and normalization processing is performed on the data of each channel, thereby obtaining the imaging processing result of the radar data.

[0054] In some embodiments, the radar reflection intensity map can be weighted by using the radar density map, and then the weighted result is processed as the data of the first (e.g., "B") image channel. In addition, the radial relative velocity map and the altitude map are respectively processed as the data of the second (e.g., "R") and third (e.g., "G") image channels.

[0055] In this embodiment, using the radar density map to weight the radar reflection intensity map can better reflect whether there is a real target at this pixel point, thereby improving the subsequent image perception result.

[0056] As Figure 3 shown, it is a schematic structural diagram of an embodiment of the radar data processing device 4 of the present invention. The radar data processing device 3 includes: an acquisition module 40 for acquiring radar data; a conversion module 41 for corresponding the radar points in the radar data to pixel points in the image, where each pixel point includes: 0, 1, or multiple radar points; a calculation module 42 for calculating multiple attributes of each pixel point based on the radar points included in each pixel point to generate multiple attribute maps; a continuity module 43 for respectively performing a continuity operation on the multiple attribute maps; and a generation module 44 for generating an imaging result of the radar data based on the multiple continuous attribute maps.

[0057] In addition, an embodiment of the present invention further provides a computer device, including: a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the steps of the method described in the embodiment of the present invention.

[0058] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program / instructions are executed by a processor, the steps of the method described in the embodiment of the present invention are implemented.

[0059] In addition, an embodiment of the present invention further provides a computer program product, including computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the method described in the embodiment of the present invention are implemented.

[0060] The descriptions of the above device, storage medium, and program product embodiments are similar to those of the above method embodiments and have similar beneficial effects to the method embodiments. For the technical details not disclosed in the device, storage medium, and program product embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0061] The above-mentioned processor can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, etc. It can be understood that other electronic devices can also be used to implement the functions of the above-mentioned processor, and the embodiments of this application do not make specific limitations.

[0062] The above computer storage medium / memory can be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Ferromagnetic Random Access Memory (FRAM), a Flash Memory, a magnetic surface memory, an optical disc, or a Compact Disc Read-Only Memory (CD-ROM), etc.; it can also be various terminals including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0063] It should be noted that the above description is only an example and not a limitation of the present invention. In other embodiments of the present invention, the method may have more, fewer or different steps, and the relationships such as the sequence, inclusion and functions among the steps may be different from those described and illustrated. For example, usually multiple steps can be combined into a single step, and a single step can also be split into multiple steps. For those of ordinary skill in the art, without creative efforts, the sequence changes of each step are also within the protection scope of the present invention.

[0064] The technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.), a processor or a microcontroller to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0065] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments.

[0066] Although the present invention has been disclosed above with preferred embodiments, the present invention is not limited thereto. Any person skilled in the art, without departing from the spirit and scope of the present invention, makes various changes and modifications, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the scope defined by the claims.

Claims

1. A radar data processing method, characterized in that: include: Get radar data; Corresponding radar points in the radar data to pixel points in the image, wherein each pixel point includes: 0, 1 or more radar points; Based on the radar points contained in each pixel point, multiple attributes of each pixel point are calculated to generate multiple attribute maps; Performing a serialization operation on each of the plurality of property graphs; and Based on the plurality of continuous attribute graphs, generating a graphical result of the radar data; Wherein, the multiple attribute maps include: a radar density map and a radar reflection intensity map; The step of generating a graphical result of the radar data based on the plurality of continuous attribute graphs includes: The radar density map is used to weight the radar reflection intensity map, and the weighted result is processed into data of the first image channel.

2. The radar data processing method according to claim 1, characterized in that: The continuation operation is performed based on a Gaussian kernel or a radial basis function.

3. The radar data processing method according to claim 2, characterized in that: The variance of the Gaussian kernel is variable, and the variance of the Gaussian kernel corresponding to each pixel is the mean of the distances between the pixel and its k nearest non-empty pixel points, where a non-empty pixel point is a pixel point that contains at least one radar point.

4. The radar data processing method according to claim 1, characterized in that: The plurality of property maps also include: a radial relative velocity map and a height map; The step of generating a graphical result of the radar data based on the plurality of continuous attribute graphs further includes: Processing the radial relative velocity map and the height map into data of the second and third image channels respectively; Normalization processing is performed on the data of the first to third image channels to generate an imaging result of the radar data.

5. The radar data processing method according to claim 4, characterized in that: The density of each pixel is the number of radar points contained in the pixel; the reflection intensity, radial relative velocity, and height of each pixel are the mean, maximum, or weighted average of the reflection intensity, radial relative velocity, and height of the radar points contained in the pixel, respectively.

6. A radar data processing device, characterized in that: include: An acquisition module, used to acquire radar data; A conversion module, used to correspond radar points in the radar data to pixel points in the image, wherein each pixel point includes: 0, 1 or more radar points; A calculation module, used for calculating multiple attributes of each pixel point based on the radar points contained in each pixel point to generate multiple attribute maps; a continuation module, configured to perform continuation operations on the plurality of property graphs respectively; and A generating module, used for generating a graphical result of the radar data based on the plurality of continuous attribute graphs; Wherein, the multiple attribute maps include: a radar density map and a radar reflection intensity map; The step of generating a graphical result of the radar data based on the plurality of continuous attribute graphs includes: The radar density map is used to weight the radar reflection intensity map, and the weighted result is processed into data of the first image channel.

7. The radar data processing device according to claim 6, characterized in that: The plurality of property maps also include: a radial relative velocity map and a height map; The step of generating a graphical result of the radar data based on the plurality of continuous attribute graphs further includes: Processing the radial relative velocity map and the height map into data of the second and third image channels respectively; Normalization processing is performed on the data of the first to third image channels to generate an imaging result of the radar data.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images