Generation method and device of 4D millimeter wave radar point cloud, electronic equipment and medium
By acquiring and splicing multi-frame laser point clouds, projecting them to 4D millimeter-wave radar tensors, and using deep neural network optimization, the problem of difficulty in optimizing and reducing their own characteristics during the generation of 4D millimeter-wave radar point clouds is solved, achieving high Massive and high density point cloud generation.
Patent Information
- Application Number
- CN202411138577.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-22
AI Technical Summary
It is difficult to take into account the problem of point cloud optimization and reducing the loss of 4D millimeter-wave radar's own characteristics when generating point clouds.
By obtaining multi-frame laser point clouds, dense scene truth value is obtained and projected onto the 4D millimeter wave radar tensor dimension. The original 4D millimeter wave radar tensor information is optimized using a pre-constructed deep neural network, and the optimized information is generated and optimized. 4D mmWave radar point cloud.
While making full use of other sensor information, the loss of the 4D millimeter wave radar itself is avoided, the quality and density of point clouds are improved, and the noise is reduced.
Smart Images

Figure CN119001697A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of radio orientation technology, and in particular to a method, device, electronic device and medium for generating a 4D millimeter wave radar point cloud. Background Art
[0002] 4D millimeter-wave radar refers to a new type of vehicle-mounted millimeter-wave radar that has four dimensions of information: speed, distance, direction angle, and pitch angle, which is different from the traditional 3D vehicle-mounted millimeter-wave radar that only has speed, distance, and direction angle. However, the point cloud quality of 4D millimeter-wave radar is relatively poor, sparse, and noisy.
[0003] In related technologies, there are two main methods for generating 4D millimeter wave radar point clouds:
[0004] One type is based on rules, improving the traditional constant false alarm rate monitoring (CFAR) method, or based on the superposition and filtering of the previous and next frame point clouds. The former cannot solve the fundamental problems of the traditional method, while the latter is prone to information loss.
[0005] The second category is learning-based methods. Existing studies have used the calibration process as supervision, using the network to replace CFAR to generate point clouds, and some studies have used LiDAR point clouds as supervision, using networks such as U-Net and GAN to generate point clouds. These learning-based methods usually have difficulty in accurately extracting the features of the original data and building reasonable network supervision signals. There are also problems in the selection of point cloud generation networks.
[0006] In summary, in the related technology, when generating 4D millimeter-wave radar point clouds, it is difficult to learn rich information from other sensors to achieve point cloud optimization without causing loss of the 4D millimeter-wave radar's own characteristics, which needs to be improved. Summary of the invention
[0007] The present application provides a method, device, electronic device and medium for generating a 4D millimeter-wave radar point cloud to solve the technical problem in the related art that it is difficult to balance point cloud optimization and reduction of the loss of 4D millimeter-wave radar's own characteristics when generating a 4D millimeter-wave radar point cloud.
[0008] The first aspect of the present application provides a method for generating a 4D millimeter-wave radar point cloud, comprising the following steps: obtaining a multi-frame laser point cloud of any target; splicing the multi-frame laser point cloud to obtain a dense scene truth value; projecting the dense scene truth value onto a 4D millimeter-wave radar tensor dimension to obtain original 4D millimeter-wave radar tensor information, and optimizing the original 4D millimeter-wave radar tensor information using a pre-built deep neural network to obtain an optimized 4D millimeter-wave radar tensor, and obtaining the 4D millimeter-wave radar point cloud of any target based on the optimized 4D millimeter-wave radar tensor.
[0009] Optionally, in one embodiment of the present application, the stitching of the multiple frames of laser point clouds to obtain a dense scene truth value includes: dividing the multiple frames of laser point clouds to obtain at least one static point cloud and / or at least one dynamic point cloud; preprocessing the static point cloud to obtain a preprocessed static point cloud, and using the preprocessed static point cloud to calculate the inter-frame pose transformation, stitching the point clouds according to the inter-frame pose transformation to obtain a dense static point cloud, and / or superimposing the dynamic point cloud with multiple frames of target positions to obtain a dense dynamic point cloud, and stitching the dense dynamic point cloud with the dense static point cloud to obtain a dense lidar point cloud; obtaining the dense scene truth value based on the 4D millimeter-wave radar tensor information and the dense lidar point cloud.
[0010] Optionally, in one embodiment of the present application, the optimizing the original 4D millimeter-wave radar tensor information using a pre-built deep neural network includes: constructing a feature extraction network, wherein the feature extraction network includes at least one of an input 3D convolution module, a cross-attention module, and an output 3D deconvolution module; superimposing Focal loss and Diceloss as a loss function of the feature extraction network, and using the loss function to calculate the corresponding output loss; training the feature extraction network based on the output loss and the dense scene truth value to obtain the optimized 4D millimeter-wave radar tensor.
[0011] Optionally, in one embodiment of the present application, obtaining the 4D millimeter-wave radar point cloud of any target based on the optimized 4D millimeter-wave radar tensor includes: extracting any target from the optimized 4D millimeter-wave radar tensor using a preset reflection intensity threshold; and obtaining the 4D millimeter-wave radar point cloud based on any target.
[0012] The second aspect of the present application provides a device for generating a 4D millimeter-wave radar point cloud, including: an acquisition module for acquiring a multi-frame laser point cloud of any target; a splicing module for splicing the multi-frame laser point cloud to obtain a dense scene truth value; a generation module for projecting the dense scene truth value onto a 4D millimeter-wave radar tensor dimension to obtain original 4D millimeter-wave radar tensor information, and using a pre-built deep neural network to optimize the original 4D millimeter-wave radar tensor information to obtain an optimized 4D millimeter-wave radar tensor, and based on the optimized 4D millimeter-wave radar tensor, obtain the 4D millimeter-wave radar point cloud of any target.
[0013] Optionally, in one embodiment of the present application, the stitching module includes: a division unit, used to divide the multi-frame laser point cloud to obtain at least one static point cloud and / or at least one dynamic point cloud; a first computing unit, used to preprocess the static point cloud to obtain a preprocessed static point cloud, and use the preprocessed static point cloud to calculate the inter-frame pose transformation, perform point cloud stitching according to the inter-frame pose transformation to obtain a dense static point cloud, and / or superimpose multiple frames of target positions on the dynamic point cloud to obtain a dense dynamic point cloud, and stitch the dense dynamic point cloud with the dense static point cloud to obtain a dense lidar point cloud; a second computing unit, used to obtain the dense scene truth based on the 4D millimeter-wave radar tensor information and the dense lidar point cloud.
[0014] Optionally, in one embodiment of the present application, the generation module includes: a construction unit, used to construct a feature extraction network, wherein the feature extraction network includes at least one of an input 3D convolution module, a cross attention module and an output 3D deconvolution module; a third calculation unit, used to superimpose Focal loss and Dice loss as a loss function of the feature extraction network, and use the loss function to calculate the corresponding output loss; a training unit, used to train the feature extraction network based on the output loss and the dense scene truth value to obtain the optimized 4D millimeter wave radar tensor.
[0015] Optionally, in one embodiment of the present application, the generation module includes: an extraction unit, used to extract any target from the optimized 4D millimeter-wave radar tensor using a preset reflection intensity threshold; and an acquisition unit, used to obtain the 4D millimeter-wave radar point cloud based on any target.
[0016] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for generating a 4D millimeter-wave radar point cloud as described in the above embodiment.
[0017] The fourth aspect of the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the method for generating a 4D millimeter-wave radar point cloud as described in the above embodiment.
[0018] The fifth aspect embodiment of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned method for generating a 4D millimeter-wave radar point cloud.
[0019] The embodiment of the present application can obtain a multi-frame laser point cloud of any target, obtain a dense scene truth value by splicing the multi-frame laser point cloud, and project the dense scene truth value onto the 4D millimeter-wave radar tensor dimension to obtain the original 4D millimeter-wave radar tensor information, and use a pre-built deep neural network for optimization to obtain an optimized 4D millimeter-wave radar tensor, and then obtain a 4D millimeter-wave radar point cloud, so as to fully utilize other sensor information to achieve point cloud optimization while not causing the loss of the 4D millimeter-wave radar's own characteristics. Thus, the technical problem in the related art that it is difficult to balance point cloud optimization and reduce the loss of 4D millimeter-wave radar's own characteristics when generating 4D millimeter-wave radar point clouds is solved.
[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0022] Figure 1 A flowchart of a method for generating a 4D millimeter-wave radar point cloud according to an embodiment of the present application;
[0023] Figure 2 A schematic diagram of the principle of constructing supervision information based on multi-frame laser point clouds according to an embodiment of the present application;
[0024] Figure 3 A schematic diagram of a 4D millimeter wave radar tensor quality optimization principle according to an embodiment of the present application;
[0025] Figure 4 4D millimeter wave radar point cloud generation method according to an embodiment of the present application;
[0026] Figure 5 A schematic diagram of the structure of a 4D millimeter wave radar point cloud generation device provided according to an embodiment of the present application;
[0027] Figure 6 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0029] The following describes the generation method, device, electronic device and medium of the 4D millimeter wave radar point cloud of the embodiment of the present application with reference to the accompanying drawings. In view of the technical problem that it is difficult to balance point cloud optimization and reduce the loss of 4D millimeter wave radar's own characteristics when generating 4D millimeter wave radar point clouds in the related art mentioned in the above background technology, the present application provides a 4D millimeter wave radar point cloud generation method, in which a multi-frame laser point cloud of any target can be obtained, and a dense scene truth value is obtained by splicing the multi-frame laser point cloud, and the dense scene truth value is projected onto the 4D millimeter wave radar tensor dimension to obtain the original 4D millimeter wave radar tensor information, and optimized using a pre-built deep neural network to obtain an optimized 4D millimeter wave radar tensor, and then obtain a 4D millimeter wave radar point cloud, so that while making full use of other sensor information to achieve point cloud optimization, it does not cause the loss of 4D millimeter wave radar's own characteristics. As a result, the technical problem that it is difficult to balance point cloud optimization and reduce the loss of 4D millimeter wave radar's own characteristics when generating 4D millimeter wave radar point clouds in the related art is solved.
[0030] Specifically, Figure 1 A flowchart of a method for generating a 4D millimeter-wave radar point cloud provided in an embodiment of the present application.
[0031] like Figure 1 As shown, the method for generating the 4D millimeter wave radar point cloud includes the following steps:
[0032] In step S101, a multi-frame laser point cloud of any target is obtained.
[0033] It is understandable that CBDES (Computing Brain Development System) consists of two parts: CBB (Computing Brain) and GAASD (Graphical ADAS-AD Software Developer). CBB consists of computing platform hardware, real-time kernel, middleware and functional software. Functional software is the core of this product, which aims to provide basic algorithm components and frameworks for various intelligent driving systems. Based on the innovative solution of "layered decoupling and cross-domain sharing" and based on the functional software function library, CBDES transfers the application algorithm development capability to the host manufacturers, supports the engineers of the host manufacturers to quickly build their own defined intelligent driving systems, and perform function adaptation and parameter adjustment. Compared with existing products and development models, it has the three advantages of "high efficiency / high quality / generative".
[0034] Based on the above system, the embodiment of the present application can construct a vehicle intelligent driving system that meets the design requirements, such as a positioning system, etc. In the embodiment of the present application, it can be used to construct an automatic driving positioning system based on 4D millimeter wave radar.
[0035] Among them, 4D millimeter-wave radar refers to a new type of vehicle-mounted millimeter-wave radar that has four dimensions of information: speed, distance, direction angle, and pitch angle, which is different from the traditional 3D vehicle-mounted millimeter-wave radar that only has speed, distance, and direction angle. However, the point cloud quality of 4D millimeter-wave radar is relatively poor, sparse, and noisy.
[0036] In order to solve the above problems, the embodiments of the present application can optimize the point cloud by learning rich information from other sensors without causing loss of the characteristics of the 4D millimeter-wave radar itself.
[0037] Using laser radar as other sensors, the embodiments of the present application can obtain multi-frame laser point clouds of any target by setting a scanning task.
[0038] In step S102, multiple frames of laser point clouds are stitched together to obtain a dense scene truth value.
[0039] Furthermore, the embodiments of the present application can generate dense scene truth based on the stitching of multiple frames of laser point clouds to achieve high-quality optimization of the point cloud.
[0040] Optionally, in one embodiment of the present application, splicing multiple frames of laser point clouds to obtain a dense scene truth, including: dividing multiple frames of laser point clouds to obtain at least one static point cloud and / or at least one dynamic point cloud; preprocessing the static point cloud to obtain a preprocessed static point cloud, and using the preprocessed static point cloud to calculate inter-frame pose transformation, splicing the point clouds according to the inter-frame pose transformation to obtain a dense static point cloud, and / or superimposing multiple frames of target positions on the dynamic point cloud to obtain a dense dynamic point cloud, and splicing the dense dynamic point cloud with the dense static point cloud to obtain a dense lidar point cloud; obtaining a dense scene truth based on 4D millimeter-wave radar tensor information and the dense lidar point cloud.
[0041] like Figure 2 As shown, the embodiment of the present application may include the following steps:
[0042] Step S1: distinguishing dynamic and static point clouds based on target labels from multi-frame laser point clouds;
[0043] Step S2: For the static point cloud, first remove the ground, then calculate the inter-frame pose transformation based on point cloud matching, complete the point cloud stitching, and obtain a dense static point cloud;
[0044] Step S3: for the dynamic point cloud, a dense dynamic point cloud is obtained based on the superposition of the target positions of multiple frames, and the dynamic point cloud is spliced with the dense static point cloud to obtain a dense lidar point cloud;
[0045] Step S4: Based on the 4D millimeter-wave radar tensor information, the dense lidar point cloud is sequentially subjected to external parameter transformation, spherical coordinate projection, region of interest extraction, voxelization, and millimeter-wave radar intensity filtering to obtain a dense 3D occupancy grid truth value, that is, a dense scene truth value.
[0046] It should be noted that there may be no dynamic point cloud part in a single-frame point cloud. In this case, the embodiment of the present application can only splice the static point cloud to obtain the corresponding dense static point cloud, and obtain the dense scene truth based on the dense static point cloud.
[0047] In step S103, the dense scene truth is projected onto the 4D millimeter-wave radar tensor dimension to obtain the original 4D millimeter-wave radar tensor information, and the pre-built deep neural network is used to optimize the original 4D millimeter-wave radar tensor information to obtain the optimized 4D millimeter-wave radar tensor, and the 4D millimeter-wave radar point cloud of any target is obtained based on the optimized 4D millimeter-wave radar tensor.
[0048] As a possible implementation method, the embodiment of the present application can project the dense scene truth back to the 4D millimeter-wave radar tensor dimension, construct a deep neural network, input the original 4D millimeter-wave radar tensor information, realize feature extraction through modules such as the self-attention mechanism, and compare the extracted 4D millimeter-wave radar tensor features with the dense scene truth to calculate the loss, realize the utilization of other sensor information, train to obtain optimized 4D millimeter-wave radar tensor information, and then obtain high-quality 4D millimeter-wave radar point cloud.
[0049] Optionally, in one embodiment of the present application, the optimizing the original 4D millimeter-wave radar tensor information using a pre-built deep neural network includes: constructing a feature extraction network, wherein the feature extraction network includes at least one of an input 3D convolution module, a cross-attention module, and an output 3D deconvolution module; superimposing Focal loss and Diceloss as a loss function of the feature extraction network, and using the loss function to calculate the corresponding output loss; training the feature extraction network based on the output loss and the dense scene truth value to obtain the optimized 4D millimeter-wave radar tensor.
[0050] like Figure 3 As shown, 4D millimeter wave radar tensor quality optimization may include the following steps:
[0051] Step S1: Build a feature extraction network based on 3D U-Net and add modules such as input 3D convolution, cross attention, and output 3D deconvolution;
[0052] Step S2: Focal loss and Dice loss are superimposed as the loss function of the network, and the loss is calculated using the output at multiple resolutions;
[0053] Step S3: Input the 4D millimeter wave radar tensor into the feature extraction network, and train the network using the dense scene truth as supervision.
[0054] Optionally, in one embodiment of the present application, a 4D millimeter-wave radar point cloud of any target is obtained based on the optimized 4D millimeter-wave radar tensor, including: extracting any target from the optimized 4D millimeter-wave radar tensor using a preset reflection intensity threshold; and obtaining a 4D millimeter-wave radar point cloud based on any target.
[0055] Furthermore, after obtaining the optimized 4D millimeter-wave radar tensor, the embodiment of the present application can extract the actual target from the optimized tensor by setting a reflection intensity threshold to obtain a 4D millimeter-wave radar point cloud.
[0056] The reflection intensity threshold can be set by those skilled in the art according to actual conditions, and is not specifically limited here.
[0057] Combination Figures 2 to 4 As shown, the working principle of the method for generating 4D millimeter wave radar point cloud of the embodiment of the present application is described in detail with an embodiment.
[0058] like Figure 4 As shown, the embodiment of the present application may include the following steps:
[0059] Step S401: Obtain multi-frame laser point clouds.
[0060] Step S402: Generate dense scene truth values. The embodiment of the present application can generate dense scene truth values based on the splicing of multiple frames of laser point clouds, construct point cloud generation supervision information, and improve the richness of supervision information.
[0061] Among them, Figure 2 As shown, the construction of supervision information based on multi-frame laser point clouds can include the following steps:
[0062] Step S1: distinguishing dynamic and static point clouds based on target labels from multi-frame laser point clouds;
[0063] Step S2: For the static point cloud, first remove the ground, then calculate the inter-frame pose transformation based on point cloud matching, complete the point cloud stitching, and obtain a dense static point cloud;
[0064] Step S3: for the dynamic point cloud, a dense dynamic point cloud is obtained based on the superposition of the target positions of multiple frames, and the dynamic point cloud is spliced with the dense static point cloud to obtain a dense lidar point cloud;
[0065] Step S4: Based on the 4D millimeter-wave radar tensor information, the dense lidar point cloud is sequentially subjected to external parameter transformation, spherical coordinate projection, region of interest extraction, voxelization, and millimeter-wave radar intensity filtering to obtain a dense 3D occupancy grid truth value, that is, a dense scene truth value.
[0066] Step S403: Obtaining the original 4D millimeter wave radar tensor. In the embodiment of the present application, the dense scene truth value can be projected onto the 4D millimeter wave radar tensor dimension to obtain the original 4D millimeter wave radar tensor information.
[0067] Step S404: Input the dense scene truth and the original 4D millimeter wave radar tensor into the deep neural network.
[0068] like Figure 3 As shown, 4D millimeter wave radar tensor quality optimization may include the following steps:
[0069] Step S1: Build a feature extraction network based on 3D U-Net and add modules such as input 3D convolution, cross attention, and output 3D deconvolution;
[0070] Step S2: Focal loss and Dice loss are superimposed as the loss function of the network, and the loss is calculated using the output at multiple resolutions;
[0071] Step S3: Input the 4D millimeter wave radar tensor into the feature extraction network, and train the network using the dense scene truth as supervision.
[0072] Step S405: Obtaining an optimized 4D millimeter-wave radar tensor.
[0073] Step S406: Outputting a 4D millimeter wave radar point cloud. In the embodiment of the present application, after obtaining the optimized 4D millimeter wave radar tensor, the actual target can be extracted from the optimized tensor by setting a reflection intensity threshold to obtain a 4D millimeter wave radar point cloud.
[0074] In summary, the embodiments of the present application can better extract the features of the original 4D millimeter-wave radar tensor, and realize the learning of supervised information through a deep neural network to obtain a high-quality 4D millimeter-wave radar point cloud.
[0075] According to the method for generating 4D millimeter-wave radar point clouds proposed in the embodiment of the present application, it is possible to obtain multi-frame laser point clouds of any target, obtain dense scene truth values by splicing multi-frame laser point clouds, and project the dense scene truth values onto the 4D millimeter-wave radar tensor dimension to obtain the original 4D millimeter-wave radar tensor information, and optimize it using a pre-built deep neural network to obtain an optimized 4D millimeter-wave radar tensor, and then obtain a 4D millimeter-wave radar point cloud, thereby fully utilizing other sensor information to achieve point cloud optimization without causing loss of the 4D millimeter-wave radar's own characteristics. This solves the technical problem in the related art that it is difficult to balance point cloud optimization and reduce the loss of the 4D millimeter-wave radar's own characteristics when generating 4D millimeter-wave radar point clouds.
[0076] Next, a device for generating a 4D millimeter-wave radar point cloud according to an embodiment of the present application is described with reference to the accompanying drawings.
[0077] Figure 5 It is a block diagram of a device for generating a 4D millimeter-wave radar point cloud according to an embodiment of the present application.
[0078] like Figure 5 As shown, the 4D millimeter wave radar point cloud generation device 10 includes: an acquisition module 100, a stitching module 200 and a generation module 300.
[0079] Specifically, the acquisition module 100 is used to acquire a multi-frame laser point cloud of any target.
[0080] The stitching module 200 is used to stitch multiple frames of laser point clouds to obtain a dense scene truth value.
[0081] The generation module 300 is used to project the dense scene truth value onto the 4D millimeter-wave radar tensor dimension to obtain the original 4D millimeter-wave radar tensor information, and use a pre-built deep neural network to optimize the original 4D millimeter-wave radar tensor information to obtain an optimized 4D millimeter-wave radar tensor, and obtain a 4D millimeter-wave radar point cloud of any target based on the optimized 4D millimeter-wave radar tensor.
[0082] Optionally, in one embodiment of the present application, the splicing module 200 includes: a dividing unit, a first computing unit and a second computing unit.
[0083] Wherein, the division unit is used to divide the multi-frame laser point cloud to obtain at least one static point cloud and / or at least one dynamic point cloud.
[0084] The first computing unit is used to preprocess the static point cloud to obtain a preprocessed static point cloud, and use the preprocessed static point cloud to calculate the inter-frame pose transformation, and stitch the point clouds according to the inter-frame pose transformation to obtain a dense static point cloud, and / or, superimpose the dynamic point cloud with multiple frames of target positions to obtain a dense dynamic point cloud, and stitch the dense dynamic point cloud with the dense static point cloud to obtain a dense lidar point cloud.
[0085] The second computing unit is used to obtain dense scene truth based on 4D millimeter wave radar tensor information and dense lidar point cloud.
[0086] Optionally, in one embodiment of the present application, the generation module 300 includes: a construction unit, a third calculation unit and a training unit.
[0087] Among them, the construction unit is used to construct a feature extraction network, wherein the feature extraction network includes at least one of an input 3D convolution module, a cross attention module and an output 3D deconvolution module.
[0088] The third calculation unit is used to superimpose Focal loss and Dice loss as the loss function of the feature extraction network, and calculate the corresponding output loss using the loss function.
[0089] A training unit is used to train a feature extraction network based on output loss and dense scene truth to obtain an optimized 4D millimeter wave radar tensor.
[0090] Optionally, in one embodiment of the present application, the generating module 300 includes: an extracting unit and an acquiring unit.
[0091] The extraction unit is used to extract any target from the optimized 4D millimeter-wave radar tensor using a preset reflection intensity threshold.
[0092] The acquisition unit is used to obtain a 4D millimeter-wave radar point cloud according to any target.
[0093] It should be noted that the aforementioned explanation of the embodiment of the method for generating a 4D millimeter-wave radar point cloud is also applicable to the device for generating a 4D millimeter-wave radar point cloud of this embodiment, and will not be repeated here.
[0094] According to the 4D millimeter wave radar point cloud generation device proposed in the embodiment of the present application, a multi-frame laser point cloud of any target can be obtained, and a dense scene truth value is obtained by splicing the multi-frame laser point cloud, and the dense scene truth value is projected onto the 4D millimeter wave radar tensor dimension to obtain the original 4D millimeter wave radar tensor information, and optimized using a pre-built deep neural network to obtain an optimized 4D millimeter wave radar tensor, and then obtain a 4D millimeter wave radar point cloud, so that while making full use of other sensor information to achieve point cloud optimization, the 4D millimeter wave radar's own characteristics are not lost. Thus, the technical problem in the related art that it is difficult to balance point cloud optimization and reduce the loss of 4D millimeter wave radar's own characteristics when generating 4D millimeter wave radar point clouds is solved.
[0095] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:
[0096] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .
[0097] When the processor 602 executes the program, the method for generating the 6D millimeter wave radar point cloud provided in the above embodiment is implemented.
[0098] Furthermore, the electronic device further comprises:
[0099] The communication interface 603 is used for communication between the memory 601 and the processor 602 .
[0100] The memory 601 is used to store computer programs that can be executed on the processor 602 .
[0101] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0102] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0103] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.
[0104] The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0105] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the above-mentioned method for generating a 4D millimeter-wave radar point cloud is implemented.
[0106] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for generating a 4D millimeter-wave radar point cloud provided in an embodiment of the present invention.
[0107] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0108] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0109] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0110] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.
[0111] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0112] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0113] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0114] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A method for generating a 4D millimeter wave radar point cloud, characterized in that: The following steps are involved: Get multi-frame laser point cloud of any target; Splicing the multiple frames of laser point clouds to obtain a dense scene truth value; The dense scene truth is projected onto the 4D millimeter-wave radar tensor dimension to obtain original 4D millimeter-wave radar tensor information, and a pre-built deep neural network is used to optimize the original 4D millimeter-wave radar tensor information to obtain an optimized 4D millimeter-wave radar tensor, and the 4D millimeter-wave radar point cloud of any target is obtained based on the optimized 4D millimeter-wave radar tensor.
2. The method according to claim 1, characterized in that The step of stitching the multiple frames of laser point clouds to obtain a dense scene truth value includes: Dividing the multi-frame laser point cloud to obtain at least one static point cloud and / or at least one dynamic point cloud; Preprocessing the static point cloud to obtain a preprocessed static point cloud, computing inter-frame pose transformation using the preprocessed static point cloud, performing point cloud stitching according to the inter-frame pose transformation to obtain a dense static point cloud, and / or, Superimposing the dynamic point cloud with multiple frames of target positions to obtain a dense dynamic point cloud, and splicing the dense dynamic point cloud with the dense static point cloud to obtain a dense lidar point cloud; The dense scene truth is obtained based on the 4D millimeter-wave radar tensor information and the dense lidar point cloud.
3. The method according to claim 1, characterized in that The optimizing the original 4D millimeter wave radar tensor information by using a pre-built deep neural network includes: Constructing a feature extraction network, wherein the feature extraction network includes at least one of an input 3D convolution module, a cross attention module, and an output 3D deconvolution module; The Focalloss and Diceloss are superimposed as the loss function of the feature extraction network, and the corresponding output loss is calculated using the loss function; The feature extraction network is trained based on the output loss and the dense scene truth value to obtain the optimized 4D millimeter-wave radar tensor.
4. The method according to claim 1, characterized in that: The obtaining the 4D millimeter-wave radar point cloud of any target based on the optimized 4D millimeter-wave radar tensor includes: Extracting any one of the targets from the optimized 4D millimeter-wave radar tensor using a preset reflection intensity threshold; The 4D millimeter-wave radar point cloud is obtained according to any one of the targets.
5. A 4D millimeter wave radar point cloud generation device, characterized in that: include: Acquisition module, used to acquire multi-frame laser point cloud of any target; A stitching module, used for stitching the multiple frames of laser point clouds to obtain a dense scene truth value; A generation module is used to project the dense scene truth value onto the 4D millimeter-wave radar tensor dimension to obtain original 4D millimeter-wave radar tensor information, and optimize the original 4D millimeter-wave radar tensor information using a pre-built deep neural network to obtain an optimized 4D millimeter-wave radar tensor, and obtain the 4D millimeter-wave radar point cloud of any target based on the optimized 4D millimeter-wave radar tensor.
6. The device according to claim 5, characterized in that The splicing module comprises: A division unit, used for dividing the multi-frame laser point cloud to obtain at least one static point cloud and / or at least one dynamic point cloud; A first computing unit is configured to preprocess the static point cloud to obtain a preprocessed static point cloud, calculate inter-frame pose transformation using the preprocessed static point cloud, perform point cloud splicing according to the inter-frame pose transformation to obtain a dense static point cloud, and / or perform multi-frame target position superposition on the dynamic point cloud to obtain a dense dynamic point cloud, and splice the dense dynamic point cloud with the dense static point cloud to obtain a dense lidar point cloud; The second computing unit is used to obtain the dense scene truth value based on the 4D millimeter wave radar tensor information and the dense lidar point cloud.
7. The device according to claim 5, characterized in that The generation module comprises: A construction unit, configured to construct a feature extraction network, wherein the feature extraction network comprises at least one of an input 3D convolution module, a cross attention module, and an output 3D deconvolution module; A third calculation unit is used to superimpose Focal loss and Dice loss as a loss function of the feature extraction network, and calculate a corresponding output loss using the loss function; A training unit is used to train the feature extraction network based on the output loss and the dense scene truth value to obtain the optimized 4D millimeter wave radar tensor.
8. The device according to claim 5, characterized in that The generation module comprises: An extraction unit, configured to extract any target from any target in the optimized 4D millimeter-wave radar tensor using a preset reflection intensity threshold; An acquisition unit is used to obtain the 4D millimeter-wave radar point cloud according to any one of the targets.
9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for generating a 4D millimeter-wave radar point cloud as described in any one of claims 1 to 4.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method for generating a 4D millimeter-wave radar point cloud as described in any one of claims 1 to 4.
Citation Information
Cited By
Automatic driving positioning method and device of vehicle, vehicle and storage medium
CN119105020A
Autopilot Positioning Method, Device, Vehicle and Storage Medium for a Vehicle
CN119105020B
4D millimeter wave radar space occupation probability estimation method based on laser radar supervision
CN120178229A
