Cross-modal four-dimensional radar denoising method and device based on laser radar supervision

Abstract

The invention provides a cross-modal four-dimensional radar de-noising method and device based on laser radar supervision, and the method comprises the steps: constructing a de-noising model composed of a feature encoder, a noise predictor and a matching module, and obtaining a marked bounding box and a real-class four-dimensional radar point cloud, and taking the marked bounding box and real-class four-dimensional radar point cloud as training data; the feature encoder extracts aggregation features of the training data; the noise predictor obtains a prediction point cloud according to the aggregation features, and each position data point in the prediction point cloud has a prediction category; the matching module aligns the prediction point cloud with the training data to obtain a noise-removed prediction point mask, and inputs the prediction point mask into the classification model to obtain a target area in the training data and a corresponding prediction category; constructing boundary loss according to the target area and the bounding box, and constructing category loss according to the prediction category and the real category; training a denoising model according to the boundary loss and the category loss; and inputting the to-be-denoised four-dimensional radar point cloud into the trained denoising model to obtain a four-dimensional radar denoising result point cloud.

Description

Technical Field

[0001] The present invention relates to the fields of autonomous driving technology, environmental perception technology and millimeter wave radar denoising technology, and in particular to a cross-modal four-dimensional radar denoising network method, device, electronic device, computer-readable storage medium and computer program product based on laser radar supervision. Background Art

[0002] The environmental perception module is crucial in autonomous driving and plays a pivotal role in the subsequent operations of the planning, decision-making, and control modules. Currently, most autonomous driving systems use cameras and lidar as their main sensors, but these sensors are not ideal in adverse weather conditions. Millimeter-wave radar sensors use relatively long wavelengths for detection and have strong anti-interference capabilities. They have received widespread attention due to their environmental adaptability and good reliability. In the field of autonomous driving, mainstream radar sensors are mainly divided into two categories: three-dimensional radar and four-dimensional radar. Compared with three-dimensional radar, four-dimensional radar only exists in millimeter-wave radar, and four-dimensional radar can provide spatial three-dimensional information such as distance, azimuth, elevation, and Doppler velocity, so that more spatial information can cope with various driving scenarios.

[0003] Compared with LiDAR, the point clouds collected by millimeter-wave radar show strong noise, which is particularly prominent in 4D radar. Strong noise significantly increases the false detection rate of the sensor, so whether it is possible to accurately distinguish valid points from noise points in 4D radar point clouds is a major challenge in 4D radar perception strategies. In the early stages, a large number of 3D radar denoising methods have been proposed: some traditional optimization-based methods, such as moving least squares and low-rank learning, use geometric projection and self-similarity for denoising, but such methods have difficulty in handling complex, noisy and highly irregular radar data. It is worth mentioning that neural network-based methods have achieved impressive results under low-noise conditions, but often cannot handle denser and more variable noise levels and larger point clouds. Some workers focus on modal fusion and try to alleviate the limitations of radar by integrating LiDAR information, but they remain unexplored on 4D radar, where noise and sparsity are more obvious, limiting their robustness and versatility in real-world scenarios. Summary of the invention

[0004] In view of the shortcomings of existing technologies, such as Figure 3 As shown, the present invention proposes a cross-modal four-dimensional radar denoising method based on laser radar supervision, which includes:

[0005] In the initial step, a denoising model consisting of a feature encoder, a noise predictor, and a matching module is constructed to obtain a four-dimensional radar point cloud with annotated bounding boxes and true categories as training data;

[0006] In the training step, the feature encoder extracts the aggregated features of the training data; the noise predictor obtains a predicted point cloud according to the aggregated features, and each position data point in the predicted point cloud has a predicted category; the matching module aligns the predicted point cloud with the training data to obtain a predicted point mask with noise removed, and inputs the predicted point mask into the classification model to obtain a target area in the training data and its corresponding predicted category; a boundary loss is constructed according to the target area and the bounding box, and a category loss is constructed according to the predicted category and the true category; and the denoising model is trained according to the boundary loss and the category loss;

[0007] In the denoising step, the four-dimensional radar point cloud to be denoised is input into the trained denoising model to obtain the four-dimensional radar denoising result point cloud.

[0008] The cross-modal four-dimensional radar denoising method based on laser radar supervision, wherein the training data includes: laser radar point cloud and four-dimensional radar point cloud taken in the same scene;

[0009] The training step includes: aligning the lidar point cloud with the four-dimensional radar point cloud, generating a supervised point mask in the form of voxels for the lidar point cloud, matching the predicted point mask with the supervised point mask to obtain a cross-modal supervision loss; and training the denoising model according to the boundary loss, the category loss and the cross-modal supervision loss.

[0010] The cross-modal four-dimensional radar denoising method based on lidar supervision, wherein the feature encoder applies multiple encoding layers to gradually extract the aggregated features from the point cloud.

[0011] The cross-modal four-dimensional radar denoising method based on laser radar supervision, wherein the denoising step includes: analyzing the four-dimensional radar denoising result point cloud through a classification model to obtain the classification of each point in the four-dimensional radar denoising result point cloud, and based on the classification of each point in the four-dimensional radar denoising result point cloud, controlling the vehicle to complete automatic or assisted driving tasks, such as assisted pedestrian avoidance and automatic emergency braking AEB tasks.

[0012] like Figure 4 As shown, the present invention also proposes a cross-modal four-dimensional radar denoising device based on laser radar supervision, which includes:

[0013] The initial module builds a denoising model consisting of a feature encoder, a noise predictor, and a matching module, and obtains a four-dimensional radar point cloud with annotated bounding boxes and true categories as training data;

[0014] The training module comprises a feature encoder that extracts aggregate features of the training data; the noise predictor obtains a predicted point cloud based on the aggregate features, and each position data point in the predicted point cloud has a predicted category; the matching module aligns the predicted point cloud with the training data to obtain a predicted point mask from which noise is removed, and the predicted point mask is input into a classification model to obtain a target area in the training data and its corresponding predicted category; a boundary loss is constructed based on the target area and the bounding box, and a category loss is constructed based on the predicted category and the true category; and the denoising model is trained based on the boundary loss and the category loss;

[0015] The denoising module inputs the four-dimensional radar point cloud to be denoised into the trained denoising model to obtain the four-dimensional radar denoising result point cloud.

[0016] The cross-modal four-dimensional radar denoising device based on laser radar supervision, wherein the training data includes: laser radar point cloud and four-dimensional radar point cloud taken in the same scene;

[0017] The training module includes: aligning the lidar point cloud with the four-dimensional radar point cloud, generating a supervised point mask in the form of voxels for the lidar point cloud, matching the predicted point mask with the supervised point mask to obtain a cross-modal supervision loss; and training the denoising model according to the boundary loss, the category loss and the cross-modal supervision loss.

[0018] The cross-modal four-dimensional radar denoising device based on laser radar supervision, wherein the denoising module includes: analyzing the four-dimensional radar denoising result point cloud through a classification model, obtaining the classification of each point in the four-dimensional radar denoising result point cloud, and based on the classification of each point in the four-dimensional radar denoising result point cloud, controlling the vehicle to complete automatic or assisted driving tasks.

[0019] The present invention also proposes an electronic device, which includes the above-mentioned cross-modal four-dimensional radar denoising device, and the electronic device is connected to an information display device, which is used to display the four-dimensional radar denoising result point cloud according to display parameters and attributes set by the user or through an artificial intelligence model.

[0020] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the cross-modal four-dimensional radar denoising method are implemented.

[0021] The present invention also proposes a computer program product, comprising a computer program, wherein the computer program implements the steps of the cross-modal four-dimensional radar denoising method when executed by a processor.

[0022] It can be seen from the above scheme that the advantages of the present invention are:

[0023] Through the analysis of four-dimensional radar noise, the present invention finds that the noise is mainly due to energy leakage around strong targets and the resolution of the radar is limited. Therefore, to improve the perception ability of the four-dimensional radar, it is necessary to identify the points in the point cloud that are related to the actual object. The present invention draws inspiration from the latest progress in multimodal fusion. Since lidar can accurately describe the real position of objects in the physical world, the present invention hopes to improve the detection capability of the four-dimensional radar-based model by constraining the four-dimensional radar through lidar specifications. The present invention proposes a cross-modal learning strategy, which uses lidar point clouds in the training process to enable the detection model to implicitly identify noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The millimeter wave radar denoising image of the present invention;

[0025] Figure 2 A diagram of the processing device of the present invention;

[0026] Figure 3 is a flow chart of the method of the present invention;

[0027] Figure 4 It is a module diagram of the device of the present invention;

[0028] Figure 5 This is a schematic diagram of the structure of a first electronic device of the present invention;

[0029] Figure 6 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0030] Figure 7 It is a schematic diagram of the structure of a second electronic device of the present invention.

[0031] Reference numerals:

[0032] A-First electronic device;

[0033] B-Cross-modal four-dimensional radar denoising device;

[0034] C-data acquisition equipment;

[0035] D-information display device;

[0036] 1000 - second electronic device;

[0037] Ⅰ-computational unit;

[0038] II-ROM;

[0039] III-RAM;

[0040] IV-bus;

[0041] V-interface;

[0042] VI - input unit;

[0043] VII-output unit;

[0044] VIII- Storage medium;

[0045] Ⅸ-Communication unit. DETAILED DESCRIPTION

[0046] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0047] Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0048] The processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0049] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0050] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0051] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0052] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0053] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0054] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0055] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0056] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0057] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0058] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0059] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0060] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0061] Through the analysis of four-dimensional radar noise, the present invention finds that the noise is mainly due to energy leakage around strong targets and the resolution of the radar is limited. Therefore, to improve the perception ability of the four-dimensional radar, it is necessary to identify the points in the point cloud that are related to the actual objects. The present invention draws inspiration from the latest progress in multimodal fusion. Since lidar can accurately describe the real position of objects in the physical world, the present invention hopes to improve the detection capability of the four-dimensional radar-based model by constraining the four-dimensional radar through lidar specifications. The present invention proposes a cross-modal learning strategy, which uses lidar point clouds in the training process to enable the detection model to implicitly identify noise. Specifically, the present invention includes the following key technical points:

[0062] Technical point 1: The present invention proposes a millimeter-wave radar denoising network based on hierarchical feature aggregation and predictive matching mechanism, which utilizes hierarchical feature aggregation to construct a multi-scale feature extraction module for four-dimensional millimeter-wave radar, and uses the fitting ability of neural networks to effectively distinguish between noise and valid point clouds in millimeter-wave radar point clouds.

[0063] Technical point 2, the core network of the present invention includes three parts: feature encoder, noise predictor and matching module. Feature encoder: gradually extracts aggregate features from radar point cloud through hierarchical feature learning architecture; Noise predictor: estimates the noise and validity of each point in radar point cloud, generates prediction mask, and the mask score reflects the validity of point cloud; Matching module: matches the filtered point cloud with the original point cloud based on KD-Tree structure to ensure the coverage of valid point cloud.

[0064] Technical point 3, according to the method of claim 1, it is characterized in that the matched point cloud is processed by the backbone network and the detection head to generate category prediction and position prediction. The training of the network is supervised using a binary cross entropy loss function and a mean square error loss function, wherein the loss function is used to optimize the predicted bounding box position and category.

[0065] Technical point 4: Use the lidar point cloud as a supervisory signal to denoise the four-dimensional millimeter-wave radar point cloud during training. The mechanism includes: the lidar point cloud is processed by a mask generator to generate a supervised point mask; the mask is matched with the predicted mask of the radar point cloud, and the loss function is calculated to guide the training of the denoising network.

[0066] Technical point 5: During the inference phase, the LiDAR data is no longer required because the trained denoising network has effective denoising capabilities. By eliminating the need for LiDAR data during the inference process, the cross-modal processing device maintains the perceived inference efficiency and practicality.

[0067] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0068] The goal of the present invention is to solve the problem of high-level noise in environmental perception of four-dimensional millimeter-wave radar. A cross-modal four-dimensional radar supervision method and device based on lidar point cloud is proposed, which uses the alignment of lidar point cloud with four-dimensional radar point cloud to effectively distinguish noise from valid point clouds. It specifically includes three parts: constructing a core module for point cloud denoising; introducing a cross-modal supervision mechanism of lidar-four-dimensional millimeter-wave radar; 3. Design of a full loss function involving mask calculation. This application describes the above steps in detail as follows.

[0069] 1. Construct a millimeter-wave radar denoising network under the hierarchical feature aggregation and prediction matching mechanism:

[0070] The denoising network designed in the present invention consists of a feature encoder Noise Predictor and matching modules The present invention adopts a hierarchical feature learning architecture to design a feature encoder. Given a high-noise radar point cloud x as input, the feature encoder Apply n encoding layers to gradually extract aggregate features from the point cloud. Each layer represents a transformation of the input, with the goal of discarding noise while retaining useful information. That is, feature extraction is performed on each point cloud. The original difference between "noise points" and "non-noise points" is small, and the feature encoder The role of is to expand the difference between the "noise point" feature and the "non-noise point" feature, in order to discard the noise and retain the effective information in the future. The connection between the layers captures the complex interaction in the hidden feature h. The feature h output by the feature encoder is n It can be expressed as:

[0071]

[0072] Then the extracted point feature h n Passed to the noise predictor The neural network of this module is responsible for estimating the noise and valid points in the radar point cloud.

[0073]

[0074] M radar is the predicted mask, which indicates the location of the valid radar point cloud. The data size of the mask is the same as the point cloud size. The mask score reflects the validity of the point cloud. The higher the score, the higher the confidence of the point cloud, and vice versa. radar Input Matching Module The matching module aligns the point cloud filtered by the noise predictor P with the original radar point cloud to ensure that the receptive field is sufficient to cover most of the valid point clouds. The present invention implements the matching module using a KD-Tree structure.

[0075]

[0076] Then the backbone network and the detection head match the point cloud P match Process and get the test result. cls and reg They are category prediction and position prediction respectively. The present invention uses binary cross entropy loss BCE(·) and mean square error MSE(·) for regression to supervise network training.

[0077] L cls =BCE(c cls ,c gt )

[0078] L reg =MSE(y reg -t gt )

[0079] where t gt is the true bounding box, c gt is the true category of the target area, such as vehicles, pedestrians, lane lines, etc.

[0080] The network aims to improve the reliability of radar point clouds in high-noise environments, which is crucial for robust perception of autonomous driving systems.

[0081] 2. Cross-modal supervision mechanism using LiDAR-Millimeter Wave Radar:

[0082] The present invention introduces a new cross-modal supervision mechanism, which uses lidar point cloud to supervise the denoising process of four-dimensional millimeter-wave radar point cloud during training, while eliminating the dependence on lidar data during reasoning.

[0083] During the training phase, both the LiDAR and 4D radar point clouds are used as input data. The LiDAR point cloud is processed by a mask generator, which aligns the LiDAR data with the 4D radar data. This module generates a supervised point mask in the form of voxels to guide the denoising process of the radar point cloud. At the same time, the radar point cloud passes through the denoising network designed in the previous section to generate a predicted mask that can distinguish valid point clouds from noisy point clouds. The predicted point mask is then matched with the supervised point mask obtained from the LiDAR data. These two masks will be used to calculate L msk , where N is the number of points and L1 represents the smoothL1 loss.

[0084]

[0085] During the inference phase, lidar data is no longer required due to the effective denoising capabilities of the trained denoising network. By eliminating the need for lidar data during inference, this cross-modal framework maintains the efficiency and practicality of lidar-based perception while benefiting from the superior accuracy of lidar supervision during training.

[0086] 3. Design of full loss function involving mask calculation:

[0087] For network training of 3D object detection models, this paper proposes a loss function involving mask calculation:

[0088] L total =α·L cls +β·L reg +γ·L msk

[0089] L total It consists of three parts: the classification loss of the classifier (L cls ), regression loss (L reg ) and the effective voxel loss (L msk ). cls Using the focal loss, L reg and L msk A smooth L1 function is used. Coefficients α, β and γ are set as hyperparameters, and their values ​​are 1, 2 and 10 respectively. The present invention uses a gradient descent algorithm to optimize weights to minimize losses.

[0090] Example 1

[0091] Figure 1The present invention provides a millimeter wave radar denoising network design method based on hierarchical feature aggregation and prediction matching mechanism, and the steps are as follows:

[0092] S11: as an input module to receive the original millimeter wave radar point cloud;

[0093] S12: Construct a feature extraction module under the hierarchical feature aggregation mechanism to extract point cloud features;

[0094] S13: Use the neural network noise prediction module based on the MLP structure to predict the noise point cloud;

[0095] S14: Use the KD-Tree based point cloud matching module to match the valid point cloud;

[0096] S15: As an output module, it outputs the radar point cloud after noise is filtered out.

[0097] Example 2

[0098] The embodiment of the present invention provides a processing device for a cross-modal four-dimensional radar supervision method based on a laser radar point cloud, such as Figure 2 As shown, the device includes: a lidar point cloud data reading module S21, a cross-modal true value point cloud mask generation module S22, a millimeter-wave radar denoising network S23 under a hierarchical feature aggregation and prediction matching mechanism, a backbone network and detection head module S24 for processing radar point clouds, a cross-modal mask loss calculation module S25, a millimeter-wave radar point cloud data reading module S26, and an overall loss calculation module S27.

[0099] LiDAR point cloud data reading module S21: This module is divided into training set and test set, and reads the LiDAR point cloud data; Cross-modal truth point cloud mask generation module S22: This module receives the LiDAR point cloud as input, and generates a supervision point mask as the supervision information of the radar point cloud; Millimeter-wave radar denoising network under hierarchical feature aggregation and prediction matching mechanism S23: This module receives the original millimeter-wave radar point cloud as input, extracts point cloud features through hierarchical feature aggregation, and predicts the noisy point cloud based on the point cloud features, so as to obtain the point cloud after filtering out the noise; Backbone network and detection head module S24 for processing radar point cloud: This module processes the point cloud after filtering out the noise The point cloud is used for feature extraction and target detection to obtain the location information of the target; cross-modal mask loss calculation module S25: this module performs loss calculation in the training stage, and performs smoothL1 loss calculation on the predicted point mask and the supervised point mask obtained from the lidar data; millimeter-wave radar point cloud data reading module S26: this module is divided into training set and test set, and reads the millimeter-wave radar point cloud data; overall loss calculation module S27: this module is used to calculate the classification loss, regression loss and effective voxel loss of the classifier calculated based on the predicted mask and the supervised mask, and finally the loss is weighted and summed through the hyperparameters to obtain the final loss result.

[0100] In the processing device of the cross-modal four-dimensional radar supervision method based on laser radar point cloud provided in an embodiment of the present invention, the working process of each module has the same technical characteristics as the cross-modal four-dimensional radar denoising network method based on laser radar supervision. Therefore, the aforementioned functions can also be realized and will not be repeated here.

[0101] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0102] like Figure 4 As shown, the present invention also proposes a cross-modal four-dimensional radar denoising device based on laser radar supervision, which includes:

[0103] The initial module builds a denoising model consisting of a feature encoder, a noise predictor, and a matching module, and obtains a four-dimensional radar point cloud with annotated bounding boxes and true categories as training data;

[0104] The training module comprises a feature encoder that extracts aggregate features of the training data; the noise predictor obtains a predicted point cloud based on the aggregate features, and each position data point in the predicted point cloud has a predicted category; the matching module aligns the predicted point cloud with the training data to obtain a predicted point mask from which noise is removed, and the predicted point mask is input into a classification model to obtain a target area in the training data and its corresponding predicted category; a boundary loss is constructed based on the target area and the bounding box, and a category loss is constructed based on the predicted category and the true category; and the denoising model is trained based on the boundary loss and the category loss;

[0105] The denoising module inputs the four-dimensional radar point cloud to be denoised into the trained denoising model to obtain the four-dimensional radar denoising result point cloud.

[0106] The cross-modal four-dimensional radar denoising device based on laser radar supervision, wherein the training data includes: laser radar point cloud and four-dimensional radar point cloud taken in the same scene;

[0107] The training module includes: aligning the lidar point cloud with the four-dimensional radar point cloud, generating a supervised point mask in the form of voxels for the lidar point cloud, matching the predicted point mask with the supervised point mask to obtain a cross-modal supervision loss; and training the denoising model according to the boundary loss, the category loss and the cross-modal supervision loss.

[0108] The cross-modal four-dimensional radar denoising device based on laser radar supervision, wherein the denoising module includes: analyzing the four-dimensional radar denoising result point cloud through a classification model, obtaining the classification of each point in the four-dimensional radar denoising result point cloud, and based on the classification of each point in the four-dimensional radar denoising result point cloud, controlling the vehicle to complete automatic or assisted driving tasks.

[0109] like Figure 5 As shown, the present invention further proposes a first electronic device A in another embodiment, which includes the above-mentioned cross-modal four-dimensional radar denoising device.

[0110] like Figure 6 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme, the data acquisition device C is used to collect four-dimensional radar point clouds and / or laser point clouds, and the information display device D is used to display the four-dimensional radar denoising result point cloud obtained by the analysis of the present invention.

[0111] The information display device D can sort and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll and play, etc. The user is presented with the key information specified by the user, and the user can understand the information more promptly without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present rich and necessary key information to the user.

[0112] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the cross-modal four-dimensional radar denoising method based on laser radar supervision provided by the above methods.

[0113] The present invention also proposes a storage medium VIII in another embodiment for storing a computer program for executing the cross-modal four-dimensional radar denoising method based on laser radar supervision. It should be understood that the storage medium in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0114] Figure 7 A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same or different from the first electronic device A.

[0115] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. In RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0116] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard, a mouse, etc.; an output unit VII, such as various types of displays, speakers, etc.; a storage medium VIII, such as a disk, an optical disk, etc.; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0117] The computing unit I may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S3. For example, in some embodiments, the method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM II and / or a communication unit IX. When the computer program is loaded into RAM III and executed by the computing unit I, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit I may be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0118] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A cross-modal four-dimensional radar denoising method based on laser radar supervision, characterized in that: include: In the initial step, a denoising model consisting of a feature encoder, a noise predictor, and a matching module is constructed to obtain a four-dimensional radar point cloud with annotated bounding boxes and true categories as training data; In a training step, the feature encoder extracts aggregate features of the training data; The noise predictor obtains a predicted point cloud according to the aggregated features, and each position data point in the predicted point cloud has a predicted category; the matching module aligns the predicted point cloud with the training data to obtain a predicted point mask with noise removed, and inputs the predicted point mask into the classification model to obtain a target area in the training data and its corresponding predicted category; a boundary loss is constructed according to the target area and the bounding box, and a category loss is constructed according to the predicted category and the true category; and the denoising model is trained according to the boundary loss and the category loss; In the denoising step, the four-dimensional radar point cloud to be denoised is input into the trained denoising model to obtain the four-dimensional radar denoising result point cloud.

2. The cross-modal four-dimensional radar denoising method based on laser radar supervision as claimed in claim 1, characterized in that: The training data includes: laser radar point cloud and four-dimensional radar point cloud taken in the same scene; The training step includes: aligning the lidar point cloud with the four-dimensional radar point cloud, generating a supervised point mask in the form of voxels for the lidar point cloud, matching the predicted point mask with the supervised point mask to obtain a cross-modal supervision loss; and training the denoising model according to the boundary loss, the category loss and the cross-modal supervision loss.

3. The cross-modal four-dimensional radar denoising method based on laser radar supervision as claimed in claim 1, characterized in that: The feature encoder applies multiple encoding layers to progressively extract the aggregated features from the point cloud.

4. The cross-modal four-dimensional radar denoising method based on laser radar supervision as claimed in claim 1, characterized in that: The denoising step includes: analyzing the four-dimensional radar denoising result point cloud through a classification model to obtain the classification of each point in the four-dimensional radar denoising result point cloud, and based on the classification of each point in the four-dimensional radar denoising result point cloud, controlling the vehicle to complete the automatic or assisted driving task.

5. A cross-modal four-dimensional radar denoising device based on laser radar supervision, characterized in that: include: The initial module builds a denoising model consisting of a feature encoder, a noise predictor, and a matching module, and obtains a four-dimensional radar point cloud with annotated bounding boxes and true categories as training data; A training module, wherein the feature encoder extracts aggregate features of the training data; The noise predictor obtains a predicted point cloud according to the aggregated features, and each position data point in the predicted point cloud has a predicted category; the matching module aligns the predicted point cloud with the training data to obtain a predicted point mask with noise removed, and inputs the predicted point mask into the classification model to obtain a target area in the training data and its corresponding predicted category; a boundary loss is constructed according to the target area and the bounding box, and a category loss is constructed according to the predicted category and the true category; and the denoising model is trained according to the boundary loss and the category loss; The denoising module inputs the four-dimensional radar point cloud to be denoised into the trained denoising model to obtain the four-dimensional radar denoising result point cloud.

6. The cross-modal four-dimensional radar denoising device based on laser radar supervision as claimed in claim 5, characterized in that: The training data includes: laser radar point cloud and four-dimensional radar point cloud taken in the same scene; The training module includes: aligning the lidar point cloud with the four-dimensional radar point cloud, generating a supervised point mask in the form of voxels for the lidar point cloud, matching the predicted point mask with the supervised point mask to obtain a cross-modal supervision loss; and training the denoising model according to the boundary loss, the category loss and the cross-modal supervision loss.

7. The cross-modal four-dimensional radar denoising device based on laser radar supervision as claimed in claim 5, characterized in that: The denoising module includes: analyzing the four-dimensional radar denoising result point cloud through a classification model to obtain the classification of each point in the four-dimensional radar denoising result point cloud, and controlling the vehicle to complete the automatic or assisted driving task based on the classification of each point in the four-dimensional radar denoising result point cloud.

8. An electronic device, characterized in that: It includes a cross-modal four-dimensional radar denoising device as described in claims 5-7, and the electronic device is connected to an information display device, which is used to display the four-dimensional radar denoising result point cloud according to display parameters and attributes set by the user or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the cross-modal four-dimensional radar denoising method described in any one of claims 1 to 4 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the cross-modal four-dimensional radar denoising method described in any one of claims 1 to 4 are implemented.

Images