Radar data enhancement method and apparatus, and storage medium

Abstract

This application relates to a radar-view data augmentation method, apparatus, and storage medium, applied in the field of radar-view target detection technology. The radar-view data augmentation method includes: acquiring radar-view point cloud data of multiple targets in a monitoring scene; the multiple targets include labeled targets and targets to be labeled; determining the point cloud category to be augmented for the target to be labeled based on the positional association between the target to be labeled and the labeled targets in the monitoring scene; and performing data augmentation on the radar-view point cloud data corresponding to the point cloud category to be augmented for the target to be labeled, based on the radar-view point cloud data of the labeled targets, to generate newly labeled radar-view point cloud data. This application solves the problem of the abnormally slow annotation speed of radar-view data.

Description

Technical Field

[0001] This application relates to the field of radar-based target detection technology, and in particular to a radar-based data enhancement method, apparatus, and storage medium. Background Technology

[0002] Traditional security terminal equipment mainly consists of visible light cameras. However, visible light cameras have low accuracy in identifying targets at long distances, resulting in inconsistent monitoring accuracy for targets within the monitored area. To improve the accuracy of target detection, radar-based area surveillance technology has emerged.

[0003] Combining data from millimeter-wave radar sensors and video sensors to obtain radar-visual data can further improve the probability of target detection and identification, and also obtain information on the target's distance, orientation, and velocity. However, due to the severe lack of publicly available radar-visual data, and the fact that radar-visual data generally requires manual annotation, the annotation speed is slow, necessitating an efficient radar-visual data augmentation method.

[0004] There is currently no effective solution to the problem of slow annotation speed for radar data in related technologies. Summary of the Invention

[0005] This embodiment provides a method, apparatus, and storage medium for enhancing radar and vision data to address the problem of slow annotation speed for radar and vision data in related technologies.

[0006] Firstly, this embodiment provides a method for enhancing radar-based data, the method comprising:

[0007] Based on millimeter-wave radar vision technology, radar vision point cloud data of multiple targets in a monitoring scenario are acquired; the multiple targets include labeled targets and targets to be labeled.

[0008] Based on the positional relationship between the target to be labeled in the monitoring scene and the labeled target, the point cloud category to be enhanced for the target to be labeled is determined;

[0009] Based on the radar-view point cloud data of the labeled targets, data augmentation is performed on the radar-view point cloud data corresponding to the point cloud category to be augmented for the targets to be labeled, generating newly labeled radar-view point cloud data.

[0010] In some embodiments, determining the point cloud category to be enhanced in the target to be labeled based on the positional association between the target to be labeled in the monitoring scene and the labeled target includes:

[0011] Obtain the visual position information of the target to be labeled in the monitoring scenario;

[0012] Based on the visual position information, determine the movement range of the target to be labeled;

[0013] Obtain the first radar-view point cloud sequence of labeled targets located within the movement range;

[0014] Obtain the second radar-view point cloud sequence of the target to be labeled within the movement range;

[0015] Based on the first and second radar-view point cloud sequences, determine the positional association between the labeled targets within the movement range and the target to be labeled; based on the positional association, determine the point cloud category to be enhanced in the target to be labeled.

[0016] In some embodiments, determining the positional association between the labeled targets within the movement range and the target to be labeled, based on the first and second radar view point cloud sequences, and determining the point cloud category to be enhanced in the target to be labeled based on the positional association, includes:

[0017] If the running trajectory corresponding to the first radar point cloud sequence and the running trajectory corresponding to the second radar point cloud sequence have the same trajectory shape and trajectory length, and the trajectory directions intersect, then the positional association relationship is determined to be a parallel relationship.

[0018] When the positional relationship is parallel, the point cloud category to be enhanced in the target to be labeled is determined to be the radial velocity of the point cloud.

[0019] In some embodiments, the step of performing data augmentation on the radar view point cloud data corresponding to the point cloud category to be augmented for the target to be annotated, based on the radar view point cloud data of the already labeled target, to generate newly annotated radar view point cloud data, further includes:

[0020] When the point cloud category to be enhanced for the target to be labeled is the radial velocity of the point cloud, the actual velocity of the target to be labeled in the monitoring scene is determined based on the radial velocity of the first radar-view point cloud sequence of the labeled target and the angle between the running trajectory corresponding to the first radar-view point cloud sequence and the radial ray.

[0021] Based on the actual velocity and the angle between the trajectory corresponding to the second radar-view point cloud sequence and the radial ray, the radial velocity of the point cloud of the target to be labeled is labeled to obtain newly labeled radar-view point cloud data.

[0022] In some embodiments, determining the positional association between the labeled targets within the movement range and the target to be labeled, based on the first and second radar view point cloud sequences, further includes:

[0023] If the overlap between the running trajectory corresponding to the first radar view point cloud sequence and the running trajectory corresponding to the second radar view point cloud sequence exceeds a preset overlap threshold, the positional association relationship is determined to be a similar relationship.

[0024] In some embodiments, the step of performing data augmentation on the radar view point cloud data corresponding to the point cloud category to be augmented for the target to be annotated, based on the radar view point cloud data of the already labeled target, to generate newly annotated radar view point cloud data, includes:

[0025] When the location association relationship is similar, the radar view point cloud data of the already labeled target is determined as the new labeled radar view point cloud data of the target to be labeled.

[0026] In some embodiments, the labeled targets include a first target and a second target; the method further includes:

[0027] If the radar-view point cloud data of the first target includes the radar-view point cloud data of the second target, it is determined that the first target occludes the second target;

[0028] Based on the target outline of the second target, the radar view point cloud data of the second target is updated to obtain the updated radar view point cloud data of the second target;

[0029] Based on the radar-view point cloud data of the first target and the updated radar-view point cloud data of the second target, the radar-view point cloud data of the first target is re-labeled.

[0030] In some embodiments, the method further includes:

[0031] When the first target occludes the second target, the updated radar-view point cloud data of the second target is randomly sampled based on a preset resampling method to obtain the re-labeled radar-view point cloud data of the second target.

[0032] Secondly, this embodiment provides a radar-based data enhancement device, which includes: an acquisition module, a processing module, and an enhancement module;

[0033] The acquisition module is used to acquire point cloud data of multiple targets in a monitoring scenario based on millimeter-wave radar vision technology; the multiple targets include labeled targets and targets to be labeled.

[0034] The processing module is used to determine the point cloud category to be enhanced for the target to be labeled based on the positional relationship between the target to be labeled in the monitoring scene and the labeled target;

[0035] The enhancement module is used to perform data enhancement on the radar view point cloud data corresponding to the point cloud category to be enhanced in the target to be labeled, based on the radar view point cloud data of the labeled target, and generate new labeled radar view point cloud data.

[0036] Thirdly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the radar data enhancement method described in the first aspect above.

[0037] Compared to related technologies, the radar-based point cloud data enhancement method, apparatus, and storage medium provided in this embodiment acquire radar-based point cloud data including multiple targets. Compared to existing lidar technologies, this provides a wider range of radar-based point cloud data, including target distance, relative velocity, and azimuth. The acquired data is more comprehensive. Subsequently, based on the labeled radar-based point cloud data, data enhancement is performed on the radar-based point cloud data of the target to be labeled. Specifically, the corresponding point cloud category to be enhanced is determined through the positional correlation between the two data clouds, thereby improving the labeling efficiency of the target. Finally, based on the different data enhancement principles corresponding to point cloud categories with different positional correlations, data enhancement is performed on the target to be labeled.

[0038] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This is a hardware structure block diagram of the terminal of the radar-visual data enhancement method provided in the embodiments of this application;

[0041] Figure 2 This is a schematic diagram of the product structure of the radar-guided equipment provided in the embodiments of this application;

[0042] Figure 3 This is a flowchart of the radar-visual data enhancement method provided in the embodiments of this application;

[0043] Figure 4 This is a schematic diagram of the distribution of millimeter-wave radar point cloud data provided in the embodiments of this application;

[0044] Figure 5 This is a schematic diagram of the radar-view annotation interface provided in an embodiment of this application;

[0045] Figure 6This is a schematic diagram illustrating the rotation invariance principle of radar point cloud data provided in the embodiments of this application;

[0046] Figure 7 This is a schematic diagram of radial velocity correction in the data augmentation method provided in this specific embodiment;

[0047] Figure 8 This is a schematic diagram illustrating the principle of approximate invariance of equidistant translation of radar data provided in this specific embodiment;

[0048] Figure 9 This is a schematic diagram illustrating the principle of approximate invariance of same-direction translation of radar data provided in this specific embodiment;

[0049] Figure 10 This is a schematic diagram of the occlusion relationship provided in this specific embodiment;

[0050] Figure 11 This is a schematic diagram of the resampling method provided in this specific embodiment;

[0051] Figure 12 This is a flowchart of a radar-based data enhancement method with clear principles and transparent processes, provided in this specific embodiment.

[0052] Reference numerals: 102, processor; 104, memory; 106, transmission device; 108, input / output device. Detailed Implementation

[0053] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0054] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by a person skilled in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, or B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0055] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the radar-based data enhancement method provided in this application embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0056] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the radar data enhancement method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0057] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0058] In the current field of surveillance technology, although both radar sensors and video sensors can detect targets, they are fundamentally two distinct devices. Radar sensors excel at obtaining accurate spatial and motion information about targets, but cannot accurately classify target types. Video sensors, on the other hand, offer high-precision target identification, but struggle to capture target motion and spatial location information. This means video sensors cannot achieve high-precision target identification at all times, due to factors such as sparse tree cover, distance, or poor lighting. Therefore, effectively fusing data from radar and video sensors can yield significantly higher target identification accuracy, motion information, and spatial location information.

[0059] In the transportation sector, multi-sensor data fusion holds great promise. Therefore, radar and video-based surveillance equipment technology is gaining increasing attention in the transportation field.

[0060] The mainstream approach for multi-sensor data fusion is currently deep learning. Multi-sensor deep learning involves three key factors: deep network design, computational power, and data. The generalization ability of a deep learning network model is directly proportional to its size; therefore, when performing radar and vision data fusion, larger and deeper models are preferred. However, this requires increasing the number of parameters, and training large-scale deep models demands greater computational power and more data to support deeper models. Therefore, the acquisition and labeling of training data have a significant impact on the generalization ability of deep learning.

[0061] However, while the scale of deep learning models and image processing units have advanced rapidly, the development of datasets has not been as significant. One important reason is the difficulty in collecting radar and vision data in different scenarios, and the need for manual annotation; for example, professional datasets, including medical image data, require annotation by professionals, resulting in significant manpower and time costs in building public datasets. When applying larger and deeper deep learning network models to real-world tasks, problems such as insufficient training data and imbalanced training data categories are often encountered. Radar and vision devices at the road testing end also face the problem of insufficient data, and there are currently no publicly available datasets for road testing.

[0062] Furthermore, as security gains increasing public attention, traditional security terminal equipment primarily consists of visible light cameras. However, visible light cameras have lower accuracy in identifying targets at greater distances compared to those at closer ranges, meaning the accuracy of target identification within the monitored area is inconsistent (even significantly so). Combining millimeter-wave radar vision with video sensors leverages the strengths of both technologies, further improving target detection and identification probabilities while also providing information on target distance, orientation, and speed. Based on this, radar vision devices, combining radar sensors and video sensors, have emerged. Figure 2 This is a schematic diagram of the product structure of the radar-seeing device provided in the embodiments of this application. Although current radar-seeing devices can acquire target radar-seeing data, there is still a problem that the annotation speed for radar-seeing data is slow, resulting in the data volume of public datasets being insufficient to support the training and application of deep learning network models.

[0063] Therefore, this application provides a clear and transparent method for enhancing radar-view data, which can improve the annotation efficiency of radar-view data acquired by radar-view devices. Figure 3 This is a flowchart of the radar-visual data enhancement method provided in the embodiments of this application, such as... Figure 3 As shown, the process includes the following steps:

[0064] Step S310: Obtain radar point cloud data of multiple targets in the monitoring scenario; multiple targets include labeled targets and targets to be labeled.

[0065] Firstly, before using a radar-guided radar device, it is usually necessary to perform radar calibration. After calibration, inputting any pixel point (u, v) will yield the target's distance and azimuth, i.e., the radar coordinate system point (x, y); conversely, inputting any radar coordinate system point (x, y) will yield the corresponding pixel position (u, v).

[0066] Based on millimeter-wave radar vision technology, measurement information of multiple targets can be acquired in monitoring scenarios, including distance, angle, radial speed, and radar cross section (RCS). A larger RCS indicates that the object is easier to detect. Millimeter-wave radar vision equipment allows for setting the measurement frequency / period as needed; for example, a common measurement period is 0.05 seconds, corresponding to an operating frequency of 20Hz. Millimeter-wave radar vision equipment can be used for real-time detection of micro-moving targets and can obtain key information such as target category, position, and velocity.

[0067] Figure 4 This is a schematic diagram illustrating the distribution of millimeter-wave radar point cloud data provided in an embodiment of this application. (Reference) Figure 4 Different moving targets have different target categories and are represented by different symbols in the diagram, such as different colors and filled triangles and circles. After acquiring point cloud data of multiple targets in a monitoring scene using millimeter-wave radar vision technology, it is necessary to perform radar vision annotation on the targets. Radar vision annotation refers to correctly and completely annotating the information of the same target.

[0068] Figure 5 This is a schematic diagram of the radar-view marking interface provided in an embodiment of this application. (Reference) Figure 5 In this diagram, box ① represents the visual information annotation result acquired by the visual sensor. The point cloud within box ① corresponds to the measurement data of the target generated by the radar-visual sensor, and the corresponding radar-visual point cloud data is ②. This indicates that the annotation result ① and ② refer to the same target, and the real target is this car. Other annotation information, such as boxes ③, ④, and ⑤, are all false target data.

[0069] However, since the annotation boxes in the current radar vision annotation interface are all manually annotated, the annotation efficiency and speed will be greatly reduced when the targets are dense. Therefore, it is impractical to manually annotate a large amount of radar vision point cloud data. Therefore, this embodiment provides a radar vision data enhancement method, which can annotate other targets to be annotated in the monitoring scene based on a small number of already annotated targets, so as to realize intelligent target annotation based on radar vision point cloud data in the monitoring scene.

[0070] Step S320: Based on the positional relationship between the target to be labeled and the labeled target in the monitoring scene, determine the point cloud category to be enhanced for the target to be labeled.

[0071] After acquiring point cloud data of multiple targets in the monitoring scene, it is necessary to obtain the positional relationship between the labeled targets and the targets to be labeled. Taking a road with multiple lanes as an example, the targets to be labeled and the labeled targets may be in the same lane or in the same movement position or trajectory in different lanes. Here, the positional relationship between different targets is not limited.

[0072] The positional relationships between different targets correspond to the relationships between different point cloud data, and their geometric transformation characteristics differ in the monitoring scene / Rayview point cloud interface. Therefore, based on this relationship, we can determine the point cloud category to be enhanced in the target to be labeled, as well as the Rayview point cloud data corresponding to the point cloud category that needs to be synchronized from the labeled targets.

[0073] The point cloud categories corresponding to the Leishi point cloud data include, but are not limited to, point cloud location, point cloud intensity, number of points, and point cloud speed. No specific restrictions are placed on the point cloud categories here.

[0074] Step S330: Based on the radar view point cloud data of the labeled targets, perform data augmentation on the radar view point cloud data corresponding to the point cloud category to be augmented for the targets to be labeled, and generate new labeled radar view point cloud data.

[0075] Once the category of the point cloud to be enhanced for the target to be labeled is determined, the radar view point cloud data of the target to be labeled needs to be enhanced according to the corresponding geometric transformation characteristics and the radar view point cloud data of the labeled target; that is, based on the radar view point cloud data of the labeled target, data enhancement transformations such as rotation and translation are performed on it to obtain the newly labeled radar view point cloud data.

[0076] The above steps, by acquiring radar-based point cloud data including multiple targets, provide a more comprehensive dataset compared to existing LiDAR technologies. This includes more detailed information such as target distance, relative velocity, and azimuth. Subsequently, based on the labeled radar-based point cloud data, data augmentation is performed on the radar-based point cloud data of the target to be labeled. Specifically, the corresponding point cloud category to be augmented is determined by the positional correlation between the two datasets, thereby improving the labeling efficiency of the target. Finally, based on the different data augmentation principles corresponding to point cloud categories with different positional correlations, data augmentation is performed on the target to be labeled.

[0077] Furthermore, the enhanced geometric transformation properties of radar point cloud data include, but are not limited to: perspective and viewing angle problems of visual targets, as well as isometric rotation invariance, isometric translation approximate invariance, and same-direction translation approximate invariance of radar targets.

[0078] Among these, perspective refers to the phenomenon where video objects appear larger when closer and smaller when farther away. Therefore, the radar viewpoint cloud data generated based on video objects must also satisfy the visual principle of objects appearing larger when closer and smaller when farther away. Viewpoint refers to the relationship between the imaging information of video objects and their viewpoint. Therefore, the generated radar viewpoint data must satisfy / cannot significantly violate viewpoint information.

[0079] In some embodiments, based on the positional association between the target to be labeled and the labeled targets in the monitoring scene, the point cloud category to be enhanced in the target to be labeled is determined, including:

[0080] Obtain the visual position information of the target to be labeled in the monitoring scene; determine the movement range of the target to be labeled based on the visual position information; obtain the first radar view point cloud sequence of the labeled targets within the movement range; obtain the second radar view point cloud sequence of the target to be labeled within the movement range;

[0081] Based on the first and second radar-view point cloud sequences, the positional relationships between labeled targets within the movement range and targets to be labeled are determined; based on the positional relationships, the point cloud categories to be enhanced in the targets to be labeled are determined.

[0082] In monitoring scenarios, for example Figure 5 On the road shown, there are multiple vehicles, either moving or parked on the roadside. First, based on radar-guided equipment, it is necessary to acquire full-image data of multiple frames corresponding to the monitored scene. Then, background visual information, such as multiple lanes and green belt settings in the monitored scene, is extracted from the full-image data of a single frame. Simultaneously, visual radar point cloud data B1, B2, ..., Bn corresponding to local targets in the full-image data are extracted. The time series of data composed of visual radar point cloud data is the radar point cloud sequence corresponding to the target.

[0083] After obtaining the full-map data of the current monitoring scene, i.e. the background visual information corresponding to the monitoring scene, it is necessary to calculate the visual position information of multiple targets. This is mainly done by determining the position of the lane location of the target to be labeled in the background visual information, thereby determining the visual position information of the target to be labeled in the monitoring scene and the movement range of the target to be labeled in the monitoring scene.

[0084] For example, after acquiring the locations of multiple lanes and green belts in the full-map data corresponding to the monitoring scene, it is necessary to determine the visual location information of the target to be labeled, that is, the lane position where the target is located, and based on the lane position, determine the current movement range of the target, that is, the range of lanes it can move within. For example, if the current monitoring scene has six lanes in both directions, that is, three lanes in one direction; if the currently labeled target is in the first / third lane, it means that the first radar point cloud sequence of the labeled target at this time is the running trajectory corresponding to the first / third lane.

[0085] When the target to be labeled is in the second lane in the same direction as the already labeled target, it means that the second radar view point cloud sequence of the target to be labeled is the running trajectory corresponding to the second lane.

[0086] Figure 6 This is a schematic diagram illustrating the rotation invariance principle of radar point cloud data provided in the embodiments of this application. (Refer to...) Figure 6 In this diagram, black lines represent lane lines, black text on a yellow background indicates lane numbers, and triangular black markers indicate radar position and orientation. Red dot clouds represent radar dot cloud data corresponding to moving targets (such as moving vehicles in lanes).

[0087] Because the point cloud position and intensity of radar are rotationally invariant, for the same target, measurements taken from different angles at equal intervals will not significantly change its measurement attributes, including the number, position, and intensity of point clouds, or their characteristic distribution will remain consistent. However, the point cloud velocity characteristics corresponding to the target need to be converted in conjunction with the point cloud velocity of the labeled target to ensure consistent characteristic distribution. It is important to avoid moving targets undergoing tangential motion relative to the radar.

[0088] If the trajectory corresponding to the first radar-view point cloud sequence and the trajectory corresponding to the second radar-view point cloud sequence have the same trajectory shape and trajectory length, and the trajectory directions intersect, the positional relationship is determined to be parallel; if the positional relationship is parallel, the point cloud category to be enhanced in the target to be labeled is determined to be the radial velocity of the point cloud.

[0089] Preferably, Figure 7 This is a schematic diagram of radial velocity correction in the data augmentation method provided in this specific embodiment. (Reference) Figure 7 At angles of -30° and 0° in the radar-view point cloud map, there are the running trajectories corresponding to the first and second radar-view point cloud sequences, respectively. At this point, the radial velocity of the target to be labeled in the second radar-view point cloud sequence can be corrected using the labeled target corresponding to the first radar-view point cloud sequence. Since the first and second radar-view point cloud sequences have the same trajectory shape and length, and their trajectories intersect at a point, this intersection point is... Figure 7 The geometric center of the point cloud in the radar vision shown in the figure can be used to determine the positional relationship between the target to be labeled and the labeled target in the actual monitoring scenario. At this time, the equidistant rotation invariance of the radar target in the geometric transformation of the radar vision data can be referenced, and the radial velocity of the point cloud in the target to be labeled can be determined according to the radial velocity of the labeled target.

[0090] Figure 8 This is a schematic diagram illustrating the principle of approximate invariance of equidistant translation of radar data provided in this specific embodiment. (Reference) Figure 8Taking a road as an example in the monitoring scenario, the black lines represent lane markings, the black text with a yellow background indicates the lane number, and the triangular black marker indicates the radar position and orientation. The red, black, blue, and green point clouds are schematic diagrams of the point clouds of moving targets. Different colored point clouds represent tangential translations within the target's movement range. Theoretically, the radar's velocity characteristics are invariant. For the same target, such as a car / standard sphere, the number and intensity of point clouds will not change significantly during tangential translation, or their characteristic distribution will remain consistent. However, the point cloud velocity characteristics need to be transformed, and the transformed characteristic distribution must be consistent with the characteristic distribution of the labeled target. However, care must be taken to avoid excessive vehicle translation, otherwise the approximate invariance will not be satisfied.

[0091] Specifically, based on the radar-view point cloud data of the labeled targets, data augmentation is performed on the radar-view point cloud data corresponding to the point cloud category to be augmented for the target to be labeled, generating newly labeled radar-view point cloud data. This includes: when the point cloud category to be augmented for the target to be labeled is the radial velocity of the point cloud, determining the actual velocity of the target to be labeled in the monitoring scene based on the radial velocity of the first radar-view point cloud sequence of the labeled targets and the angle between the running trajectory corresponding to the first radar-view point cloud sequence and the radial ray; and labeling the radial velocity of the point cloud of the target to be labeled based on the actual velocity and the angle between the running trajectory corresponding to the second radar-view point cloud sequence and the radial ray, thus obtaining newly labeled radar-view point cloud data.

[0092] like Figure 7 As shown, the green line represents the trajectory of the labeled target, and the blue line represents the trajectory of the target to be labeled; V represents the actual speed of the labeled target and the target to be labeled in the actual monitoring scenario; v0 r v1 represents the radial velocity of the original green target trajectory (i.e., the trajectory of the labeled target). r This represents the radial velocity of the target trajectory (i.e., the trajectory of the target to be labeled) after the blue area has been translated. This indicates the angle between the trajectory of the labeled target and the radial ray. This represents the angle between the trajectory of the target to be labeled, which is parallel to the labeled target, and the radial ray.

[0093] Therefore, the radial velocity of the target to be labeled can be determined by the radial velocity and angle of the labeled target, which can be expressed by the formula:

[0094] ;

[0095] ;

[0096] Using the above formula, when the motion trajectories of the labeled target and the target to be labeled are parallel, the radial velocity of the target to be labeled can be calculated, which is the newly labeled radar point cloud data, and then the target to be labeled can be subjected to data augmentation labeling processing.

[0097] Because different labeled targets have different positional relationships with the target to be labeled, the corresponding positional relationships also differ. Positional relationships include, but are not limited to, parallel relationships, similar relationships, and occlusion relationships.

[0098] In some embodiments, when the positional association is similar, determining the positional association between the labeled target and the target to be labeled within the movement range based on the first and second radar view point cloud sequences further includes: if the overlap between the running trajectory corresponding to the first and second radar view point cloud sequences exceeds a preset overlap threshold, the positional association is determined to be similar. When the positional association is similar, the radar view point cloud data of the labeled target is determined as the new labeled radar view point cloud data of the target to be labeled.

[0099] When the position management relationship between labeled targets and unlabeled targets is similar, the principle of approximate invariance of same-direction translation in radar data can be used to perform data augmentation processing on the radar view point cloud data of the unlabeled targets.

[0100] Figure 9 This is a schematic diagram illustrating the principle of approximate invariance of same-direction translation of radar data provided in this specific embodiment. (Reference) Figure 9 The black lines represent lane markings, and the black text with a yellow background between adjacent lane lines indicates the lane number. The triangular black markers indicate the radar position and orientation. The red, green, and black point clouds are schematic diagrams of moving targets with similar trajectories. Different colored point clouds represent radial translations within a certain range. For a target, such as a car or a standard sphere, the number and velocity of its point clouds do not change significantly during radial translation, or their characteristic distribution remains consistent. However, the point cloud intensity characteristics change slightly. Care should be taken to avoid excessive vehicle translation, otherwise the approximate invariance of translation in the same direction will not be satisfied.

[0101] Will Figure 9The red, green, and black radar point cloud data are used as radar point cloud data for red / green / black moving targets in the monitoring scenario, respectively. The targets corresponding to these radar point cloud data exhibit similar trajectories, spatial locations, target types, and viewing angles, thus establishing a similarity relationship in their positions. When the red moving target is already labeled and the green moving target is yet to be labeled, the red radar point cloud data corresponding to the same location but different time points can be interchanged with the green radar point cloud data. This enhances the point cloud intensity features in the green radar point cloud data, resulting in newly labeled radar point cloud data for the target to be labeled, i.e., the green moving target.

[0102] For example, if the green moving target can reach the position of the red moving target at time t+1, and the radial velocities of the two moving targets are the same, then the radar point cloud data corresponding to the red moving target at time t can be used as the newly labeled radar point cloud data corresponding to the green moving target at time t+1.

[0103] In some embodiments, the labeled targets include a first target and a second target; the method further includes: determining that the first target occludes the second target if the radar-view point cloud data of the first target includes the radar-view point cloud data of the second target; updating the radar-view point cloud data of the second target based on the target outline of the second target to obtain updated radar-view point cloud data of the second target; and relabeling the radar-view point cloud data of the first target based on the radar-view point cloud data of the first target and the updated radar-view point cloud data of the second target.

[0104] When the first target occludes the second target, the radar view point cloud data of the updated second target is randomly sampled based on a preset resampling method to obtain the re-annotated radar view point cloud data of the second target.

[0105] In the data augmentation processing of Rayvision point cloud data, occlusion handling is a crucial aspect that must be considered and is highly susceptible to errors. For example, in traffic scenarios, where traffic flow is heavy, occlusion is a very common phenomenon. Figure 10 This is a schematic diagram of the occlusion relationship provided in this specific embodiment. (Refer to...) Figure 10 According to the target bounding box (red box), vehicles A and B are labeled. At this point, vehicle A is the first target, and vehicle B is the second target. Because vehicle B is occluded by vehicle A in the image, visual and radar point cloud information for vehicle B is absent from the visual labeling results. However, in reality, a small amount of visual and radar point cloud information for vehicle B can still be detected. Therefore, for the occluded target, its labeling needs to be modified to an outer contour label, and the radar point cloud data of vehicle B needs to be resampled to obtain the relabeled radar point cloud data for vehicle B.

[0106] For car A that obscures car B, the radar-view point cloud data of the first target in the annotation box of car A needs to be updated based on the radar-view point cloud data of the second target in the annotation box of car B. Specifically, the radar-view point cloud data of car A is updated by subtracting the radar-view point cloud data of the second target from the radar-view point cloud data of the first target in both annotation boxes.

[0107] Furthermore, radar point clouds have relatively weak visualization and information comprehension capabilities. Therefore, based on detection principles, such as point cloud number fluctuation models including range perturbation models, velocity perturbation models, angle perturbation models, and intensity perturbation models, the labeled radar point cloud data can be resampled to achieve enhanced processing of the radar point cloud data.

[0108] Figure 11 This is a schematic diagram of the resampling method provided in this specific embodiment. (Reference) Figure 11 Here, a 1D diagram is used to represent the resampling of spatial location. Black markers 1, 2, and 3 on a green background represent the probability distribution of the spatial locations of the target's three radar points. The red dashed line represents its probability model. At this point, the target's radar point cloud data needs to be resampled, specifically by normalizing the area enclosed to (0, 1) and resampling. After resampling, the number and location of points will change. For example, the black markers 1 and 2 on a blue background after resampling represent the probability distribution of the spatial locations of the target's three radar points. Specifically, random resampling is performed using a Gaussian distribution. The resampling process for velocity and intensity is the same as for spatial location.

[0109] The present embodiment will be described and explained below through specific examples.

[0110] Figure 12 This is a flowchart of a radar-based data enhancement method with clear principles and transparent processes, provided in this specific embodiment. For example... Figure 12 As shown, taking radar-visual data in vehicle roads as an example, this radar-visual data enhancement method includes the following steps:

[0111] Step 1, Equipment Setup. The equipment should be installed as directly as possible.

[0112] Step 2, parameter configuration. Configuration includes correctly configuring lane lines, marking green belts, and other key information.

[0113] Step 3, Calibration.

[0114] Step 4: Collect data.

[0115] After running for a period of time, the radar-guided equipment collects a certain amount of radar-guided data.

[0116] Step 5, Background Extraction.

[0117] Step 6, manual annotation. Each frame of radar video is manually annotated with the targets and their corresponding relationships.

[0118] Step 7: Extract visual target data.

[0119] Step 8: Extract radar target data.

[0120] Step 9, Data Augmentation: Translation.

[0121] Step 10, Data Augmentation: Exchange.

[0122] Step 11, Data Augmentation: Occlusion.

[0123] Step 12, Data Augmentation: Resampling.

[0124] Step 13, Data Saving. Save the original manually labeled data and the augmented data separately.

[0125] Step 14, Manual Review. Manually review the augmented data to check for significant anomalies. If any are found, discard them. Only retain data without significant anomalies and proceed to Step 15.

[0126] Step 15, End.

[0127] The specific methods in the above steps can be referred to the specific methods in the above embodiments, and will not be described in detail here.

[0128] The above solution can effectively address the problem of insufficient radar vision data due to the diversity of radar vision data, and improve the annotation speed of radar vision data.

[0129] It should be noted that the steps shown in the above flowchart or the flowchart in the accompanying figures can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here. For example, the steps of different data augmentation methods can be substituted for each other.

[0130] This embodiment also provides a radar-based data enhancement device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," and "subunit," etc., used below refer to combinations of software and / or hardware that perform a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0131] The radar-based data enhancement device includes an acquisition module, a processing module, and an enhancement module;

[0132] The acquisition module is used to acquire point cloud data of multiple targets in a monitoring scenario; the multiple targets include labeled targets and targets to be labeled.

[0133] The processing module is used to determine the point cloud category to be enhanced for the target to be labeled based on the positional relationship between the target to be labeled and the labeled target in the monitoring scene.

[0134] The enhancement module is used to perform data enhancement on the point cloud data of the target to be enhanced, based on the point cloud data of the labeled target, and generate new labeled point cloud data.

[0135] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0136] This embodiment also provides an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0137] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0138] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0139] S1, acquires radar point cloud data of multiple targets in the monitoring scenario; multiple targets include labeled targets and targets to be labeled;

[0140] S2, based on the positional relationship between the target to be labeled and the labeled target in the monitoring scene, determine the point cloud category to be enhanced for the target to be labeled;

[0141] S3, based on the radar view point cloud data of the labeled targets, performs data augmentation on the radar view point cloud data corresponding to the point cloud category to be augmented for the targets to be labeled, and generates new labeled radar view point cloud data.

[0142] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0143] Furthermore, in conjunction with the radar-based data enhancement methods provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the radar-based data enhancement methods described in the above embodiments.

[0144] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0145] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0146] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0147] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for radar data enhancement, characterized in that, The method includes: Acquire point cloud data of multiple targets in a monitoring scenario; the multiple targets include labeled targets and targets to be labeled. Based on the positional relationship between the target to be labeled in the monitoring scene and the labeled target, the point cloud category to be enhanced for the target to be labeled is determined; Based on the radar-view point cloud data of the labeled targets, data augmentation is performed on the radar-view point cloud data corresponding to the point cloud category to be augmented for the targets to be labeled, generating newly labeled radar-view point cloud data.

2. The method of claim 1, wherein, The step of determining the point cloud category to be enhanced for the target to be labeled based on the positional association between the target to be labeled in the monitoring scene and the labeled targets includes: Obtain the visual position information of the target to be labeled in the monitoring scenario; Based on the visual position information, determine the movement range of the target to be labeled; Obtain a first radar-view point cloud sequence of labeled targets within the movement range; obtain a second radar-view point cloud sequence of the target to be labeled within the movement range; Based on the first and second radar-view point cloud sequences, determine the positional association between the labeled targets within the movement range and the target to be labeled; based on the positional association, determine the point cloud category to be enhanced in the target to be labeled.

3. The method of claim 2, wherein, The step involves determining the positional association between the labeled targets within the movement range and the target to be labeled, based on the first and second radar view point cloud sequences. Based on the location association, the point cloud category to be enhanced in the target to be labeled is determined, including: If the running trajectory corresponding to the first radar point cloud sequence and the running trajectory corresponding to the second radar point cloud sequence have the same trajectory shape and trajectory length, and the trajectory directions intersect, then the positional association relationship is determined to be a parallel relationship. When the positional relationship is parallel, the point cloud category to be enhanced in the target to be labeled is determined to be the radial velocity of the point cloud.

4. The radar-visual data enhancement method according to claim 3, characterized in that, The step of performing data augmentation on the radar view point cloud data corresponding to the point cloud category to be augmented for the target to be annotated, based on the already labeled target's radar view point cloud data, to generate newly labeled radar view point cloud data, further includes: When the point cloud category to be enhanced for the target to be labeled is the radial velocity of the point cloud, the actual velocity of the target to be labeled in the monitoring scene is determined based on the radial velocity of the first radar-view point cloud sequence of the labeled target and the angle between the running trajectory corresponding to the first radar-view point cloud sequence and the radial ray. Based on the actual velocity and the angle between the trajectory corresponding to the second radar-view point cloud sequence and the radial ray, the radial velocity of the point cloud of the target to be labeled is labeled to obtain newly labeled radar-view point cloud data.

5. The radar-visual data enhancement method according to claim 2, characterized in that, The step of determining the positional association between the labeled targets within the movement range and the target to be labeled based on the first radar view point cloud sequence and the second radar view point cloud sequence further includes: If the overlap between the running trajectory corresponding to the first radar view point cloud sequence and the running trajectory corresponding to the second radar view point cloud sequence exceeds a preset overlap threshold, the positional association relationship is determined to be a similar relationship.

6. The radar-visual data enhancement method according to claim 5, characterized in that, The process of augmenting the radar view point cloud data corresponding to the point cloud category to be augmented for the target to be annotated, based on the already labeled target's radar view point cloud data, to generate newly labeled radar view point cloud data includes: When the location association relationship is similar, the radar view point cloud data of the already labeled target is determined as the new labeled radar view point cloud data of the target to be labeled.

7. The radar-visual data enhancement method according to claim 2, characterized in that, The labeled targets include a first target and a second target; the method further includes: If the radar-view point cloud data of the first target includes the radar-view point cloud data of the second target, it is determined that the first target occludes the second target; Based on the target outline of the second target, the radar view point cloud data of the second target is updated to obtain the updated radar view point cloud data of the second target; Based on the radar-view point cloud data of the first target and the updated radar-view point cloud data of the second target, the radar-view point cloud data of the first target is re-labeled.

8. The radar-visual data enhancement method according to claim 7, characterized in that, The method further includes: When the first target occludes the second target, the updated radar-view point cloud data of the second target is randomly sampled based on a preset resampling method to obtain the re-labeled radar-view point cloud data of the second target.

9. A radar-based data enhancement device, characterized in that, The device includes: an acquisition module, a processing module, and an enhancement module; The acquisition module is used to acquire point cloud data of multiple targets in a monitoring scenario; the multiple targets include labeled targets and targets to be labeled. The processing module is used to determine the point cloud category to be enhanced for the target to be labeled based on the positional relationship between the target to be labeled in the monitoring scene and the labeled target; The enhancement module is used to perform data enhancement on the radar view point cloud data corresponding to the point cloud category to be enhanced in the target to be labeled, based on the radar view point cloud data of the labeled target, and generate new labeled radar view point cloud data.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the Ravis data enhancement method according to any one of claims 1 to 8.

Images