Radar point cloud data enhancement method and device for embodied intelligence

Abstract

Embodiments of the present application relate to the field of artificial intelligence, and provide a radar point cloud data enhancement method and device for embodied intelligence, comprising: generating a synthetic semantic scene through a multi-stage pyramid discrete diffusion model from a three-dimensional semantic scene to be enhanced; performing hard filtering on the synthetic semantic scene to obtain a target synthetic scene; performing multi-bounce ray tracing sparsification on the target synthetic scene to obtain a synthetic sparse point cloud paired with the target synthetic scene; performing object pasting enhancement on the synthetic sparse point cloud and the target synthetic scene based on a rare class object library to obtain a synthetic training pair; and using the synthetic training pair to train a three-dimensional semantic scene completion model. Embodiments of the present application can construct a high-quality synthetic training pair with rational structure, consistent semantics, sensor physical reality and balanced long-tail categories, solve the problems of insufficient real labeled data and unbalanced class distribution, and improve the training effect and environmental perception ability of the three-dimensional semantic scene completion model in complex scenes.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to a method and apparatus for enhancing radar point cloud data for embodied intelligence. Background Technology

[0002] With the development of machine vision, 3D perception, and robotics, embodied intelligent systems such as autonomous driving, service robots, and inspection robots are gradually moving towards dynamic and complex real-world application scenarios. These systems typically rely on sensors such as LiDAR and depth cameras to acquire environmental information to perform functions such as navigation and obstacle avoidance, path planning, and task decision-making. Radar point cloud data, because it can directly reflect spatial geometry and adapt to complex lighting conditions, has become an important source of perception data for embodied intelligent systems. In semantic scene completion tasks, it is necessary to recover a more complete spatial structure and semantic attributes based on sparse observations from a limited perspective, providing support for subsequent perception, interaction, and decision-making. Therefore, how to acquire sufficient, stable, and model-trainable radar point cloud data has become one of the urgent technical problems to be solved in the development of embodied intelligent perception technology. Summary of the Invention

[0003] This application provides a radar point cloud data augmentation method and apparatus for embodied intelligence, which can solve the technical problems of insufficient real labeled data, unbalanced category distribution, and insufficient task adaptability of existing augmentation methods in related technologies, improve the training effect of 3D semantic scene completion model and the environmental perception capability of embodied intelligence system in complex scenes.

[0004] In a first aspect, embodiments of this application provide a radar point cloud data augmentation method for embodied intelligence. This method is applied to an embodied intelligence system and includes: The three-dimensional semantic scene to be enhanced is generated in multiple stages using a pyramid discrete diffusion model to obtain a synthetic semantic scene; the pyramid discrete diffusion model generates the synthetic semantic scene sequentially according to a resolution hierarchy from coarse to fine. The synthesized semantic scene is subjected to multi-level hard filtering and probability distribution divergence joint screening to obtain a target synthesized scene that meets the quality constraints; Multi-echo ray tracing sparsification is performed on the target synthetic scene to obtain a synthetic sparse point cloud paired with the target synthetic scene; Based on a rare class object library, object pasting enhancement is performed on the synthesized sparse point cloud and the corresponding scene voxel ground value in the target synthesized scene to obtain a synthesis training pair composed of the enhanced synthesized sparse point cloud and the scene voxel ground value; the synthesis training pair is used to train a 3D semantic scene completion model.

[0005] Secondly, embodiments of this application provide a radar point cloud data enhancement device for embodied intelligence, which has the function of implementing the radar point cloud data enhancement method for embodied intelligence provided in the first aspect above. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above function, and the modules can be software and / or hardware. In one embodiment, the device is applied to an embodied intelligence system, and the device includes: The generation module is configured to generate a synthetic semantic scene from the 3D semantic scene to be enhanced through a multi-stage process using a pyramid discrete diffusion model; the pyramid discrete diffusion model generates the synthetic semantic scene sequentially according to a resolution hierarchy from coarse to fine. The filtering module is configured to perform multi-level hard filtering and probability distribution divergence joint filtering on the synthetic semantic scene to obtain a target synthetic scene that meets the quality constraints. The ray tracing module is configured to perform multi-echo ray tracing sparsification on the target synthetic scene to obtain a synthetic sparse point cloud paired with the target synthetic scene; The enhancement module is configured to perform object pasting enhancement on the synthetic sparse point cloud and the corresponding scene voxel ground truth values ​​in the target synthetic scene based on a rare class object library, so as to obtain a synthetic training pair composed of the enhanced synthetic sparse point cloud and the scene voxel ground truth values; the synthetic training pair is used to train a 3D semantic scene completion model.

[0006] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the radar point cloud data augmentation method for embodied intelligence as described in the first aspect.

[0007] Fourthly, embodiments of this application provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the radar point cloud data augmentation method for embodied intelligence as described in the first aspect.

[0008] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to execute the radar point cloud data enhancement method for embodied intelligence provided in the first aspect.

[0009] Compared to existing technologies, this application embodiment employs a data augmentation scheme combining multi-stage pyramid discrete diffusion generation, multi-level quality screening, multi-echo ray tracing sparsification, and rare class object library pasting enhancement to construct synthetic training pairs that are strictly paired with the ground truth of dense scenes. Since this application embodiment is designed for embodied intelligent perception tasks by collaboratively enhancing synthetic semantic scenes, synthetic sparse point clouds, and long-tail categories, rather than simply using geometric transformation enhancement, simple copy-paste enhancement, or merely generating sparse point cloud data as in existing technologies, this application embodiment can improve data scale while maintaining scene structure rationality, semantic consistency, sensor physical realism, and class distribution balance. Therefore, the enhanced data obtained in this application embodiment has a higher degree of matching with actual task requirements. Because the generated synthetic sparse point cloud maintains a strict pairing with the scene voxel ground truth, and the enhanced long-tail category coverage is more comprehensive, the synthetic training pairs obtained in this application embodiment can improve the environmental perception performance and long-tail target recognition capability of the 3D semantic scene completion model in complex scenes. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the radar point cloud data augmentation method for embodied intelligence in an embodiment of this application. Figure 2 This is a block diagram of the enhancement pipeline for the radar point cloud data enhancement method for embodied intelligence in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the principle of the pyramid discrete diffusion model in the embodiments of this application; Figure 4 This is a schematic diagram illustrating the principle of multi-echo ray tracing sparsity in an embodiment of this application; Figure 5 This is a schematic diagram showing the actual operation results of the data enhancement pipeline in the embodiments of this application; Figure 6 This is a schematic diagram of the radar point cloud data enhancement device for embodied intelligence according to an embodiment of this application. Detailed Implementation

[0011] Currently, how to acquire radar point cloud data that is large enough, of stable quality, and suitable for model training has become one of the urgent technical problems to be solved in the development of embodied intelligent perception technology.

[0012] In related technologies, the training of 3D semantic scene completion models typically relies on a large amount of data consisting of pairing sparse point cloud observations with dense semantic ground truths, but obtaining such data is difficult. For example, dense 3D semantic annotation requires voxel-by-voxel labeling, which is costly in terms of manpower and time. Furthermore, in real-world data, targets such as pedestrians, cyclists, and traffic signs appear far less frequently than common targets such as roads and buildings, exhibiting a long-tail distribution, which leads to insufficient model recognition capabilities for rare categories.

[0013] To address the aforementioned technical challenges, various data augmentation schemes have been employed. For example, geometric augmentation only transforms point clouds through flipping and rotation, without introducing new semantic content, offering limited improvement for rare categories. Copy-and-paste augmentation, on the other hand, uses existing real-world data, failing to generate entirely new scene geometry, and often resulting in stitching artifacts at the pasted boundaries. Generative model-based scene synthesis methods typically lack tightly matched dense semantic ground truths and offer limited constraints on the quality and semantic distribution of the synthesized scene. Furthermore, while some scene generation methods can output complete semantic scenes, they fail to provide paired sparse observations consistent with real sensor observation characteristics or manage the distribution of long-tail categories. Therefore, a novel technical solution is urgently needed for data augmentation of 3D semantic scene completion models.

[0014] This application provides a radar point cloud data augmentation method for embodied intelligence, which can be executed by a radar point cloud data augmentation device. This device can be an application program implementing the corresponding function, or a server or terminal device with the corresponding application installed.

[0015] It should be noted that the server involved in the embodiments of this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0016] The terminal devices involved in the embodiments of this application can be devices that provide voice and / or data connectivity to users, handheld devices with wireless connectivity, or other processing devices connected to a wireless modem. They can also be vehicle-mounted or airborne computing units deployed on intelligent platforms such as autonomous vehicles, service robots, inspection robots, or warehouse autonomous mobile robots. For example, mobile phones or computers with mobile terminals can be portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile devices that exchange voice and / or data with a wireless access network.

[0017] The radar point cloud data enhancement method for embodied intelligence provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0018] Reference Figure 1 , Figure 1 This is a flowchart illustrating a radar point cloud data augmentation method for embodied intelligence, provided as an embodiment of this application. The method can be executed by a radar point cloud data augmentation device and is used in an embodied intelligence system. The method includes steps 101-104: Step 101: The 3D semantic scene to be enhanced is generated in multiple stages using a pyramid discrete diffusion model to obtain a synthetic semantic scene. The pyramid discrete diffusion model generates the synthetic semantic scene sequentially according to a resolution hierarchy from coarse to fine.

[0019] Step 102: Perform multi-level hard filtering and probability distribution divergence joint screening on the synthetic semantic scene to obtain the target synthetic scene that meets the quality constraints.

[0020] Step 103: Perform multi-echo ray tracing sparsification on the target composite scene to obtain a composite sparse point cloud paired with the target composite scene.

[0021] Step 104: Based on the rare class object library, object pasting enhancement is performed on the synthesized sparse point cloud and the corresponding scene voxel ground truth values ​​in the target synthesized scene to obtain a synthesis training pair consisting of the enhanced synthesized sparse point cloud and the scene voxel ground truth values. The synthesis training pair is used to train the 3D semantic scene completion model.

[0022] Figure 2 An enhancement pipeline diagram for a radar point cloud data enhancement method for embodied intelligence, provided in an embodiment of this application. Figure 2 The diagram illustrates four interconnected processing stages corresponding to steps 101-104. Step 101 corresponds to the pyramid discrete diffusion scene generation stage, used to generate a complete synthetic semantic scene; Step 102 corresponds to the synthetic sample quality filtering stage, used to perform multi-level hard filtering and probability distribution divergence joint screening on the synthetic semantic scene generated in Step 101 to obtain a target synthetic scene that meets quality constraints; Step 103 corresponds to the multi-echo ray tracing sparsification stage, used to virtually scan the target synthetic scene based on the hardware physical characteristics of the target radar sensor to generate a paired synthetic sparse point cloud; Step 104 corresponds to the rare class object library pasting enhancement stage, used to synchronously enhance the synthetic sparse point cloud and the corresponding scene voxel ground truth based on the rare class object library, thereby obtaining a synthetic training pair. Figure 2As can be seen, the embodiments of this application form a complete data augmentation pipeline by sequentially connecting the above four stages, from generating complete semantic scenes, screening sample quality, simulating sparse observations to enhancing long-tail categories. The generated synthetic training pairs can be merged with the real training set to train the three-dimensional semantic scene completion model.

[0023] Specifically, Figure 2 The system comprises four interconnected functional modules: Subsystem A: Pyramid Discrete Diffusion. This subsystem is used to generate a complete 3D semantic scene. It adopts a multi-stage generation method from coarse to fine, including stages S1, S2, and S3. Stage S1 corresponds to a lower resolution of 32×32×4 and is used to generate a coarse-grained semantic scene; stage S2 corresponds to an intermediate resolution of 64×64×8 and is used to further refine the coarse-grained scene; stage S3 corresponds to a target resolution of 256×256×32 and is used to output a complete high-resolution 3D semantic scene.

[0024] Subsystem B: Ray Tracing Sparsity. This subsystem converts the complete semantic scene generated by Subsystem A into a sparse input that matches the form of real radar observations. The figure shows its use of 64 beams, 2048 azimuth angles, and Bresenham 3D ray tracing, combined with mechanisms such as effective range and first-hit stop, to output sparse sensor input. .

[0025] Subsystem C: Rare Class Object Library. This subsystem is used to enhance long-tail category targets. The diagram shows that it constructs a rare class object library through a 26-connectivity component extraction method, covering 10 rare semantic categories, and performs object pasting completion through ground anchoring, thereby providing a complete semantic scene. Strengthen the long-tail categories in the data.

[0026] Synthetic training is applied to the output module. After the above processing, the system output consists of sparse input. and corresponding complete semantic truth value The constructed synthetic training pairs ( , This synthetic training pair can be merged with the real training set for subsequent SSC model training.

[0027] In terms of results, the three subsystems mentioned above can be run offline once to generate synthetic training pairs of different sizes, such as 10,000, 31,000, and 57,000 sets of samples, and can be used to improve the performance of downstream 3D semantic scene completion models. The downstream S²D² model achieved a 38.54% mIoU on the SemanticKITTI validation set under a 31K configuration (+2.37% compared to the baseline without synthetic data).

[0028] Steps 101-104 will be explained in detail below with reference to the accompanying drawings.

[0029] Step 101: The 3D semantic scene to be enhanced is generated in multiple stages using a pyramid discrete diffusion model to obtain a synthetic semantic scene. The pyramid discrete diffusion model generates the synthetic semantic scene sequentially according to a resolution hierarchy from coarse to fine.

[0030] In this embodiment, the 3D semantic scene to be enhanced can be an original ground truth voxel scene collected from a real dataset, or a pre-defined initial scene representation. The pyramid discrete diffusion model includes multiple stages with progressively increasing resolution levels, and conditional features are transferred between adjacent stages through a scale-adaptive fusion module.

[0031] The pyramid discrete diffusion model refers to a generative model that generates and refines the semantics of a 3D voxel scene in a hierarchical manner, from low resolution to high resolution. This model does not directly perform semantic prediction of all voxels on the target high-resolution 3D mesh all at once. Instead, it first generates the overall layout and global semantic structure of the scene at a lower resolution, and then gradually supplements local details and boundary information at higher resolutions. This reduces computational overhead and memory usage while improving the structural stability and semantic consistency of the generated scene. For example, the pyramid discrete diffusion model may include three stages with progressively increasing resolutions, such as 32×32×4 voxels, 64×64×8 voxels, and 256×256×32 voxels, with conditional features transferred between adjacent stages through a scale-adaptive fusion module.

[0032] Synthetic semantic scenes refer to voxel-based scene representations generated using a pyramid discrete diffusion model, possessing complete 3D spatial occupancy information and semantic category annotations. A synthetic semantic scene includes the positional distribution, occupancy state, and corresponding semantic category information of each voxel in the 3D semantic scene, such as categories like road surface, sidewalk, building, vegetation, vehicle, pedestrian, cyclist, traffic sign, and utility pole. Compared to point cloud data containing only sparse scan points, synthetic semantic scenes are a dense 3D semantic voxel representation, serving as the basis for subsequent generation of paired sparse point clouds and construction of ground truth training samples.

[0033] As an optional embodiment, in step 101, in the initial stage, multi-step discrete polynomial diffusion is performed starting from the full mask tensor under the lowest resolution 3D mesh, gradually denoising each voxel from the mask state to a semantic category, generating a coarse-grained semantic scene. This stage is completed at the lowest resolution and is mainly used to determine the overall layout, main structure, and global semantic distribution of the scene.

[0034] In each subsequent stage, the coarse-grained semantic scene generated in the previous stage is upsampled and feature-mapped through a scale-adaptive fusion module. This feature is then injected as a conditional feature into the medium-resolution 3D mesh corresponding to the current stage's resolution. Discrete multinomial diffusion is then performed on the full mask tensor of the current resolution to generate the medium-resolution semantic scene at the current resolution. In this way, subsequent stages can perform detail completion and semantic refinement under the constraints of the global structure determined in the previous stage, thus maintaining the consistency of the scene structure while progressively increasing the resolution.

[0035] At the highest resolution stage, the medium-resolution semantic scene generated in the previous stage is upsampled by the scale-adaptive fusion module and divided into multiple spatially overlapping sub-scene blocks. Discrete polynomial diffusion generation is performed independently on each sub-scene block, and the voxel prediction values ​​of adjacent sub-scene blocks in the overlapping region are averaged and fused to obtain the synthetic semantic scene at the target resolution. This avoids structural breaks at the sub-block splicing points while controlling computational overhead.

[0036] Specifically, assuming the lowest resolution 3D mesh used in the initial stage is 32×32×4 voxels. In this stage, the discrete multinomial diffusion network starts from the full mask tensor and, through a preset number of steps of progressive denoising, restores each voxel from its undetermined mask state to a specific semantic category, thereby generating a coarse-grained semantic scene. This coarse-grained semantic scene is mainly used to determine the overall spatial layout, basic structure distribution, and global semantic relationships within the scene, such as the approximate distribution of roads, buildings, vegetation, and target objects.

[0037] Subsequently, assuming the medium-resolution 3D mesh used in the next stage is 64×64×8 voxels, the coarse-grained semantic scene generated in the previous stage is upsampled through a scale-adaptive fusion module. The upsampled result is then subjected to learnable feature mapping, which is injected as conditional features into the discrete multinomial diffusion network corresponding to the current stage's resolution. Simultaneously, progressive denoising is performed using the full-mask tensor at the current resolution to generate a medium-resolution semantic scene with richer local details and clearer semantic boundaries. In this way, the current stage can progressively refine the local structure and semantic details within the constraints of the global structure determined in the previous stage, thereby improving resolution while maintaining the consistency of the overall scene structure.

[0038] Further optionally, the scale-adaptive fusion module is used to upsample the semantic scene generated in the previous resolution stage between adjacent resolution stages, and to use the upsampled features as the conditional features of the discrete multinomial diffusion denoising network in the next resolution stage in a learnable manner, so that the next resolution stage is refined under the global structural constraints of the previous resolution stage.

[0039] In the highest resolution stage, the target resolution 3D mesh is assumed to be 256×256×32 voxels. Since directly performing discrete diffusion generation on the entire 3D scene at this resolution would result in high memory usage and computational overhead, the medium-resolution semantic scene generated in the previous stage is first upsampled to the target resolution using a scale-adaptive fusion module. Then, it is divided into multiple spatially overlapping sub-scene blocks, for example, 4×4 horizontally overlapping sub-scene blocks. Each sub-scene block can be 64×64×32 voxels, and a preset width overlap region, such as 8 voxels, is set between adjacent sub-scene blocks. Subsequently, discrete polynomial diffusion generation is performed independently on each sub-scene block, and the voxel prediction results of adjacent sub-scene blocks within the overlap region are averaged and fused to finally obtain the complete synthetic semantic scene at the target resolution.

[0040] The above embodiments not only reduce the consumption of computing resources in the high-resolution scene generation process, but also reduce the structural discontinuity problem at the splicing boundary of sub-scene blocks, thereby improving the overall coherence and local detail realism of the generated synthetic semantic scene.

[0041] For example, Figure 3 The three-stage generation details of the pyramid discrete diffusion model in the embodiments of this application are shown. Figure 3 Stages S1, S2, and S3 correspond to three generation stages from low resolution to high resolution. In stage S1, the resolution is 32×32×4. It initializes from a full-mask tensor and performs 100 steps of discrete polynomial diffusion to output a coarse-grained semantic scene. This stage primarily determines the overall spatial layout and global semantic structure of the scene with relatively low computational complexity. In stage S2, the resolution is increased to 64×64×8. The prediction results from stage S1 are upsampled and used as conditional input. A scale-adaptive fusion module injects the global structural information from the previous stage into the current stage network, while discrete polynomial diffusion continues to be performed, resulting in a medium-resolution semantic scene. This stage primarily supplements local details and refines semantic boundaries based on the inherited global structure. In stage S3, the resolution is further increased to 256×256×32. Due to the significant increase in the number of voxels at this resolution, the output of the previous stage is first upsampled, and then the target scene is divided into 4×4 spatially overlapping sub-scene blocks. Diffusion generation is performed independently on each sub-scene block, and finally the overlapping areas are fused to obtain the final complete semantic scene. .Depend on Figure 3It is also known that conditional information is transmitted between adjacent stages through the scale-adaptive fusion module, while the highest resolution stage solves the memory bottleneck problem in the high-resolution generation process by overlapping sub-scene segmentation and fusion, thereby ensuring the overall coherence and semantic consistency of the generated scene while controlling the consumption of computing resources.

[0042] Step 102: Perform multi-level hard filtering and probability distribution divergence joint screening on the synthesized semantic scene to obtain a target synthesized scene that meets the quality constraints.

[0043] Due to the randomness of the generation process, low-quality samples may exist in the synthesized semantic scene, such as structural degradation, semantic distribution imbalance, or physical relationship misalignment. Therefore, this step employs a multi-level hard filtering combined with probability distribution divergence to eliminate low-quality samples, retaining only the target synthesized scene that meets the quality constraints.

[0044] As an optional embodiment, in step 102, based on scene geometric integrity, the synthesized semantic scene is subjected to voxel occupancy hard filtering, discarding degenerate samples whose non-empty voxel proportion exceeds a preset real experience range; for scenes that pass the voxel occupancy hard filtering, structural class necessity hard filtering is performed based on scene semantic connectivity, statistically analyzing the coverage richness of preset basic structure categories within the scene, discarding samples that do not contain road surface categories or whose number of structural categories is lower than a diversity threshold; for scenes that pass the structural class necessity hard filtering, gravity consistency hard filtering is performed based on physical reality consistency, calculating the statistical height mean of the ground semantic category set and the tall semantic category set in the height axis direction, discarding spatial geometric misalignment samples whose statistical height mean does not conform to a preset relative height constraint; for scenes that pass the gravity consistency hard filtering, the Jensen-Shannon divergence between the semantic category distribution probability vector and the real scene reference distribution probability vector is calculated, and samples with Jensen-Shannon divergence greater than a preset missing quality hyperparameter threshold are removed to obtain the target synthesized scene.

[0045] Specifically, firstly, based on the geometric integrity of the scene, a hard filtering of voxel occupancy is performed on the synthesized semantic scene. This involves statistically analyzing the proportion of non-empty voxels among all voxels in the synthesized semantic scene and comparing this proportion with a preset range of real-world experience. For example, the aforementioned voxel occupancy rate can be expressed as the following formula: ;in, This represents the set of all voxels in a synthetic semantic scene. Represents all satisfying voxels The set of voxels whose values ​​are not equal to 0, that is, the set of all non-empty voxels. Indicates the number of non-empty voxels. This indicates the total number of elements in the scene. The preset real-world experience range can be set to 2% to 12%. When the percentage is less than 2%, the scene can be considered too sparse, with issues of missing content or structural degradation. When the percentage of non-empty voxels exceeds 12%, the scene can be considered overcrowded or has abnormal filling, thus deviating from the actual scene's occupancy distribution characteristics. Therefore, samples with a non-empty voxel percentage exceeding 2% to 12% can be discarded, and only scenes with an occupancy rate within this range can be retained for subsequent screening steps.

[0046] Secondly, for scenes that pass the voxel occupancy hard filtering, a structural class necessity hard filtering is performed based on scene semantic connectivity and basic structural completeness. Specifically, a set of basic structural categories can be pre-defined, including structural semantic categories such as road surfaces, sidewalks, buildings, vegetation, and grassland. Then, the occurrence and coverage richness of basic structural categories in the current synthesized semantic scene are statistically analyzed. If a scene does not contain a road surface category, it can be considered that the scene lacks basic supporting structures. If the number of structural categories is lower than the diversity threshold, the scene can be considered to have a simple overall structure and insufficient semantic support. The diversity threshold for the number of structural categories can be set to no less than two. Therefore, samples that do not contain a road surface category or have fewer than two structural categories are removed to ensure that the retained scenes have a relatively reasonable basic structure and semantic skeleton.

[0047] Then, for scenes that pass the structural necessity hard filtering, gravity consistency hard filtering is performed based on physical reality consistency. Specifically, the semantic categories in the scene can be divided into a ground semantic category set and a tall semantic category set, and the mean position of the relevant categories in each set along the height axis can be statistically analyzed. The ground semantic category set can include categories such as roads, parking lots, sidewalks, other ground surfaces, and grass, while the tall semantic category set can include categories such as buildings, vegetation, and tree trunks. If the statistical mean height of the ground semantic category set is abnormally high, it indicates that there may be situations such as roads being suspended, which do not conform to the laws of gravity. If the statistical mean height of the tall semantic category set is abnormally low, it indicates that there may be situations where tall objects such as buildings and vegetation are sunken or inverted. Therefore, spatial geometric misalignment samples whose statistical mean height does not meet the preset relative height constraints can be discarded, thereby improving the physical rationality of the retained scene in terms of spatial structure.

[0048] Finally, for scenes that pass the gravity consistency hard filtering, the Jensen-Shannon divergence between their semantic category distribution probability vector and the real scene reference distribution probability vector is calculated, and selection is performed based on the divergence results. Specifically, the proportion of voxels corresponding to each semantic category in the current synthetic semantic scene can be counted first to construct the semantic category distribution probability vector of the synthetic scene. Then, based on multiple real scenes in the real dataset, the real scene reference distribution probability vector is obtained. To avoid the zero probability term affecting the divergence calculation, Laplace smoothing can be performed on the probability vector before calculation. Afterward, the Jensen-Shannon divergence between the synthetic scene distribution and the real scene reference distribution is calculated and compared with a preset missing quality hyperparameter threshold. The preset missing quality hyperparameter threshold can be set to 0.35. When the Jensen-Shannon divergence is greater than 0.35, it indicates that the synthetic scene deviates significantly from the real scene in terms of semantic category distribution and should be removed. When the Jensen-Shannon divergence is not greater than 0.35, the sample is retained as the target synthetic scene that meets the quality constraints.

[0049] For example, suppose This represents the semantic category distribution probability vector of the synthesized scene. This represents the probability vector of the reference distribution for a real-world scene. Specifically, this can be achieved by statistically analyzing multiple real-world scenes in the real training dataset to obtain the average voxel percentage of each semantic category in the real-world scene. For example... This is a reference distribution of the same dimension obtained statistically from 500 real SemanticKITTI training scenarios. Optionally, the probability vector can be Laplace smoothed before computation, for example using... Laplace smoothing parameter processing. Therefore, the Jensen–Shannon divergence is calculated using the following formula:

[0050] in, For is and The average distribution of, i.e. Introduction The purpose is to express the Jensen-Shannon divergence as the average of two Kullback-Leibler divergences, thereby giving the divergence better symmetry and stability. Represents distribution Relative to the intermediate distribution The Kullback-Leibler divergence is used to measure the distribution of the synthesized scene. Relative to the intermediate distribution The degree of difference. Represents distribution Relative to the intermediate distribution The Kullback-Leibler divergence is used to measure the reference distribution of the real scene. Relative to the intermediate distribution The degree of difference. Let represent 2 as the natural logarithm. In this formula, is used... As a normalization coefficient, it is used to normalize the Jensen-Shannon divergence to a fixed range, usually so that its value falls within the [0, 1] interval, which facilitates subsequent threshold comparison and sample selection. This represents the average of the two Kullback-Leibler divergences, and also represents the intermediate distribution. Depend on and Each accounts for half of the total.

[0051] Through the above calculations, the Jensen-Shannon divergence can be normalized to the interval [0, 1]. Then, the Jensen-Shannon divergence is compared with a preset missing-mass hyperparameter threshold, which can be set to 0.35. To indicate and The Jensen-Shannon divergence between two probability distributions is used to measure the degree of difference between them. In this application, This value measures the deviation between the category distribution of a synthesized semantic scene and a reference distribution of the real scene. A smaller value indicates that the synthesized scene is closer to the real scene distribution. A larger value indicates a more significant deviation. A value >0.35 indicates that the synthesized scene deviates significantly from the real scene in terms of semantic category distribution and should be removed. If the value is ≤0.35, the sample is retained as the target synthetic scene that satisfies the quality constraints.

[0052] Further optionally, after removing samples with Jensen-Shannon divergence greater than a preset missing quality hyperparameter threshold in step 102 to obtain the target synthetic scene, the remaining samples after Jensen-Shannon divergence removal can be sorted in ascending order of divergence from small to large, and the samples with the highest preset proportion are retained as the target synthetic scene.

[0053] Specifically, the remaining samples after Jensen-Shannon divergence removal can be sorted in ascending order of divergence from smallest to largest. Since a smaller divergence value indicates that the semantic category distribution of the sample is closer to the real-world reference distribution, samples with smaller divergences can be retained first. The preset proportion can be set to 50%, meaning that the top 50% of the sorted remaining samples are retained as target synthetic scenes. This screening method not only removes low-quality samples that significantly deviate from the real distribution but also, based on threshold screening, further selects high-quality scenes that are closer to the semantic statistical features of the real scene. This provides more stable and reliable input data for subsequent multi-echo ray tracing sparsity enhancement and object pasting enhancement.

[0054] Step 103: Perform multi-echo ray tracing sparsification on the target synthetic scene to obtain a synthetic sparse point cloud paired with the target synthetic scene.

[0055] To make the synthetic data closely resemble the observation characteristics of real radar, this step uses a multi-echo ray tracing sparsifier. Combining the hardware physical characteristics of the target radar sensor with the sensor noise distribution, a virtual radar scan simulation is performed on the dense target synthetic scene to obtain a synthetic sparse point cloud that is paired one-to-one with the target synthetic scene.

[0056] As an optional embodiment, in step 103, the non-uniform vertical elevation angle distribution and azimuth resolution of the target radar sensor are obtained, and multiple rays are emitted from the preset sensor position to the target composite scene according to the equally divided azimuth angle; ray tracing and noise perturbation sampling are performed on each ray, and the observed voxels and invalid voxels are determined according to the feedback echo hit results and geometric occlusion relationship to obtain the composite sparse point cloud.

[0057] Specifically, firstly, the hardware physical parameters of the target radar sensor are acquired, and a virtual scanning model is established based on these parameters. For example, the target radar sensor can be matched with a Velodyne HDL-64E lidar. Assuming the beam parameters include 64 non-uniform vertical elevation beams, the elevation angle distribution range can be from -24.85° to +4.97°, with higher distribution density in some angle ranges to characterize the non-uniform vertical scanning characteristics of a real radar. Assuming the azimuth parameters include 2048 equally divided azimuth angles for transmitting scanning rays in the horizontal range. Based on these parameters, multiple rays can be emitted from a preset sensor position towards the target synthetic scene to simulate the actual scanning process of the target radar sensor on a dense 3D semantic voxel scene. The preset sensor position can be set at a fixed scanning origin in the target voxel grid, for example, at (0, 128, 10).

[0058] Then, ray tracing and noise perturbation sampling are performed on each ray, and the observed voxels and invalid voxels are determined based on the feedback echo hit results and geometric occlusion relationships to obtain a synthetic sparse point cloud.

[0059] Further optionally, in the above steps, each ray is geometrically stepped along the direction of travel with a preset step size. When stepping to a non-empty voxel, an echo intersection is triggered, and distance noise and angle noise are added at the echo intersection to disturb the sampling position. The sampling point falling into the non-empty voxel is back-projected back to the voxel coordinates and marked as a physical observation voxel. After triggering the echo intersection, the ray is advanced forward a preset distance to cross the currently hit voxel, and continues to track the next echo intersection along the original ray direction to obtain the first and second echo intersections of each ray until the maximum number of echoes of the target radar sensor is reached or the maximum scanning cutoff distance is exceeded. Voxels that are not hit by any ray are marked as invalid voxels, thus obtaining the synthetic sparse point cloud that conforms to the geometric occlusion rules and is matched with the target synthetic scene.

[0060] Specifically, firstly, the spatial propagation direction of the current light beam is determined based on its vertical elevation and azimuth angles, and then geometrically stepped along this direction with a preset step size. For example, the preset step size can be set to 5cm. During the stepping process, if the current stepping position enters a non-empty voxel in the target composite scene, it is determined that this position corresponds to an echo intersection point, and the spatial position of this echo intersection point is recorded. In this way, the gradual propagation process of the laser beam in the 3D scene can be simulated, and the echo trigger position can be determined based on the first hit relationship of the non-empty voxels.

[0061] After detecting the echo intersection, range noise and angular noise can be introduced at the intersection to sample the perturbation position of the theoretical hit location. Specifically, the range noise can adopt a Gaussian distribution with a standard deviation of 0.03 meters. The angular noise can adopt a Gaussian distribution in both the horizontal and vertical angular directions, with a standard deviation of 0.002 radians for each echo. For each echo, a preset number of perturbation position samplings can be performed, for example, two perturbation position samplings, to obtain multiple perturbation sampling points. Then, each perturbation sampling point is back-projected into the voxel coordinate system. If the corresponding sampling point falls into a non-empty voxel, the non-empty voxel is marked as the physical observation voxel. In this way, the random error characteristics of real lidar in range and angle measurement can be simulated better, making the resulting synthetic sparse point cloud more closely resemble the real scanning results in spatial distribution.

[0062] Furthermore, after triggering the current echo intersection, the tracking of the ray is not immediately terminated. Instead, the ray continues to advance forward a preset distance along its original propagation direction to pass the currently hit voxel and continue searching for subsequent echoes. For example, the forward advancement distance can be set to 0.2 meters. After passing the currently hit voxel, geometric steps are performed along the original ray direction with the preset step size until another non-empty voxel is reached, thus obtaining the next echo intersection. For each ray, its first and second echo intersections can be obtained to simulate the multi-echo detection mode of the target radar sensor. For example, the maximum number of echoes from the target radar sensor can be limited to 2. If the ray propagation distance exceeds the maximum scanning cutoff distance during tracking, the tracking of the ray is terminated. For example, the maximum scanning cutoff distance can be set to 80 meters. Thus, multiple effective echoes that the same ray may generate can be retained in scenarios with partial obstruction, edge structures, or complex spatial targets.

[0063] In this embodiment, the first echo intersection point refers to the position where a single ray emitted by the target radar sensor first hits a non-empty voxel in the target composite scene as it geometrically steps along the propagation direction. The second echo intersection point refers to the position where, after obtaining the first echo intersection point, the ray continues forward along the original propagation direction, passes the currently hit voxel, and hits a subsequent non-empty voxel again. By obtaining the first and second echo intersection points corresponding to the same ray, the multi-echo detection process of the target radar sensor under conditions of partial obstruction, edge reflection, or complex scenes can be simulated, thereby making the generated composite sparse point cloud more consistent with the observation characteristics of real radar.

[0064] Finally, after all rays have been tracked, the observed voxels and invalid voxels are determined based on the echo hit results and geometric occlusion relationships of each ray. Specifically, non-empty voxels hit by any echo intersection or perturbation sampling point can be used as observed voxels to construct the synthetic sparse point cloud. Voxels not hit by any ray are marked as invalid voxels. Since rays follow the geometric rules of forward hit and backward occlusion during propagation, voxels located behind the hit target and occluded by the entity in front will not be mistakenly marked as observed voxels, thus ensuring that the generated synthetic sparse point cloud satisfies the geometric occlusion rules in real lidar scanning. Therefore, a synthetic sparse point cloud paired with the target synthetic scene can be obtained.

[0065] For example, Figure 4 This illustration shows a schematic diagram of the principle of multi-echo ray tracing sparsity in an embodiment of this application, simulating the Velodyne HDL-64E lidar. Figure 4In the diagram, the left side shows the sensor and its corresponding virtual scanning origin. The virtual scanning origin can be set at a fixed position in the target voxel grid, for example, voxel coordinates (0, 128, 10). Starting from this virtual scanning origin, multiple rays are emitted towards the target synthetic scene according to the scanning parameters of the target radar sensor to simulate the scanning process of a real lidar on a three-dimensional semantic voxel scene. In conjunction with the embodiments of this application, the target radar sensor can be matched with a Velodyne HDL-64E lidar, which may include 64 non-uniformly distributed elevation beams, with an elevation range roughly covering -24.85° to +4.97°, and a denser distribution in the [-10°, 0°] interval; each beam corresponds to 2048 azimuth sampling directions, thereby forming multiple scanning rays in the horizontal range and performing ray tracing within a preset maximum scanning cutoff distance.

[0066] Figure 4 The grid area on the right represents the voxel space corresponding to the target composite scene. Figure 4 The dots in the diagram represent the first non-empty voxels hit during the propagation of light, i.e., the observed voxels corresponding to the first echo intersection. These voxels can be written into the synthetic sparse point cloud. The gray dashed lines in the diagram represent light rays that do not hit any non-empty voxels. Correspondingly, such light rays do not form effective echoes along their propagation path, and the voxels along their path can be marked as unobserved or invalid regions. For any emitted light ray, when it takes a geometric step of a preset step size along the propagation direction and enters a non-empty voxel for the first time, the first echo intersection can be formed. After passing the currently hit voxel, the light ray can continue to propagate along the original direction to search for the second echo intersection, thereby realizing the simulation of the double echo characteristics. Figure 4 As can be seen, after the light rays originate from the virtual scanning origin, they follow the geometric propagation rules of forward hit priority and backward occlusion failure. Therefore, voxels located after hitting the target in front and occluded by the entity in front will not be incorrectly written into the synthesized sparse point cloud. Thus, Figure 4 This paper intuitively demonstrates how the embodiments of this application realize the transformation process from target synthetic scene to paired synthetic sparse point cloud by using the correspondence between virtual sensor position, non-uniform beam distribution, light propagation path, echo hit position and miss path. It also reflects the technical features of the embodiments of this application in terms of multi-echo modeling, geometric occlusion constraints and observation realism.

[0067] For example, after detecting the echo intersection point corresponding to a certain ray, noise disturbance sampling can be performed around the echo intersection point to simulate the random errors in the distance and angle measurement processes of a real lidar. Specifically, for each echo, SamplesPerReturn = 2 noise samples can be additionally sampled. The position of the k-th noise sample can be represented as: ;in, Indicates the virtual scan origin position. This indicates the distance traveled along the direction of light propagation at the current echo intersection point. This represents the ray direction vector after perturbations in vertical elevation and azimuth angles. Represents distance noise, which follows a Gaussian distribution. ~ N(0, ), its standard deviation It can be set to 0.03 m. and These represent angular noise in the horizontal and vertical directions, respectively, and both follow a Gaussian distribution. , ~ N(0, ),in = = 0.002 rad. After backprojecting the noise sample positions back into the voxel coordinate system, if the corresponding sample falls into a non-empty voxel, then that voxel is marked as observed, i.e., denoted as . [u]=1.

[0068] For example, for the same echo intersection point, two noisy perturbation sample positions can be generated. If one or more of these samples fall into a non-empty voxel in the target synthetic scene, the corresponding voxel is written into the synthetic sparse point cloud, thus making the generated sparse input closer to the actual scanning results of the real Velodyne HDL-64E lidar in terms of point distribution density and spatial discreteness. The numerical range corresponding to the distance noise standard deviation of 0.03 m and the angle noise standard deviation of 0.002 rad can match the publicly reported distance accuracy of approximately 2-3 cm and angle resolution on the order of approximately 0.06° of the HDL-64E.

[0069] Furthermore, after all rays have completed tracing and noise perturbation sampling, voxels not hit by any rays can be marked as unobserved or invalid voxels, denoted as V[v]=1, and these voxels do not participate in subsequent evaluation. In this way, the generated synthetic sparse point cloud not only preserves the multi-echo characteristics and sensor noise features, but also strictly adheres to the geometric occlusion rule, meaning that voxels located behind entities in front and occluded will not be mistakenly marked as observed voxels. Therefore, compared to simple random downsampling or only using first-hit tracing, it can more realistically simulate the observation process of actual lidar.

[0070] Step 104: Based on the rare class object library, object pasting enhancement is performed on the synthesized sparse point cloud and the corresponding scene voxel ground truth values ​​in the target synthesized scene to obtain a synthesis training pair composed of the enhanced synthesized sparse point cloud and the scene voxel ground truth values. The synthesis training pair is used to train the 3D semantic scene completion model.

[0071] To further improve the imbalance of long-tail categories, this step enhances the synthesis of sparse point clouds and their corresponding scene voxel ground truth values ​​by pasting objects based on a pre-built rare class object library.

[0072] Optionally, the process of pre-constructing a rare class object library can be implemented as follows: collect ground truth voxel scenes from real training datasets, perform 3D connectivity component analysis on each scene, and extract multiple semantic region entities that are spatially independent and connected; set a voxel count threshold for entities, and select entities whose categories belong to a preset long-tail category set as rare class objects; store the selected rare class objects in a structured format combining sparse coordinates and semantic labels into the rare class object library, and set an upper limit capacity for the number of storage instances of each long-tail category.

[0073] In the above steps, firstly, multiple ground-value voxel scenes are obtained from a real training dataset, such as the SemanticKITTI training set. Then, three-dimensional connectivity component analysis is performed on each ground-value voxel scene to identify semantically independent and interconnected semantic regions within the scene. Specifically, the three-dimensional connectivity component analysis can employ 26-connectivity, establishing connectivity between the target voxel and its 26 adjacent voxels in three-dimensional space, thereby extracting a more complete three-dimensional object entity.

[0074] After obtaining the connected components, further filtering can be performed based on the voxel count threshold and semantic category of the entities. Specifically, connected components with a voxel count within a preset range can be retained, for example, entities with a voxel count greater than or equal to 5 and less than or equal to 10,000, to filter out excessively small noise regions and excessively large non-target regions. Subsequently, from the entities that meet the voxel count threshold, entities belonging to a preset long-tail category set are selected as rare class objects. The preset long-tail category set may include at least one of bicycles, motorcyclists, pedestrians, utility poles, and traffic signs. For example, it may further include rare semantic categories such as motorcycles, trucks, other vehicles, riders, and tree trunks.

[0075] After identifying the rare class objects, the voxel coordinate information and corresponding semantic label information of each object can be structurally represented and stored in the rare class object library in a format combining sparse coordinates and semantic labels. To avoid an imbalance in the object library distribution caused by too many samples of a single category, and to control the capacity of the object library, an upper limit can be set on the number of storage instances for each long-tail category, for example, a maximum of 500 instances can be retained for each long-tail category. In this way, a rare class object library covering multiple long-tail targets can be constructed, facilitating subsequent retrieval and pasting calls, providing basic data support for subsequent object pasting enhancement.

[0076] As an optional embodiment, in step 104, the first stage of minimum coverage pasting is performed. The target composite scene is traversed, and for rare long-tail categories in the target composite scene where the number of voxels is lower than the minimum coverage threshold, corresponding object entities are extracted from the rare class object library for supplementary pasting, so that the number of voxels in the pasted rare long-tail categories reaches the minimum coverage threshold. The second stage of adaptive balancing pasting is performed, initiating multiple rounds of adaptive balancing loops. In each loop, the voxel volume of each long-tail category in the current scene is counted, and object entities corresponding to the rarest category are extracted from the rare class object library for supplementary pasting, until the voxel volume of each long-tail category reaches balance or the number of loops reaches a preset upper limit. During the two-stage pasting process, spatial placement constraints are applied to the extracted object entities. After each spatial placement is completed, the composite sparse point cloud and the corresponding scene voxel ground truth in the target composite scene are updated synchronously to maintain the pairing consistency between the sparse point cloud and the dense ground truth.

[0077] In the above embodiments, firstly, the rare class object library can be pre-constructed offline. Specifically, based on the ground truth values ​​of scene voxels in the real training dataset, three-dimensional connectivity component analysis can be performed on each three-dimensional semantic scene to extract spatially independent object entities. The three-dimensional connectivity component analysis uses 26-connectivity, establishing connectivity between the target voxel and voxels in its 26 adjacent directions, thereby extracting the spatial structure of the three-dimensional target more completely. The extracted connectivity components can be further filtered by combining semantic category and voxel count, retaining object entities belonging to a preset long-tail category set. The preset long-tail category set may include at least one of bicycles, motorcyclists, pedestrians, utility poles, and traffic signs. For example, it can also be expanded to include rare categories such as motorcycles, trucks, other vehicles, riders, and tree trunks. To control the size of the object library while considering category richness, a maximum storage instance capacity can be set for each long-tail category, where the maximum storage instance capacity for each long-tail category can be set to 500. The filtered object entities can be stored in the form of sparse coordinates combined with semantic labels for subsequent pasting and retrieval.

[0078] Secondly, a first-stage minimum coverage pasting process can be performed. Specifically, each rare long-tail category in the target synthesis scene can be traversed, and the number of voxels corresponding to each rare long-tail category can be counted. If the number of voxels for a certain category is lower than the minimum coverage threshold, the object entity corresponding to that category is extracted from the rare class object library and pasted into the target synthesis scene and its corresponding synthetic sparse point cloud to supplement the voxel coverage of that category. The minimum coverage threshold can be set to 50 voxels. Through the first-stage minimum coverage pasting, it can be ensured that each rare long-tail category has at least basic coverage in a single target synthesis scene, thereby avoiding the absence or low proportion of some long-tail categories in the training samples.

[0079] Furthermore, after completing the first stage of minimum coverage pasting, a second stage of adaptive balancing pasting can be performed. Specifically, multiple rounds of adaptive balancing loops can be initiated. In each loop, the voxel volume of each long-tail category in the current scene is counted, and the rarest category with the fewest voxels is determined. Subsequently, the object entity corresponding to the rarest category is extracted from the rare class object library, and supplementary pasting is performed under the premise of satisfying spatial placement constraints to gradually improve the distribution balance among the long-tail categories. The adaptive balancing loop can continue until the voxel volume of each long-tail category in the current scene reaches a relative balance, or the number of loops reaches a preset upper limit. The preset upper limit can be set to 10 rounds. Through the second stage of adaptive balancing pasting, not only can categories with significantly insufficient voxel numbers be supplemented, but the relative imbalance between different long-tail categories can also be further alleviated.

[0080] Further optionally, spatial placement constraints include mesh boundary constraints that ensure entities are completely enclosed within scene boundaries, non-overlay constraints that prohibit overlaying existing scene voxels of different categories, and ground anchoring constraints that require the entity placement point to be directly below a preset ground class set.

[0081] During the two-stage pasting process described above, spatial placement constraints can be applied to the extracted object entities to ensure the reasonableness of the pasting result in terms of spatial relationships and physical semantics. For example, spatial placement constraints may include: grid boundary constraints, requiring the object entity to be completely contained within the voxel boundaries of the target scene after placement to avoid boundary overflow; non-overlapping constraints, requiring the object entity not to cover existing non-empty voxels of different categories in the scene after placement to avoid semantic conflicts or spatial overlap; and ground anchoring constraints, requiring that the object entity's placement point must be directly below a preset ground class set to ensure the object is in a reasonable supporting position rather than floating. For example, the preset ground class set may include categories such as road surface, sidewalk, parking lot, and grass. The formal pasting of the object entity is only performed when the candidate placement location simultaneously satisfies the above constraints.

[0082] Understandably, mesh boundary constraints are used to ensure that the object to be pasted is completely within the 3D voxel mesh range corresponding to the target scene after spatial placement. Specifically, after determining the candidate placement positions of the object, the overall coordinate boundary after placement can be calculated based on the circumscribed voxel range of the object, and it can be determined whether this overall coordinate boundary exceeds the voxel mesh boundary of the current scene. If any part of the object exceeds the scene boundary after placement, the candidate position is determined not to meet the mesh boundary constraints, and pasting is not performed; only when all voxels of the object fall within the valid mesh range of the target scene is subsequent constraint judgment allowed. Through this constraint, problems such as object truncation due to out-of-bounds movement, structural incompleteness, or writing to invalid regions can be avoided.

[0083] Secondly, the non-overlay constraint is used to avoid spatial conflicts between newly pasted object entities and existing semantic voxels in the target scene. Specifically, after determining the candidate placement location, the coordinates of each voxel of the object entity at that location are compared one by one with the existing non-empty voxels in the current target scene. If the object entity, after placement, would overlay existing non-empty voxels of a different category, it indicates that the location would cause semantic conflict or spatial overlap, and the candidate location should be determined not to satisfy the non-overlay constraint. Pasting is only allowed to continue when the object entity, after placement, does not overlay existing non-empty voxels of a different category. This constraint avoids unreasonable situations such as pedestrians and vehicles overlapping, traffic signs inserted into buildings, or utility poles passing through other targets, thereby ensuring the consistency of the enhanced scene in terms of semantics and spatial relationships.

[0084] Secondly, ground anchoring constraints are used to ensure that object entities have reasonable support relationships, preventing objects from floating or being buried in the ground. Specifically, after determining the candidate placement location, the voxel category directly below the object entity's placement point can be detected to determine whether it belongs to a preset ground category set. The preset ground category set can include semantic categories such as roads, sidewalks, parking lots, and grass that can provide effective support for the target object. If the voxel directly below the object entity's placement point does not belong to the preset ground category set, the object is considered to lack a reasonable support foundation, does not meet the ground anchoring constraint, and is not pasted; the object entity is only allowed to be placed at the candidate location if the voxel directly below the placement point is from the preset ground category set. Through this constraint, objects such as pedestrians, bicycles, motorcyclists, utility poles, and traffic signs can be made to more conform to the physical placement rules in real-world scenarios.

[0085] Furthermore, in the actual pasting process, the aforementioned mesh boundary constraints, non-overlapping constraints, and ground anchoring constraints can be used in combination. Only when the candidate placement location simultaneously satisfies all spatial placement constraints will the corresponding object entity be formally pasted into the target scene, and the synthetic sparse point cloud and scene voxel ground truth values ​​will be updated simultaneously. Through the above methods, the spatial rationality, semantic consistency, and physical realism of the enhanced object pasting can be improved.

[0086] Finally, after each object entity is placed in space, the synthesized sparse point cloud and the corresponding scene voxel ground truth in the target synthesized scene are updated synchronously to maintain the pairing consistency between the sparse point cloud and the dense ground truth. Specifically, when an object entity is successfully pasted into the target scene, the semantic voxel corresponding to the object entity is written into the scene voxel ground truth, and the corresponding observation information in the synthesized sparse point cloud is updated synchronously, so that the enhanced sparse input and the enhanced dense ground truth maintain a one-to-one correspondence. Thus, a synthesis training pair consisting of the enhanced synthesized sparse point cloud and the scene voxel ground truth can be obtained. This synthesis training pair can be directly used to train the 3D semantic scene completion model to improve the model's perception ability of long-tail object categories and the overall scene completion effect.

[0087] Furthermore, subsystems A, B, C, and D can be executed sequentially, outputting a synthetic training pair consisting of a synthetic sparse point cloud and ground truth voxels in each round. Only samples that pass the quality filter of subsystem B enter the subsequent ray tracing sparsification and object pasting enhancement stages. In one example, embodiments of this application can generate synthetic training sets of different sizes, such as 10,000, 31,000, and 57,000 sets forming training pairs, and then combine these synthetic training pairs with the real training set for downstream 3D semantic scene completion model training. Accordingly, experimental results on the SemanticKITTI validation set show that the completion performance of the downstream model can be significantly improved after using the synthetic training pairs constructed in embodiments of this application, especially in rare categories such as motorcyclists and other vehicles, indicating that the data augmentation method provided in embodiments of this application can effectively improve the adverse effects of insufficient long-tail category samples on model training.

[0088] For example, Figure 5 A schematic diagram showing the results of the data enhancement pipeline after actual operation in an embodiment of this application is illustrated. Figure 5 Three perspectives of the same synthetic scene are presented, with the left image showing the complete synthetic semantic scene generated after pyramid discrete diffusion. The perspective view, with the middle image showing the synthetic sparse point cloud obtained after performing multi-echo ray tracing sparsification on the complete synthetic semantic scene. The perspective view, with the right image being a top-down BEV semantic map of the same scene, is used to demonstrate the spatial distribution of various semantic categories within the scene, especially rare categories, after enhancement through the rare class object library pasting. Figure 5 As can be intuitively seen, the embodiments of this application, through pyramid diffusion generation, ray tracing sparsification, and rare class enhancement, can form a closed-loop result from the complete semantic scene to the sparse radar input and then to the top-down semantic verification, thereby verifying the effectiveness of the generated synthetic training in terms of structural integrity, observational realism, and semantic distribution rationality.

[0089] In this embodiment, the 3D semantic scene completion model refers to a model used to recover the complete 3D spatial occupancy state and semantic category distribution of a scene based on sparse observation input. This model can receive synthetic sparse point clouds obtained by multi-echo ray tracing sparsification as input and output the corresponding complete scene voxel ground truth prediction results, thereby achieving the completion of dense 3D semantic scenes observed by sparse LiDAR. In other words, the input of the 3D semantic scene completion model is a sparse point cloud or its corresponding sparse voxel representation, and the output is a complete 3D semantic voxel scene, which can be used to characterize the distribution of semantic categories such as road surfaces, sidewalks, buildings, vegetation, vehicles, pedestrians, cyclists, traffic signs, and utility poles in 3D space. Based on the synthetic training pairs constructed in this application, the 3D semantic scene completion model can learn the mapping relationship from sparse input to dense semantic ground truth, thereby improving the model's ability to understand the 3D environment in complex scenes.

[0090] Optionally, the 3D semantic scene completion model includes a parameter-frozen base model and a discrete diffusion correction network. The base model is used to perform preliminary feature extraction and coarse prediction on the input synthetic sparse point cloud, and outputs a source prediction distribution representing the semantic probability distribution of each voxel. The discrete diffusion correction network is used to perform a few-step correction based on the source prediction distribution to improve the accuracy of the semantic completion result in terms of structural integrity, boundary details, and category discrimination. The parameter-frozen base model can be a pre-trained 3D semantic scene completion backbone network, whose parameters are not updated during the training of the discrete diffusion correction network, thus allowing the training process to focus on learning subsequent correction capabilities.

[0091] Based on this, the synthetic training pair is used to train the discrete diffusion correction network, including: inputting the synthetic sparse point cloud from the synthetic training pair into the base model; performing feature extraction and preliminary prediction on the synthetic sparse point cloud to obtain a source prediction distribution representing the semantic probability distribution of each voxel; using the source prediction distribution as the starting point of the diffusion correction process, and using the ground truth values ​​of scene voxels in the synthetic training pair as supervision, training the discrete diffusion correction network to perform a few-step correction on the source prediction distribution. In this way, without changing the main structure of the base model, the discrete diffusion correction mechanism can compensate for problems such as local semantic errors, structural breaks, and insufficient long-tail category recognition in the base model output.

[0092] For example, when training a discrete diffusion correction network, conditional information can be constructed based on the bird's-eye view semantic features obtained by projecting the source prediction distribution along the height dimension, and the sparse voxel occupancy map obtained from the synthesized sparse point cloud. The source prediction distribution, diffusion time step, and conditional information are then input into the discrete diffusion correction network. Specifically, the bird's-eye view semantic features can provide global semantic layout information of the scene from a top-down perspective, the sparse voxel occupancy map can characterize the actual observation position and occupancy constraints of the input point cloud in three-dimensional space, and the diffusion time step indicates the current discrete diffusion stage of the correction process. Under the joint constraints of the above multi-source conditional information, the discrete diffusion correction network progressively refines and corrects the source prediction distribution, making the corrected voxel semantic prediction results closer to the true voxel values ​​of the scene.

[0093] Furthermore, considering downstream usage methods, the 3D semantic scene completion model can be a specific downstream SSC model, such as S... 2 D 2 The model can also be any other model capable of converting sparse LiDAR input into complete 3D semantic voxel output. Compared to training with only real data, training the discrete diffusion correction network using the synthetic training pair constructed in this application can enhance the model's ability to complete long-tailed categories, complex occluded regions, and sparse observation regions without changing the parameters of the original base model, thereby improving the overall 3D semantic scene completion performance.

[0094] In this embodiment, a synthetic semantic scene is generated through multi-stage pyramid discrete diffusion, and high-quality scenes are retained by combining multi-level hard filtering and Jensen-Shannon divergence screening. Then, a synthetic sparse point cloud, strictly paired with dense ground truth, is generated based on the multi-echo ray tracing mechanism of the target radar sensor. Finally, object pasting enhancement is performed on long-tail categories using a rare class object library, thereby constructing a synthetic training pair that combines structural rationality, semantic consistency, sensor physical realism, and balanced category distribution. This effectively alleviates the problems of insufficient real-world labeled data, inadequate coverage of long-tail categories, and insufficient realism and task adaptability of traditional enhancement methods. It helps improve the training effect of the 3D semantic scene completion model and the environmental perception capability of the embodied intelligent system in complex scenes.

[0095] The above describes a radar point cloud data enhancement method for embodied intelligence in the embodiments of this application. The radar point cloud data enhancement device for embodied intelligence that performs the above-described radar point cloud data enhancement method for embodied intelligence will be described below.

[0096] See Figure 6 ,like Figure 6The diagram illustrates the structure of a radar point cloud data augmentation device for embodied intelligence. This device can be applied to data processing equipment related to embodied intelligence systems, autonomous driving systems, service robots, inspection robots, or other mobile robots. It is used to perform data augmentation processing on the 3D semantic scene to be augmented, generating synthetic training pairs that can be used to train a 3D semantic scene completion model. In this embodiment, Figure 6 The radar point cloud data augmentation device for embodied intelligence shown can achieve the aforementioned Figure 1 The steps of the radar point cloud data enhancement method for embodied intelligence in the corresponding embodiments are described. The functions implemented by the radar point cloud data enhancement device for embodied intelligence can be achieved either through hardware or through hardware executing corresponding software. The hardware or software may include one or more modules corresponding to the above functions, and the modules may be software modules and / or hardware modules. Specifically, the radar point cloud data enhancement device for embodied intelligence may include a generation module, a filtering module, a ray tracing module, and an enhancement module.

[0097] The generation module is configured to generate a synthetic semantic scene from the 3D semantic scene to be enhanced through a multi-stage process using a pyramid discrete diffusion model; the pyramid discrete diffusion model generates the synthetic semantic scene sequentially according to a resolution hierarchy from coarse to fine. The filtering module is configured to perform multi-level hard filtering and probability distribution divergence joint filtering on the synthetic semantic scene to obtain a target synthetic scene that meets the quality constraints. The ray tracing module is configured to perform multi-echo ray tracing sparsification on the target synthetic scene to obtain a synthetic sparse point cloud paired with the target synthetic scene; The enhancement module is configured to perform object pasting enhancement on the synthetic sparse point cloud and the corresponding scene voxel ground truth values ​​in the target synthetic scene based on a rare class object library, so as to obtain a synthetic training pair composed of the enhanced synthetic sparse point cloud and the scene voxel ground truth values; the synthetic training pair is used to train a 3D semantic scene completion model.

[0098] It should be understood that the functions of the generation module, filtering module, ray tracing module, and enhancement module can be implemented with reference to the foregoing. Figure 1 The relevant descriptions of steps 101 to 104 in the corresponding embodiments will not be repeated here.

[0099] The radar point cloud data enhancement device for embodied intelligence in the embodiments of this application has been described above from the perspective of modular functional entities. The radar point cloud data enhancement device for embodied intelligence in the embodiments of this application will be described below from the perspective of hardware processing.

[0100] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0101] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0102] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0103] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0105] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0106] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0107] The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. A method for augmenting radar point cloud data for embodied intelligence, characterized in that, The method is applied to an embodied intelligence system, and the method includes: The three-dimensional semantic scene to be enhanced is generated in multiple stages using a pyramid discrete diffusion model to obtain a synthetic semantic scene; the pyramid discrete diffusion model generates the synthetic semantic scene sequentially according to a resolution hierarchy from coarse to fine. The synthesized semantic scene is subjected to multi-level hard filtering and probability distribution divergence joint screening to obtain a target synthesized scene that meets the quality constraints; Multi-echo ray tracing sparsification is performed on the target synthetic scene to obtain a synthetic sparse point cloud paired with the target synthetic scene; Based on a rare class object library, object pasting enhancement is performed on the synthesized sparse point cloud and the corresponding scene voxel ground value in the target synthesized scene to obtain a synthesis training pair composed of the enhanced synthesized sparse point cloud and the scene voxel ground value; the synthesis training pair is used to train a 3D semantic scene completion model.

2. The radar point cloud data augmentation method for embodied intelligence according to claim 1, characterized in that, The process of generating a synthetic semantic scene by using a pyramid discrete diffusion model in multiple stages to enhance the 3D semantic scene to be obtained includes: In the initial stage, multi-step discrete polynomial diffusion is performed starting from the full mask tensor under the lowest resolution 3D mesh to gradually denoise each voxel from the mask state to the semantic category, generating a coarse-grained semantic scene. In each subsequent stage, the coarse-grained semantic scene generated in the previous stage is upsampled and feature-mapped through the scale adaptive fusion module, and injected as a conditional feature into the medium-resolution 3D mesh corresponding to the current resolution. It is then combined with the full mask tensor of the current resolution for discrete polynomial diffusion to generate the medium-resolution semantic scene at the current resolution. At the highest resolution stage, the medium-resolution semantic scene generated in the previous stage is upsampled by the scale-adaptive fusion module and divided into multiple spatially overlapping sub-scene blocks. Discrete polynomial diffusion generation is performed independently on each sub-scene block, and the voxel prediction values ​​of adjacent sub-scene blocks in the overlapping area are averaged and fused to obtain the synthetic semantic scene at the target resolution.

3. The radar point cloud data augmentation method for embodied intelligence according to claim 1, characterized in that, The step of performing multi-level hard filtering and probability distribution divergence joint screening on the synthesized semantic scene to obtain a target synthesized scene that meets quality constraints includes: Based on scene geometric integrity, voxel occupancy hard filtering is performed on the synthetic semantic scene to discard degenerate samples whose non-empty voxel occupancy exceeds the preset real experience range. For the scene that passes the voxel occupancy hard filtering, structural class necessity hard filtering is performed based on the scene semantic connectivity. The coverage richness of the preset basic structure categories in the scene is counted, and samples that do not contain road surface categories or whose number of structure categories is lower than the diversity threshold are discarded. For scenarios that pass the structural necessity hard filtering, gravity consistency hard filtering is performed based on physical reality consistency. The statistical height mean of the ground semantic category set and the tall semantic category set in the height axis direction is calculated respectively, and spatial geometric misalignment samples whose statistical height mean does not meet the preset relative height constraint are discarded. For the scene that passes the gravity consistency hard filtering, calculate the Jensen-Shannon divergence between the semantic category distribution probability vector and the real scene reference distribution probability vector, and remove samples whose Jensen-Shannon divergence is greater than a preset missing quality hyperparameter threshold to obtain the target synthesized scene.

4. The radar point cloud data augmentation method for embodied intelligence according to claim 3, characterized in that, After removing samples whose Jensen-Shannon divergence is greater than a preset missing quality hyperparameter threshold to obtain the target synthetic scene, the process further includes: The remaining samples after Jensen-Shannon divergence removal are sorted in ascending order of divergence from smallest to largest, and the samples at the top of the sorted list are retained as the target synthetic scene.

5. The radar point cloud data augmentation method for embodied intelligence according to claim 1, characterized in that, The step of performing multi-echo ray tracing sparsification on the target synthetic scene to obtain a synthetic sparse point cloud paired with the target synthetic scene includes: The non-uniform vertical elevation angle distribution and azimuth resolution of the target radar sensor are obtained, and multiple rays of light are emitted from the preset sensor position into the target composite scene according to the equally divided azimuth angles. Ray tracing and noise perturbation sampling are performed on each ray, and the observed voxels and invalid voxels are determined based on the feedback echo hit results and geometric occlusion relationships to obtain the synthesized sparse point cloud.

6. The radar point cloud data augmentation method for embodied intelligence according to claim 5, characterized in that, The process of performing ray tracing and noise perturbation sampling on each ray, and determining the observed voxels and invalid voxels based on the feedback echo hit results and geometric occlusion relationships to obtain the synthesized sparse point cloud includes: For each ray, geometric steps are performed along the direction of travel with a preset step size. When the step reaches a non-empty voxel, an echo intersection is triggered. Distance noise and angle noise are added at the echo intersection to perform perturbation position sampling. The sampling points that fall into the non-empty voxel are back-projected back to the voxel coordinates and marked as physical observation voxels. After triggering the echo intersection, the light beam is advanced forward a preset distance to pass the currently hit voxel, and continues to track the next echo intersection along the original light beam direction to obtain the first and second echo intersections of each light beam until the maximum number of echoes of the target radar sensor is reached or the maximum scanning cutoff distance is exceeded. Voxels that are not hit by any light are marked as invalid voxels, resulting in the synthetic sparse point cloud that conforms to the geometric occlusion rules and is paired with the target synthetic scene.

7. The radar point cloud data augmentation method for embodied intelligence according to claim 1, characterized in that, The method of enhancing the synthesized sparse point cloud and the corresponding scene voxel ground truth values ​​in the target synthesized scene by object pasting based on the rare class object library, to obtain a synthesis training pair consisting of the enhanced synthesized sparse point cloud and the scene voxel ground truth values, includes: Perform the first stage of minimum coverage pasting, traverse the target compositing scene, and for rare long-tail categories in the target compositing scene whose number of voxels is lower than the minimum coverage threshold, extract the corresponding object entities from the rare class object library for supplementary pasting, so that the number of voxels of the pasted rare long-tail categories reaches the minimum coverage threshold. The second stage of adaptive balancing pasting is executed, and multiple rounds of adaptive balancing loops are started. In each round of loops, the voxel volume of each long-tail category in the current scene is counted, and the object entity corresponding to the rarest category is extracted from the rare class object library for supplementary pasting until the voxel volume of each long-tail category reaches balance or the number of loops reaches the preset upper limit. During the two-stage pasting process, spatial placement constraints are applied to the extracted object entities; After each spatial placement is completed, the synthesized sparse point cloud and the corresponding scene voxel ground truth in the target synthesized scene are updated synchronously to maintain the pairing consistency between the sparse point cloud and the dense ground truth.

8. The radar point cloud data augmentation method for embodied intelligence according to claim 7, characterized in that, The rare class object library is pre-built through the following steps, including: Collect ground truth voxel scenes from real training datasets, perform 3D connectivity component analysis on each scene, and extract multiple spatially independent and connected semantic region entities. Set a threshold for the number of voxels of an entity, and filter out entities whose categories belong to a preset long-tail category set as rare class objects; The selected rare class objects are stored in the rare class object library in a structured format that combines sparse coordinates and semantic tags, and an upper limit capacity is set for the number of storage instances of each long-tail category.

9. The radar point cloud data augmentation method for embodied intelligence according to claim 1, characterized in that, The three-dimensional semantic scene completion model includes a parameter-frozen base model and a discrete diffusion correction network; The synthetic training pair is used to train the discrete diffusion correction network, including: The synthetic sparse point cloud from the synthetic training pair is input into the base model, and features are extracted and preliminary predictions are made on the synthetic sparse point cloud to obtain the source prediction distribution used to characterize the semantic probability distribution of each voxel. Using the source prediction distribution as the starting point of the diffusion correction process and the scene voxel ground truth values ​​in the synthetic training pair as supervision, the discrete diffusion correction network is trained to perform a few-step correction on the source prediction distribution.

10. A radar point cloud data augmentation device for embodied intelligence, characterized in that, The device is used in an embodied intelligence system, and the device includes: The generation module is configured to generate a synthetic semantic scene from the 3D semantic scene to be enhanced through a multi-stage process using a pyramid discrete diffusion model; the pyramid discrete diffusion model generates the synthetic semantic scene sequentially according to a resolution hierarchy from coarse to fine. The filtering module is configured to perform multi-level hard filtering and probability distribution divergence joint filtering on the synthetic semantic scene to obtain a target synthetic scene that meets the quality constraints. The ray tracing module is configured to perform multi-echo ray tracing sparsification on the target synthetic scene to obtain a synthetic sparse point cloud paired with the target synthetic scene; The enhancement module is configured to perform object pasting enhancement on the synthetic sparse point cloud and the corresponding scene voxel ground truth values ​​in the target synthetic scene based on a rare class object library, so as to obtain a synthetic training pair composed of the enhanced synthetic sparse point cloud and the scene voxel ground truth values; the synthetic training pair is used to train a 3D semantic scene completion model.

Images