Radar overhead map semantic segmentation method and device for embodied intelligence

Abstract

This application relates to the field of artificial intelligence, providing a method and apparatus for semantic segmentation of radar bird's-eye view images for embodied intelligence. The method includes: inputting a sparse 3D point cloud into a base model to obtain a 3D source prediction distribution; projecting the 3D source prediction distribution along the vertical direction into a column, and taking the semantic category corresponding to the highest non-empty voxel in the vertical direction to obtain a baseline bird's-eye view semantic map; using the one-hot encoding of the baseline bird's-eye view semantic map as a 2D structured source distribution, inputting the 2D structured source distribution, diffusion time step, and conditional information into a 2D discrete diffusion correction network, using the 2D structured source distribution as the starting point of the diffusion correction process to obtain a semantic probability distribution correction result; and determining the bird's-eye view semantic segmentation result based on the semantic probability distribution correction result. This application can obtain a more accurate bird's-eye view semantic segmentation result through fewer or even single-step correction, improving semantic segmentation accuracy while reducing inference costs.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically to a method and apparatus for semantic segmentation of radar bird's-eye view for embodied intelligence. Background Technology

[0002] With the continuous development of autonomous driving, embodied intelligent agents and mobile robot technologies, accurate three-dimensional perception and understanding of the surrounding environment has become the foundation for the stable operation of related systems.

[0003] Bird's-eye view semantic segmentation is a key task in 3D environmental perception. It primarily involves projecting sparse point cloud observations collected by sensors such as LiDAR along a vertical direction into a 2D semantic map viewed from above, and assigning corresponding semantic categories to each location within this map. Because bird's-eye views can express the spatial layout and semantic information of a scene in a compact 2D form, they have become the primary 2D intermediate representation for path planning and decision-making in autonomous vehicles. They are also a common input for the top-down world model of embodied intelligent agents, as well as for path planning and obstacle avoidance decisions by platforms such as autonomous mobile robots, inspection robots, cleaning robots, and multi-rotor drones.

[0004] However, in existing pure LiDAR BEV semantic segmentation techniques, the mIoU semantic segmentation accuracy of 2D dedicated BEV segmentation methods is generally only 12%–27%, far lower than that of multimodal methods. While diffusion-based correction methods (such as SegRefiner and BEVDiffuser) can improve semantic segmentation accuracy to some extent, they are mainly applicable to 2D images and have high costs for multi-step inference, resulting in low processing efficiency and failing to meet the timeliness requirements of LiDAR sparse data. In addition, the diffusion forward process in existing technologies does not match the semantic distribution of BEVs, leading to insufficient semantic segmentation efficiency and accuracy. Therefore, there is an urgent need to propose a novel technical solution to address at least one of the above-mentioned technical problems. Summary of the Invention

[0005] This application provides a method and apparatus for semantic segmentation of radar bird's-eye view for embodied intelligence, which can simultaneously reduce inference costs while improving semantic segmentation accuracy, thereby improving semantic segmentation accuracy and processing efficiency.

[0006] In a first aspect, embodiments of this application provide a semantic segmentation method for radar bird's-eye view images used in embodied intelligence systems. This method is applied to an embodied intelligence system and includes: The environmental observation data of the target scene is voxelized to obtain a sparse 3D point cloud to be processed. The sparse 3D point cloud is input into the base model, and features are extracted and preliminary predictions are made on the sparse 3D point cloud to obtain the 3D source prediction distribution. The 3D source prediction distribution is used to characterize the semantic probability distribution of each voxel in the 3D mesh of the target scene. The 3D source prediction distribution is projected vertically, and the semantic category corresponding to the highest non-empty voxel in the vertical direction at each 2D grid position is taken to obtain the baseline bird's-eye view semantic map. Using the one-hot encoding of the semantic graph of the baseline bird's-eye view as the two-dimensional structured source distribution, the two-dimensional structured source distribution, diffusion time step, and conditional information are input into the two-dimensional discrete diffusion correction network. The two-dimensional structured source distribution is used as the starting point of the diffusion correction process to predict the semantic probability distribution correction result. The two-dimensional discrete diffusion correction network performs the correction from the two-dimensional structured source distribution to the true bird's-eye view distribution in the probability simplex space. Based on the semantic probability distribution correction results, the semantic category of each two-dimensional grid position is determined, and the semantic segmentation result of the bird's-eye view of the target scene is obtained.

[0007] Secondly, embodiments of this application provide a radar bird's-eye view semantic segmentation device for embodied intelligence, which is applied to an embodied intelligence system. The device includes: The preprocessing module is configured to perform voxelization on the environmental observation data of the target scene to obtain a sparse 3D point cloud to be processed. The prediction module is configured to input sparse 3D point clouds into the base model, perform feature extraction and preliminary prediction on the sparse 3D point clouds, and obtain the 3D source prediction distribution; the 3D source prediction distribution is used to characterize the semantic probability distribution of each voxel in the 3D mesh of the target scene. The projection module is configured to project the 3D source prediction distribution along the vertical direction in the column direction, and take the semantic category corresponding to the highest non-empty voxel in the vertical direction of each 2D grid position to obtain the reference bird's-eye view semantic map. The correction module is configured to use the one-hot encoding of the baseline bird's-eye view semantic graph as the two-dimensional structured source distribution. The two-dimensional structured source distribution, diffusion time step, and conditional information are input into the two-dimensional discrete diffusion correction network. The two-dimensional structured source distribution is used as the starting point of the diffusion correction process to predict the semantic probability distribution correction result. The two-dimensional discrete diffusion correction network performs the correction from the two-dimensional structured source distribution to the true bird's-eye view distribution in the probability simplex space. The segmentation module is configured to determine the semantic category of each two-dimensional grid position based on the semantic probability distribution correction result, and obtain the bird's-eye view semantic segmentation result of the target scene.

[0008] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the radar bird's-eye view semantic segmentation method for embodied intelligence as described in the first aspect.

[0009] Fourthly, embodiments of this application provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the radar bird's-eye view semantic segmentation method for embodied intelligence as described in the first aspect.

[0010] In this embodiment, a 3D source prediction distribution is first obtained by completing the 3D semantic scene of the sparse 3D point cloud using a base model. Then, the 3D source prediction distribution is projected vertically to obtain a baseline bird's-eye view semantic map. The one-hot encoding of the baseline bird's-eye view semantic map is used as a 2D structured source distribution and input into a 2D discrete diffusion correction network for correction to obtain the bird's-eye view semantic segmentation result. Since this embodiment uses the baseline bird's-eye view semantic map obtained by projecting the 3D source prediction distribution as the starting point of the diffusion correction process, instead of using random noise or image-like masks as the starting point in the prior art, the 2D discrete diffusion correction network in this embodiment does not need to generate a complete semantic distribution from scratch. It only needs to learn the correction direction from the 2D structured source distribution to the true bird's-eye view distribution in the probability simplex space. This allows the 2D discrete diffusion correction network to converge with fewer correction steps, or even obtain the semantic probability distribution correction result through a single step correction. As can be seen, in this embodiment, the reference bird's-eye view semantic map itself inherits the semantic information learned by the base model in the vertical direction. The two-dimensional discrete diffusion correction network only needs to perform residual correction on it, without having to repeatedly learn the complete mapping from sparse observations to dense semantics. Since the reference bird's-eye view semantic map and conditional information are obtained from radar point clouds without relying on camera input, the bird's-eye view semantic segmentation results obtained in this embodiment can improve semantic segmentation accuracy by relying only on radar point clouds and reduce the cost of correction inference, thereby adapting to the various application requirements of autonomous driving, embodied intelligent agents, and mobile robots for semantic segmentation accuracy and processing efficiency. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the radar bird's-eye view semantic segmentation method for embodied intelligence in an embodiment of this application. Figure 2 This is a schematic diagram of the two-dimensional U-Net modified network structure according to an embodiment of this application; Figure 3 This is a schematic diagram of a bird's-eye view semantic segmentation task on Sequence 08 of the SemanticKITTI validation set, representing an embodiment of this application. Figure 4This is a schematic diagram of the radar bird's-eye view semantic segmentation device for embodied intelligence in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computing device according to an embodiment of this application. Detailed Implementation

[0012] This application provides a radar bird's-eye view semantic segmentation method and apparatus for embodied intelligence, applicable to embodied intelligence systems that perform three-dimensional perception and understanding of the surrounding environment. The embodied intelligence system may include a bird's-eye view generation device and a semantic segmentation device, which can be integrated or deployed separately. The bird's-eye view generation device is used at least to process environmental observation data of a target scene to obtain a baseline bird's-eye view semantic map. The semantic segmentation device is used to correct the input baseline bird's-eye view semantic map to obtain a bird's-eye view semantic segmentation result. The bird's-eye view generation device may be an application program that processes environmental observation data to obtain a baseline bird's-eye view semantic map, or a server or terminal device with the application installed. The semantic segmentation device may be a semantic segmentation program that corrects the baseline bird's-eye view semantic map to obtain a bird's-eye view semantic segmentation result. The semantic segmentation program may be, for example, a semantic segmentation model containing a two-dimensional discrete diffusion correction network. The semantic segmentation device may also be a terminal device with the semantic segmentation model deployed on it.

[0013] The solutions provided in this application involve technologies such as machine learning in artificial intelligence, 3D point cloud perception, semantic scene completion, and discrete diffusion models, which are specifically illustrated through the following embodiments: Machine learning, a branch of artificial intelligence, refers to the technology that enables computers to automatically learn the inherent patterns and feature representations of large amounts of sample data through algorithms, and then make predictions or decisions based on this information. Machine learning typically includes a training phase and a deployment phase: In the training phase, the model parameters are iteratively optimized based on labeled sample data to make the model output approximate the desired result. In the deployment phase, the trained model parameters are used to perform forward computation on new input data, and the corresponding prediction result is output. In the embodiments of this application, both the base model and the two-dimensional discrete diffusion correction network can be pre-trained using machine learning methods.

[0014] 3D point cloud sensing refers to the technology of using sensors such as LiDAR and depth cameras to collect 3D point cloud data of a target scene, and then processing and analyzing the 3D point cloud data to obtain the spatial geometric structure and semantic attributes of the scene. A 3D point cloud consists of discretely distributed points in space and can directly reflect the three-dimensional positional relationships of an object's surface. Due to the influence of factors such as sensor viewing angle, scanning beam, and occlusion, the collected 3D point cloud is usually sparse, meaning that only a limited spatial area is occupied by the observation points.

[0015] Semantic scene completion refers to the technique of inferring a dense 3D scene containing complete geometric structure and semantic categories based on sparse point cloud observations obtained from a limited viewpoint. In semantic scene completion, the target scene is usually divided into a regularly arranged 3D grid, which is composed of multiple voxels. Each voxel corresponds to a spatial unit in the scene, and the output of semantic scene completion is the semantic category or semantic probability distribution of each voxel.

[0016] Discrete diffusion models are generative models oriented towards discrete state spaces. These models typically include a forward process and a backward process: the forward process gradually transitions from an initial state to an intermediate state corresponding to the target distribution according to a preset time step. The backward process involves the network starting from the intermediate states and progressively predicting and reconstructing the target distribution. Unlike continuous diffusion models oriented towards continuous state spaces and based on Gaussian noise, the state values ​​of discrete diffusion models are discrete categories, making them suitable for modeling the distribution of discrete variables such as semantic categories. In this embodiment, the semantic category distribution of each voxel or grid position can be represented as points in a probabilistic simplex space, where the probabilistic simplex space is a space composed of vectors whose components are non-negative and whose sum is one.

[0017] Bird's-eye view semantic segmentation refers to the technique of projecting a target scene vertically onto a two-dimensional plane viewed from above, resulting in a bird's-eye view composed of multiple two-dimensional grid positions, and assigning a corresponding semantic category to each two-dimensional grid position in the bird's-eye view. The bird's-eye view expresses the spatial layout and semantic information of the scene in a compact two-dimensional form, and can be used as input for downstream tasks such as path planning and obstacle avoidance decision-making.

[0018] Currently, existing bird's-eye view semantic segmentation techniques can be broadly categorized into three types. The first type is discriminative two-dimensional direct regression methods, which voxelize sparse point clouds and project them onto a bird's-eye view, then use a two-dimensional convolutional network for pixel-by-pixel classification. Because this type of method directly regresses dense two-dimensional semantic maps from sparse observations, its training signals are sparse, and the number of samples between semantic categories is severely imbalanced, resulting in a significant bottleneck in improving segmentation accuracy. The second type is multimodal fusion methods, which fuse camera image features with LiDAR features before performing bird's-eye view semantic segmentation. This type of method relies on camera input and is difficult to apply when only LiDAR is deployed. The third type is diffusion-based segmentation correction methods, which use a diffusion model to denoise and correct the initial output of the segmentation model. However, the diffusion forward process of this type of method is often designed for two-dimensional image masks or continuous features, which is incompatible with the sparse distribution characteristics of LiDAR bird's-eye views, where most locations are empty and only a few locations carry high-value semantics. Furthermore, its correction process usually requires multiple backsampling steps, resulting in high inference latency, making it difficult to meet the processing efficiency requirements of real-time deployment. Furthermore, when performing semantic segmentation of bird's-eye view images, this type of method often fails to fully utilize the vertical semantic information contained in the existing 3D perception results.

[0019] In summary, existing technologies struggle to achieve sufficiently high-precision semantic segmentation results for bird's-eye view images while relying solely on LiDAR and considering processing efficiency, making this a pressing technical problem to be solved in related perception systems.

[0020] In some embodiments, the bird's-eye view generation device and the semantic segmentation device are deployed separately. The radar bird's-eye view semantic segmentation method provided in this application can be implemented by a bird's-eye view semantic segmentation system. This bird's-eye view semantic segmentation system may include a server 01 and a terminal device 02.

[0021] The server 01 can be the bird's-eye view generation device, in which a base model can be deployed, the base model being used to complete the 3D semantic scene of the sparse 3D point cloud.

[0022] The terminal device 02 can be the semantic segmentation device, which can be equipped with the semantic segmentation model, such as a semantic segmentation model trained using machine learning methods and containing a two-dimensional discrete diffusion correction network.

[0023] The terminal device 02 can forward the collected environmental observation data of the target scene to the server 01. The server 01 can perform voxelization processing on the environmental observation data to obtain a sparse 3D point cloud, input the sparse 3D point cloud into a base model to obtain a 3D source prediction distribution, and perform column projection of the 3D source prediction distribution along the vertical direction to obtain a reference bird's-eye view semantic map. Then, the reference bird's-eye view semantic map is sent to the terminal device 02. The terminal device 02 can use the one-hot encoding of the reference bird's-eye view semantic map as a 2D structured source distribution, input it into the 2D discrete diffusion correction network for correction, and the resulting bird's-eye view semantic segmentation result is the bird's-eye view semantic segmentation result of the target scene.

[0024] It should be noted that the computing devices involved in the embodiments of this application may be servers and / or terminal devices.

[0025] The server involved in the embodiments of this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0026] The terminal devices involved in the embodiments of this application may be devices that provide voice and / or data connectivity to users, handheld devices with wireless connectivity, or other processing devices connected to a wireless modem, or vehicle-mounted or airborne computing units deployed on embodied intelligent platforms such as autonomous vehicles, service robots, inspection robots, cleaning robots, or multi-rotor drones.

[0027] Reference Figure 1 , Figure 1 This is a flowchart illustrating a radar bird's-eye view semantic segmentation method for embodied intelligence, provided in an embodiment of this application. The method can be executed by a radar bird's-eye view semantic segmentation device for embodied intelligence. The method includes the following steps 101-105: Step 101: Perform voxelization on the environmental observation data of the target scene to obtain the sparse 3D point cloud to be processed.

[0028] The target scene refers to the surrounding environment in which the embodied intelligence system operates, requiring three-dimensional perception and understanding. For example, in the perception scenario of an autonomous vehicle, the target scene can be the road environment within a certain range around the vehicle, including structural targets such as road surfaces, sidewalks, buildings, and vegetation, as well as dynamic or sparse targets such as pedestrians, cyclists, motorcyclists, traffic signs, and utility poles. In the perception scenario of an embodied intelligence or indoor mobile robot, the target scene can be the indoor or operational environment within a certain range around the robot itself.

[0029] The environmental observation data refers to the observation data collected by the embodied intelligent system from the target scene using sensors such as LiDAR. For example, the environmental observation data can be LiDAR point cloud obtained by a vehicle-mounted LiDAR performing a single-frame scan of the target scene, or it can be point cloud data collected by a solid-state LiDAR or a depth camera.

[0030] Voxelization refers to dividing the three-dimensional space of the target scene into a regularly arranged three-dimensional mesh, and determining the occupancy state of each voxel based on the environmental observation data, thereby obtaining a sparse three-dimensional point cloud composed of voxels occupied by the observed points. Due to factors such as the sensor's observation angle, scanning beam, and occlusion relationships, only a limited number of voxels in the sparse three-dimensional point cloud are occupied, while a large number of other voxels are empty, hence the term sparse.

[0031] Step 101 above organizes the original environmental observation data into a regular three-dimensional grid representation, providing a unified input format for subsequent feature extraction and preliminary prediction of the base model.

[0032] Specifically, in step 101, the three-dimensional space where the target scene is located can be divided into regularly arranged three-dimensional meshes along the horizontal and vertical directions. For example, the size of the three-dimensional mesh is 256×256×32, that is, divided into 256 units in each of the two horizontal directions and 32 units in the vertical direction. In the perception scenario of an autonomous vehicle, the three-dimensional mesh can cover an area of ​​51.2 meters × 51.2 meters in the horizontal direction, with the rear axle of the vehicle as the origin. After obtaining the three-dimensional mesh, each observation point in the environmental observation data is mapped to its corresponding voxel according to its spatial position. Voxels mapped to by observation points are recorded as non-empty voxels and marked as occupied, while voxels not mapped to by any observation point are recorded as empty voxels. The resulting sparse three-dimensional point cloud, characterized by the occupancy state of each voxel in the regular three-dimensional mesh, represents the sparse geometric structure of the target scene and serves as the input to the base model in step 102.

[0033] Step 102: Input the sparse 3D point cloud into the base model, perform feature extraction and preliminary prediction on the sparse 3D point cloud, and obtain the 3D source prediction distribution.

[0034] The 3D source prediction distribution is used to characterize the semantic probability distribution of each voxel in the 3D mesh of the target scene, that is, the probability of each voxel in the 3D mesh for each candidate semantic category. The base model is a pre-trained model for 3D semantic scene completion, which can infer a dense 3D scene containing complete geometric structure and semantic category from sparse 3D point cloud. Therefore, the 3D source prediction distribution already contains the semantic information of the target scene in the vertical direction.

[0035] Optionally, the base model is a pre-trained 3D semantic scene completion model. The base model includes a parameter-frozen 3D base model and a 3D discrete diffusion correction network. The 3D discrete diffusion correction network uses the voxel-level semantic probability distribution output by the 3D base model as the starting point for the 3D diffusion correction process, performs a few steps to correct the voxel-level semantic probability distribution, and uses the corrected voxel-level semantic probability distribution as the 3D source prediction distribution. Therefore, this step can directly reuse an existing high-quality 3D semantic scene completion model as the base model, without needing to retrain the 3D perception network.

[0036] In practical applications, the base model can be any pre-trained 3D semantic scene completion model capable of outputting a voxel-by-voxel semantic category distribution. The specific network structure of the base model is not limited in the embodiments of this application.

[0037] For example, the base model can be a discriminative 3D semantic scene completion model, such as LMSCNet, S3CNet, JS3C-Net, SCPNet, or PaSCo. Taking SCPNet as an example, it distills the knowledge of a multi-frame dense teacher network into a single-frame student network, and can predict the semantic probability distribution of each voxel from a single-frame sparse 3D point cloud, which can be directly used as the base model in this step.

[0038] For example, the base model can also be a 3D semantic scene completion model with a 3D correction step. In this case, the base model includes a parameter-frozen 3D base model and a 3D discrete diffusion correction network. The 3D base model (such as SCPNet) performs preliminary predictions on the sparse 3D point cloud to obtain the semantic probability distribution of each voxel in the 3D mesh. The 3D discrete diffusion correction network uses the semantic probability distribution as the starting point of the 3D diffusion correction process, performs a few steps or even a single step correction in the probabilistic simplex space, and uses the corrected 3D semantic probability distribution as the 3D source prediction distribution. Since the 3D discrete diffusion correction network uses the prediction distribution of the 3D base model rather than random noise or a full mask as the starting point of the diffusion correction process, it only needs to learn the correction direction pointing to the true scene distribution while keeping the weights of the 3D base model frozen, which can further improve the accuracy of the 3D source prediction distribution, thereby providing higher quality input for subsequent column projection and 2D correction.

[0039] For example, the training data used during the training of the base model may include training pairs composed of real-world data, synthetic training pairs obtained through offline data augmentation, or a training set obtained by mixing the two in a preset ratio. The training pairs composed of real-world data may be sparse 3D point clouds collected from real scenes and their voxel-by-voxel annotations of semantic ground truth. The synthetic training pairs may be data pairs automatically synthesized through an offline data augmentation process, requiring no manual voxel annotation (synthetic sparse 3D point clouds, scene voxel ground truth). Since the frequency of rare categories such as pedestrians, cyclists, motorcyclists, traffic signs, and utility poles in real-world data is much lower than that of common categories such as roads, buildings, and vegetation, exhibiting a long-tail distribution, the prediction accuracy of the base model for rare categories is often limited when training the base model using only real-world data. Therefore, the synthetic training pairs can be introduced into the training data of the base model to expand the training data scale and supplement rare category samples.

[0040] Optionally, the synthetic training pair consists of a synthetic sparse 3D point cloud and its paired scene voxel ground truth values, wherein the synthetic sparse 3D point cloud and the scene voxel ground truth values ​​correspond one-to-one in space, and serve as input and supervision during the training of the base model, respectively. The synthetic training pair can be obtained through an offline multi-stage data augmentation process, which exemplarily includes the following sequentially executed processes: First, the original ground truth voxels are generated through multi-stage pyramid discrete diffusion, and complete synthetic semantic scenes are generated sequentially according to the resolution hierarchy from coarse to fine, so as to obtain a brand-new scene with complete geometric structure and semantic attributes without relying on real acquisition.

[0041] Secondly, the generated synthetic semantic scenes are subjected to multi-level hard filtering and probability distribution divergence joint screening to remove samples with geometric degradation, structural incompleteness, physical relationship misalignment, or semantic category distribution deviation from the real scene, and retain the target synthetic scene that meets the quality constraints.

[0042] Third, the selected target synthetic scene is subjected to multi-echo ray tracing sparsification, and virtual radar scanning simulation is performed by combining the hardware physical characteristics of the target radar sensor and the sensor noise distribution to obtain a synthetic sparse 3D point cloud that matches the target synthetic scene and conforms to the characteristics of real radar observation.

[0043] Fourth, based on the rare class object library, object pasting enhancement is performed on the synthesized sparse 3D point cloud and the corresponding scene voxel ground value in the target synthesized scene. While maintaining the consistency of the pairing between sparse point cloud and dense ground value, rare long-tail categories are supplemented and balanced to obtain the synthesis training pair.

[0044] Alternatively, the multi-stage data augmentation process can be executed offline once to produce synthetic training pairs of any size. The synthetic training pairs produced are then merged with the training pairs composed of the real collected data and used for training the base model.

[0045] The synthetic training pairs used in the training of the base model have several advantages. First, the synthetic training pairs are automatically generated by the multi-stage data augmentation process and come with paired scene voxel ground truth values, eliminating the need for additional manual voxel annotation of the synthetic scene. This allows for the expansion of training data scale without increasing annotation costs. Second, object pasting augmentation directly supplements and balances rare long-tail categories at the data level, enabling the base model to access more rare category samples during training. This improves the imbalance of rare long-tail categories in the training data and enhances the prediction accuracy of the base model for rare categories. Since the 3D source prediction distribution output by the base model is projected into a baseline bird's-eye view semantic map through the column-directed projection, and the one-hot encoding of the baseline bird's-eye view semantic map serves as the starting point for the subsequent 2D discrete diffusion correction process, the improvement in the prediction accuracy of the base model on rare categories will propagate along the aforementioned processing chain, correspondingly improving the final bird's-eye view semantic segmentation result, especially contributing to improving the accuracy of the bird's-eye view semantic segmentation result on rare categories.

[0046] Optionally, the 3D source prediction distribution output by the base model corresponds to a 256×256×32 3D grid, and the baseline bird's-eye view semantic map is a 256×256 2D semantic map with 20 semantic categories. This resolution is consistent with the ground truth resolution of the point cloud dataset, allowing the subsequent bird's-eye view semantic segmentation results to be directly integrated with downstream tasks.

[0047] Step 103: Project the three-dimensional source prediction distribution along the vertical direction, and for each two-dimensional grid position, take the semantic category corresponding to the highest non-empty voxel in the vertical direction of the current position to obtain the reference bird's-eye view semantic map.

[0048] In this embodiment, the 3D source prediction distribution is the prediction result output by the base model in step 102 after performing 3D semantic scene completion on the sparse 3D point cloud. It is used to characterize the semantic probability distribution of each voxel in the 3D mesh where the target scene is located, that is, the probability that each voxel in the 3D mesh belongs to each candidate semantic category. Since the base model is a 3D semantic scene completion model, the 3D source prediction distribution already contains the complete geometric structure and semantic information of the target scene in the vertical direction.

[0049] A 3D mesh is a regular grid obtained by dividing the 3D space of the target scene along two horizontal directions and one vertical direction. The 3D mesh is divided into several layers along the vertical direction, and each layer corresponds to a bird's-eye view plane composed of 2D mesh positions on the horizontal plane. The 2D mesh position is a horizontal position on the bird's-eye view plane, which corresponds to a voxel column in the vertical direction in the 3D mesh. The voxel column is composed of multiple voxels arranged from bottom to top in the vertical direction at that horizontal position.

[0050] Column projection refers to: for each 2D grid position in the 3D grid, traversing the voxel column corresponding to that position vertically, and determining the voxel with the largest vertical coordinate and a non-empty semantic category as the highest non-empty voxel at that position. The highest non-empty voxel is the voxel at the highest position in the voxel column that is predicted to belong to a certain non-empty semantic category (i.e., not belonging to the empty category representing free space), which to some extent corresponds to the object surface first observed when viewed from above at that horizontal position. Furthermore, the semantic category with the highest probability in the semantic probability distribution of the highest non-empty voxel is taken as the semantic category of the baseline bird's-eye view semantic map at that 2D grid position.

[0051] Column projection can be made into a differentiable projection through soft maximum value operation during the training phase, and can be directly implemented through hard maximum value operation during the deployment phase.

[0052] The baseline bird's-eye view semantic graph is a top-down 2D semantic graph composed of the semantic categories of the aforementioned 2D grid positions. It provides the initial semantic distribution for the subsequent 2D discrete diffusion correction process. Column projection can be made differentiable during the training phase using soft-maximization operations to facilitate gradient backpropagation. During the deployment phase, the highest non-empty voxel and its semantic category can be directly determined using hard-maximization operations.

[0053] Through step 103, the reference bird's-eye view semantic graph directly inherits the semantic information learned by the base model in the vertical direction. This is equivalent to compressing the three-dimensional semantic scene completion result into a two-dimensional semantic graph from a top-down perspective along the vertical direction. This provides a starting point that is close to the truth rather than random for the subsequent two-dimensional discrete diffusion correction, avoiding the two-dimensional network from repeatedly learning the complete mapping from sparse observations to dense bird's-eye view semantics from zero.

[0054] As an optional embodiment, in step 103, all two-dimensional grid positions are traversed, and the following processing procedure is performed on each two-dimensional grid position to obtain the reference bird's-eye view semantic map: Specifically, for each two-dimensional grid position, each voxel in the voxel column corresponding to the current position is traversed along the vertical direction, and the voxel with the largest vertical coordinate and non-empty semantic category in the voxel column is determined as the highest non-empty voxel at the current position; the semantic probability distribution of the highest non-empty voxel in the three-dimensional source prediction distribution is obtained, and the semantic category with the highest probability in the semantic probability distribution is determined as the semantic category of the reference bird's-eye view semantic map at the current position, and the process jumps to the next position. The column projection is implemented as a differentiable projection through soft maximum value operation during the training phase of the two-dimensional discrete diffusion correction network, and the semantic category corresponding to the highest non-empty voxel is determined through hard maximum value operation during the deployment phase.

[0055] Specifically, the 3D source prediction distribution corresponds to a 3D grid, and each 2D grid position on the bird's-eye view plane corresponds to a voxel column in the vertical direction. The column projection is to process each 2D grid position on the bird's-eye view plane one by one: for the current 2D grid position, traverse each voxel in its corresponding voxel column along the vertical direction to determine a voxel from the voxel column to represent the semantics of the current position.

[0056] When traversing the voxel column corresponding to the current position vertically, the voxel with the largest vertical coordinate and a non-empty semantic category is identified as the highest non-empty voxel at the current position. Here, "non-empty semantic category" means that the voxel is predicted to belong to an actual semantic category, rather than an empty category representing free space. The largest vertical coordinate indicates that the voxel is at the highest position in the voxel column. In other words, the highest non-empty voxel is the first occupied voxel encountered when viewing from top to bottom at this 2D grid position, representing the semantics of the topmost object surface at that position in the bird's-eye view.

[0057] After determining the highest non-empty voxel at the current location, the semantic probability distribution of the highest non-empty voxel in the 3D source prediction distribution is obtained. This semantic probability distribution gives the probability that the voxel belongs to each candidate semantic category. The semantic category with the highest probability in the semantic probability distribution is selected as the semantic category of the baseline bird's-eye view semantic map at the current location. Then, the process is repeated at the next 2D grid location. After traversing all 2D grid locations, the top-view 2D semantic map, arranged from the semantic categories determined at each location, is the baseline bird's-eye view semantic map.

[0058] In the training phase of the 2D discrete diffusion correction network, the aforementioned column projection can be implemented as a differentiable projection through soft-maximization, that is, the highest non-empty voxel and its semantic category are selected from the voxel column in a differentiable manner. This allows the column projection step to participate in gradient backpropagation, facilitating end-to-end training of the 2D discrete diffusion correction network. In the deployment phase, the highest non-empty voxel and its corresponding semantic category can be directly determined through hard-maximization, reducing the computational overhead during deployment. Thus, the column projection remains differentiable during training and remains simple and efficient during deployment, which is consistent in principle.

[0059] For example, suppose the sparse 3D point cloud X to be processed is a 3D SSC underlying model g with parameters frozen. (i.e., the base model), firstly through the three-dimensional SSC base model g Obtain the corresponding voxel-level probability prediction (i.e., three-dimensional source prediction distribution).

[0060] Furthermore, for each two-dimensional grid location Traverse 32 voxels vertically and select the voxel with the largest vertical coordinate and a non-empty semantic category. Using the semantic category with the highest probability in the 3D source prediction distribution of the voxel as the baseline, the bird's-eye view semantic map is positioned at the current 2D grid location. Category tags ,Right now:

[0061] in, The semantic map of the baseline bird's-eye view is located in the two-dimensional grid. The semantic category label at the position, i.e., the output of the column projection. Predicting the distribution of three-dimensional sources in voxels semantic category The predicted probability is derived from the three-dimensional SSC basic model g. The output is the 3D semantic scene completion of the sparse 3D point cloud X. For example, the 3D source prediction distribution corresponds to a 256×256×32 3D grid and 20 semantic categories. ∈ K=20. For position The subscript of the highest non-empty voxel in the vertical direction is defined as: = max{ z∈{0, …, 31} : ≠ empty}, that is, the vertical index of the voxel with the largest vertical coordinate in the voxel column corresponding to the position and which is predicted to belong to a certain non-empty semantic category. Take Largest semantic category That is, the category with the highest probability in the semantic probability distribution at the highest non-empty voxel is taken as the category label of the baseline bird's-eye view semantic map at that position.

[0062] The projection described above is a parameterless operation. During the training phase, it can be implemented as a differentiable projection using soft argmax to facilitate gradient backpropagation. During the deployment phase, hard argmax can be used directly to determine the semantic category corresponding to the highest non-empty voxel, thereby reducing computational overhead. The resulting baseline bird's-eye view semantic graph directly inherits the semantic information learned by the base model in the vertical direction.

[0063] For example, on the SemanticKITTI validation set, the benchmark bird's-eye view semantic map obtained by the above projection can achieve a mIoU of 34.27%, which is 7.24 percentage points higher than the previous strongest pure 2D dedicated method 2D S3CNet (27.03%). After the 2D discrete diffusion correction in step 104, the bird's-eye view semantic segmentation result can be further improved to 36.09% mIoU, which is 9.06 percentage points higher than the previous strongest pure LiDAR 2D dedicated method.

[0064] Step 104: Using the one-hot encoding of the reference bird's-eye view semantic graph as a two-dimensional structured source distribution, the two-dimensional structured source distribution, diffusion time step, and conditional information are input into a two-dimensional discrete diffusion correction network. The two-dimensional structured source distribution is used as the starting point of the diffusion correction process to predict the semantic probability distribution correction result.

[0065] In this embodiment, the two-dimensional discrete diffusion correction network performs a correction from the two-dimensional structured source distribution to the truth bird's-eye view distribution within the probabilistic simplex space.

[0066] For each two-dimensional grid location in the bird's-eye view, its semantic category can be represented by a probability vector. Each component of the probability vector corresponds to the probability of each candidate semantic category, and each component has a non-negative value and a sum of one, meaning the probability vector lies within the probabilistic simplex space. The probability vectors of all two-dimensional grid locations in the bird's-eye view constitute a semantic probability distribution of the bird's-eye view within the probabilistic simplex space.

[0067] The two-dimensional structured source distribution is a semantic probability distribution obtained by one-hot encoding of the baseline bird's-eye view semantic graph. It represents the preliminary prediction of the semantic category of each two-dimensional grid position by the base model after column-directed projection. The ground truth bird's-eye view distribution is the semantic probability distribution corresponding to the ground truth bird's-eye view semantic graph of the target scene. It represents the true semantic category of each two-dimensional grid position. Both are located in the same probabilistic simplex space.

[0068] The correction performed by the 2D discrete diffusion correction network is a process of gradually adjusting the 2D structured source distribution within the probabilistic simplex space to approximate the true bird's-eye view distribution, starting from the 2D structured source distribution. Since the correction process always takes place within the probabilistic simplex space, each intermediate result satisfies the non-negativity and normalization constraints of the semantic probability distribution. Therefore, mathematically, the correction corresponds to a migration from one semantic probability distribution to another within the probabilistic simplex space, rather than a modification of pixel values ​​or continuous features.

[0069] In this embodiment, one-hot encoding refers to constructing a probability vector with a dimension equal to the total number of semantic categories for each two-dimensional grid position in the reference bird's-eye view semantic graph. The component corresponding to the semantic category of the current position is set to one, and the remaining components are set to zero. The resulting probability vector is the one-hot encoded vector for the current position. The probability vector is a vertex of the probabilistic simplex space.

[0070] Since the probabilistic simplex space is the space formed by all probability vectors whose components are non-negative and whose sum is one, when a probability vector has only one component of one and all other components of zero, the probability vector falls exactly at the boundary extremum of the probabilistic simplex space, which corresponds to a vertex of the probabilistic simplex space. The vertex indicates that the position of the two-dimensional grid belongs to a certain semantic category with probability one and does not belong to any other semantic categories at all. It is a deterministic and unambiguous semantic category representation.

[0071] In other words, the two-dimensional structured source distribution obtained by arranging the one-hot encoded vectors at each position of the semantic map of the baseline bird's-eye view according to the two-dimensional grid position is a deterministic distribution composed of several vertices in the probabilistic simplex space. Using this deterministic distribution as the starting point of the diffusion correction process is equivalent to starting the correction from a clear preliminary semantic judgment, which is different from the method of starting from a non-deterministic distribution located inside the probabilistic simplex space, such as random noise or uniform distribution.

[0072] In this embodiment of the application, the condition information includes a bird's-eye view occupancy map obtained by aggregating the sparse 3D point cloud column by column along the vertical direction onto the bird's-eye view plane. The bird's-eye view occupancy map is used to characterize the occupancy status of each 2D grid position of the sparse 3D point cloud on the bird's-eye view plane.

[0073] Specifically, the sparse 3D point cloud is distributed within a 3D mesh. Each 2D mesh position on the bird's-eye view plane corresponds to a voxel column in the vertical direction. Vertical column-by-column aggregation refers to statistically aggregating the occupancy status of each voxel in its corresponding voxel column for each 2D mesh position, and using the aggregation result as the value of the bird's-eye view occupancy map at that 2D mesh position. For example, when a voxel in a certain voxel column is occupied by the sparse 3D point cloud, the value of the bird's-eye view occupancy map at the corresponding 2D mesh position can be marked as occupied; otherwise, it can be marked as unoccupied, thus obtaining the bird's-eye view occupancy map on the bird's-eye view plane.

[0074] The bird's-eye view occupancy map reflects the actual coverage of the original observations on the bird's-eye view plane. Inputting this as conditional information into the two-dimensional discrete diffusion correction network can provide spatial constraints from the original observations for the correction process, guiding the two-dimensional discrete diffusion correction network to make more reliable corrections in the areas actually covered by the observations.

[0075] Unlike existing diffusion-based correction methods that use random noise or image-based masks as the starting point of the diffusion process, this step uses the one-hot encoding of the semantic graph of the baseline bird's-eye view obtained by projecting the three-dimensional source prediction distribution as the structured starting point. This allows the two-dimensional discrete diffusion correction network to learn only the correction direction from the structured source distribution to the true bird's-eye view distribution, i.e., to perform residual correction on a distribution that is close to the true value. Therefore, it can converge with fewer correction steps, and even obtain the semantic probability distribution correction result through a single-step correction, thus reducing the inference cost.

[0076] As an optional embodiment, the two-dimensional discrete diffusion correction network is a two-dimensional U-Net with skip connections. The two-dimensional U-Net includes four sequentially connected downsampling layers, a bottleneck layer, and four upsampling layers. Based on this, the input channels of the two-dimensional discrete diffusion correction network include: the semantic category channel containing the two-dimensional structured source distribution, the time step embedding channel obtained by sinusoidal embedding of the diffusion time step, and the condition channel containing the conditional information. The output of the two-dimensional discrete diffusion correction network is the bird's-eye view semantic probability distribution on the probability simplex space, and this bird's-eye view semantic probability distribution is used to determine the semantic probability distribution correction result.

[0077] The 2D U-Net is a convolutional network with an encoder-decoder structure. The four downsampling layers constitute the encoder, progressively reducing the spatial resolution of the feature maps along the network depth direction while extracting semantic features from local to global perspectives. The four upsampling layers constitute the decoder, progressively restoring the spatial resolution of the feature maps along the network depth direction until it reaches the same spatial size as the bird's-eye view plane. The bottleneck layer, located between the encoder and decoder at the lowest spatial resolution, is used to further fuse features within the global receptive field. Skip connections are positioned between the downsampling layers and the upsampling layers with corresponding spatial resolutions. These connections transfer and concatenate the high-resolution features retained during the downsampling stage to the corresponding layers in the upsampling stage. This allows the 2D discrete diffusion correction network to retain local details and boundary information while utilizing global semantic information, thereby obtaining more accurate local boundaries while maintaining the overall structure of the bird's-eye view. It should be noted that the configuration of the four downsampling layers, bottleneck layer, and four upsampling layers is exemplary and not the only limitation on the number of layers in the 2D U-Net.

[0078] For example, suppose One-hot encoding of the semantic graph of the baseline bird's-eye view. This is a one-hot encoding of the truth view semantic graph. During the structured source forward pass, the one-hot encoding of the truth view semantic graph and the two-dimensional structured source distribution are deterministically linearly interpolated in the probability simplex space to construct the diffusion intermediate states for training.

[0079] in, The two-dimensional structured source distribution is obtained by arranging the one-hot encoded vectors of each two-dimensional grid position in the semantic graph of the baseline bird's-eye view according to their positions. This is the one-hot encoding corresponding to the truth-value bird's-eye view semantic graph. To be consistent with diffusion time step The corresponding interpolation coefficients satisfy = 1、 = 0 and follow The increase of is monotonically decreasing. When When = 0, the diffusion intermediate state is exactly equal to the one-hot encoding of the truth view semantic graph. When t = T, the intermediate diffusion state is exactly equal to the two-dimensional structured source distribution. .because and All lie in the probabilistic simplex space Inside, its convex combination Always in Therefore, the intermediate states of diffusion satisfy the non-negativity and normalization constraints of the semantic probability distribution at any diffusion time step, without the need for additional projection operations.

[0080] For example, refer to Figure 2 The two-dimensional U-Net correction network structure shown takes the current iteration state x_t as input and outputs a fine-grained bird's-eye view semantic probability distribution. _0. The network consists of three parts: an encoder, a bottleneck layer, and a decoder, each with skip connections. The encoder comprises four sequentially connected downsampling layers: the first layer (E0) has a feature map spatial resolution of 256×256 and 32 channels; the second layer (E1) has a spatial resolution of 128×128 and 64 channels; the third layer (E2) has a spatial resolution of 64×64 and 128 channels; and the fourth layer (E3) has a spatial resolution of 32×32 and 256 channels. Along the depth direction of the encoder, the spatial resolution of the feature map is halved layer by layer, while the number of channels is doubled layer by layer, in order to extract semantic features from local to global.

[0081] Figure 2 In the middle, the bottleneck layer (B) has a spatial resolution of 16×16 and 512 channels. It is located between the encoder and decoder and at the lowest resolution of the network. It is used to fuse global semantic features under the maximum receptive field.

[0082] Figure 2 In the decoder, there are four upsampling layers symmetrical to the encoder: D3 has a spatial resolution of 32×32 and 256 channels; D2 has a spatial resolution of 64×64 and 128 channels; D1 has a spatial resolution of 128×128 and 64 channels; and D0 has a spatial resolution of 256×256 and 32 channels. Along the decoder direction, the spatial resolution of the feature map is restored layer by layer, ultimately restoring it to the same 256×256 spatial size as the input bird's-eye view plane.

[0083] Figure 2 In the code, skip connections are provided between each layer of the encoder and the corresponding resolution layer of the decoder. Figure 2 (Represented by dashed arrows in the middle) is used to directly transfer and stitch the high-resolution detail features preserved in the downsampling stage to the corresponding layer in the upsampling stage, so as to preserve local boundary and detail information while restoring spatial resolution.

[0084] Figure 2 In this network, the input x_t is a multi-channel feature map, which is composed of three types of channels concatenated along the channel dimension: K semantic category channels (carrying the two-dimensional structured source distribution of the current iteration, K=20), a time-step embedding channel obtained by broadcasting the diffusion time step to space after sinusoidal embedding, and a conditional channel (carrying the occupancy map of the LiDAR bird's-eye view). The output of this network is... _0 represents the semantic probability distribution of the detailed bird's-eye view, with a spatial size of 256×256. Each two-dimensional grid position outputs a K-dimensional semantic probability vector, located in the probability simplex space. Inside.

[0085] It should be noted that the above configuration of four downsampling layers, bottleneck layer and four upsampling layers is an exemplary configuration and is not the only limitation on the number of layers in the two-dimensional U-Net.

[0086] The training objective of the above-mentioned two-dimensional U-Net correction network can be expressed as: .

[0087] in, As the KL posterior loss, the KL divergence between the estimated distribution of the output of the two-dimensional discrete diffusion correction network and the one-hot encoding of the ground-value view semantic graph is calculated on the bird's-eye view domain. This serves as the main training objective, making the semantic probability distribution of the network output approximate the true semantic category at each two-dimensional grid position. Using Lovász-Softmax 2D surface loss, additional constraints are imposed on the semantic category boundary regions in the bird's-eye view domain, improving the accuracy of the corrected bird's-eye view semantic segmentation results at category boundaries. The weighting coefficient for the Lovász-Softmax 2D surface loss is, for example, set to 0.3. Bottleneck layer-assisted classification loss introduces classification supervision at the bottleneck layer of the two-dimensional U-Net, enabling the global features extracted by the bottleneck layer to have semantic classification capabilities, which helps to improve the network's perception of the global semantic structure. The weight coefficients for the bottleneck layer's auxiliary classification loss are, for example, taken as 5 × 10. -4 . The trainable parameters of the two-dimensional discrete diffusion correction network are updated by backpropagation of the joint loss described above. The above weight configuration enables... As the primary optimization objective and As an auxiliary constraint, it ensures the accuracy of the main semantic category prediction while taking into account the accuracy of the category boundary and the expressive power of the bottleneck layer.

[0088] In this embodiment, the input of the two-dimensional discrete diffusion correction network is composed of the semantic category channel, the time step embedding channel, and the conditional channel concatenated along the channel dimension. The semantic category channel carries the two-dimensional structured source distribution. The number of channels in the semantic category channel is equal to the total number of semantic categories, and each channel corresponds to a candidate semantic category. The semantic category channel as a whole represents the current semantic probability distribution of each two-dimensional grid position in the bird's-eye view, and is the object to be corrected by the two-dimensional discrete diffusion correction network, i.e., the starting point of the diffusion correction process. In the case of single-step deployment, the two-dimensional structured source distribution is the one-hot encoding of the reference bird's-eye view semantic graph. The time step embedding channel carries the diffusion time step information. After sinusoidal embedding, the diffusion time step is expanded to the same spatial size as the bird's-eye view plane, enabling the two-dimensional discrete diffusion correction network to perceive the current diffusion time step and output a correction adapted to that diffusion time step. The conditional channel carries the conditional information, such as the bird's-eye view occupancy map obtained by vertically aggregating the sparse 3D point cloud. This conditional information provides spatial constraints from the original observations for the correction process, guiding the 2D discrete diffusion correction network to make more reliable corrections in the areas covered by the original observations. The three types of channels, after being concatenated, constitute the complete input for a single forward computation of the 2D discrete diffusion correction network, enabling the network to utilize the preliminary semantic judgment to be corrected, the current diffusion time step, and the spatial constraints of the original observations simultaneously during correction.

[0089] The output of the 2D discrete diffusion correction network provides the probability that each 2D grid position in the bird's-eye view belongs to each candidate semantic category. For each 2D grid position, the components of the output probability vector are non-negative and sum to 1, meaning they lie within the probability simplex space, thus the output is a valid semantic probability distribution. For example, normalization can be performed at the end of the 2D discrete diffusion correction network to ensure the output falls within the probability simplex space. The bird's-eye view semantic probability distribution is used to determine the semantic probability distribution correction result. In a single-step deployment, the bird's-eye view semantic probability distribution can be directly determined as the semantic probability distribution correction result. In a multi-step iterative deployment, the bird's-eye view semantic probability distribution serves as an estimate of the ground truth bird's-eye view distribution for the current correction step, participating in subsequent closed-form updates and simplex projections. The semantic probability distribution correction result is then processed in step 105 to select the semantic category with the highest probability in the semantic probability distribution for each 2D grid position, resulting in the bird's-eye view semantic segmentation result of the target scene.

[0090] As an optional embodiment, it is assumed that the conditional information includes the bird's-eye view occupancy map of the LiDAR. Based on this, before inputting the two-dimensional structured source distribution, diffusion time step, and conditional information into the pre-trained two-dimensional discrete diffusion correction network in step 104, the sparse three-dimensional point cloud can be aggregated column by column along the vertical direction to the bird's-eye view plane to obtain the bird's-eye view occupancy map, and the bird's-eye view occupancy map is used as the conditional information; the two-dimensional structured source distribution, the time step features obtained after mapping the diffusion time step, and the bird's-eye view occupancy map are concatenated to obtain the input of the two-dimensional discrete diffusion correction network.

[0091] Specifically, firstly, a sparse 3D point cloud is distributed within a 3D grid. Each 2D grid position on the bird's-eye view plane corresponds to a voxel column in the vertical direction. For each 2D grid position, the voxels in its corresponding voxel column are traversed along the vertical direction. It is counted whether there are voxels in the voxel column occupied by the sparse 3D point cloud. If so, the value of the bird's-eye view occupancy map at that 2D grid position is marked as occupied; otherwise, it is marked as unoccupied. After traversing all 2D grid positions, a bird's-eye view occupancy map with the same spatial size as the bird's-eye view plane is obtained. This bird's-eye view occupancy map is a single-channel occupancy map, and its position-by-position reflection reflects the actual coverage of the original sparse observations on the bird's-eye view plane, which is different from the 2D structured source distribution representing semantic categories predicted by the base model.

[0092] Furthermore, the diffusion time step is mapped to obtain a time step feature vector representing the current diffusion time step. For example, the diffusion time step can first be sinusoidally embedded and then mapped using a multilayer perceptron to obtain the time step feature vector. Since both the two-dimensional structured source distribution and the bird's-eye view occupancy map are two-dimensional features arranged position-by-position on the bird's-eye view plane, and the time step feature vector has no spatial dimension, the time step feature vector can also be broadcast along the spatial dimension, expanding it into a time step feature with the same spatial size as the bird's-eye view plane, so as to align with the two-dimensional structured source distribution and the bird's-eye view occupancy map in the channel dimension.

[0093] Finally, the two-dimensional structured source distribution, the broadcast time-step features, and the bird's-eye view occupancy map are concatenated along the channel dimension to obtain the input of the two-dimensional discrete diffusion correction network. The number of channels occupied by the two-dimensional structured source distribution equals the total number of semantic categories. The channels occupied by the bird's-eye view occupancy map provide spatial constraints for the original observations, and the channels containing the time-step features provide information about the current diffusion time step. The input obtained through channel concatenation simultaneously carries the preliminary semantic distribution to be corrected, the original observation coverage, and the current diffusion time step information. This serves as the input for a single forward computation of the two-dimensional discrete diffusion correction network, enabling it to correct the two-dimensional structured source distribution under the combined constraints of the bird's-eye view occupancy map and the diffusion time step.

[0094] As an optional embodiment, step 104, using the one-hot encoding of the reference bird's-eye view semantic graph as the two-dimensional structured source distribution, includes: obtaining the semantic categories at each two-dimensional grid position in the reference bird's-eye view semantic graph; for each two-dimensional grid position, constructing a probability vector with a dimension equal to the total number of semantic categories, setting the component in the probability vector that matches the semantic category of the current position to 1, and setting the remaining components that do not match the semantic category of the current position to 0, to obtain the one-hot encoding vector of the current position; the one-hot encoding vector is a vertex of the probabilistic simplex space; arranging the one-hot encoding vectors of each two-dimensional grid position according to the two-dimensional grid position to obtain the two-dimensional structured source distribution.

[0095] It is understandable that the two-dimensional structured source distribution and the truth bird's-eye view distribution are located in the same probability simplex space and serve as the starting point for the diffusion correction process.

[0096] The one-hot encoded vector is a vertex of the probabilistic simplex space. The probabilistic simplex space is the space formed by all probability vectors whose components are non-negative and whose sum is one. When a probability vector has only one component of one and all other components of zero, the probability vector falls exactly at a vertex of the probabilistic simplex space, indicating that the current two-dimensional grid position belongs to a certain semantic category with probability one and does not belong to any other semantic category.

[0097] For example, assuming the total number of semantic categories is 20, for a certain two-dimensional grid position, if its semantic category in the reference bird's-eye view semantic map is the third category, then a 20-dimensional probability vector is constructed for that position. The component corresponding to the third category in this probability vector is set to 1, and the remaining nineteen components are all set to 0. The resulting 20-dimensional vector is the one-hot encoded vector for that position. The above processing is performed on all two-dimensional grid positions in the reference bird's-eye view semantic map, and the results are arranged according to the two-dimensional grid positions to obtain the two-dimensional structured source distribution.

[0098] The two-dimensional structured source distribution and the ground truth bird's-eye view distribution reside in the same probabilistic simplex space: the ground truth bird's-eye view distribution is the semantic probability distribution corresponding to the semantic graph of the target scene's ground truth bird's-eye view, while the two-dimensional structured source distribution is the semantic probability distribution corresponding to the preliminary prediction of semantic categories at each location after column projection of the base model. Both are formed by arranging probability vectors at each two-dimensional grid location. Therefore, the two-dimensional structured source distribution can serve as the starting point for the diffusion correction process, which proceeds gradually towards the ground truth bird's-eye view distribution within the probabilistic simplex space, starting from the two-dimensional structured source distribution.

[0099] Furthermore, the two-dimensional structured source distribution, diffusion time step, and conditional information are input into a two-dimensional discrete diffusion correction network. Using the two-dimensional structured source distribution as the starting point of the diffusion correction process, the semantic probability distribution correction result is predicted. Depending on the number of correction steps used during deployment, it can be divided into the following two cases: Scenario 1: Single-step inference deployment Under single-step inference deployment, the total number of steps in the diffusion correction process is set to 1, and the diffusion time step is set to the maximum diffusion time step. The two-dimensional structured source distribution, the maximum diffusion time step, and the conditional information are input into the two-dimensional discrete diffusion correction network. An intermediate result output by the two-dimensional discrete diffusion correction network is obtained through a single forward computation. The intermediate result is normalized to obtain an estimated distribution located in the probability simplex space. The estimated distribution is the prediction of the ground truth bird's-eye view distribution by the two-dimensional discrete diffusion correction network. The estimated distribution is directly determined as the semantic probability distribution correction result.

[0100] The two-dimensional discrete diffusion correction network is a network obtained without distillation training, and the output of the single forward computation is the average correction direction from the two-dimensional structured source distribution to the ground truth bird's-eye view distribution.

[0101] In other words, since the correction path from the two-dimensional structured source distribution to the truth bird's-eye view distribution in the probability simplex space is a straight line, the two-dimensional discrete diffusion correction network, after being trained on multiple diffusion time steps, can directly give the average correction direction along the correction path. Therefore, without the need for additional distillation training, the estimated distribution of the truth bird's-eye view distribution can be obtained from the two-dimensional structured source distribution with only one forward computation, and the semantic probability distribution correction result can be obtained accordingly, thereby reducing the inference cost.

[0102] Scenario 2: Multi-step iterative deployment In a multi-step iterative deployment, the diffusion correction process is divided into multiple consecutive correction sub-steps. The time scheduling coefficients corresponding to each correction sub-step are determined, and the two-dimensional structured source distribution is used as the current state of the first correction sub-step. For each correction sub-step, the current state of the current correction sub-step, the corresponding diffusion time step, and the conditional information are input into the two-dimensional discrete diffusion correction network. Forward computation yields the estimated distribution of the current correction sub-step, which is a prediction of the true bird's-eye view distribution. The distribution difference between the estimated distribution and the two-dimensional structured source distribution is calculated; this difference indicates the correction direction from the two-dimensional structured source distribution to the true bird's-eye view distribution. The time scheduling coefficients corresponding to the current correction sub-step and the time scheduling coefficients corresponding to the next adjacent correction sub-step in the time progression direction are obtained. The coefficient difference between the time scheduling coefficients corresponding to the next correction sub-step and the time scheduling coefficients corresponding to the current correction sub-step is calculated. Multiply the distribution difference by the coefficient difference to obtain the state correction increment for the current correction sub-step, and add the current state to the state correction increment to obtain the update result for the current correction sub-step. Perform simplex projection on the update result to obtain the projected state for the current correction sub-step. Finally, use the projected state as the current state for the next correction sub-step, iterate until a preset number of correction sub-steps, and then determine the projected state of the last correction sub-step as the semantic probability distribution correction result.

[0103] Specifically, within the probabilistic simplex space, the forward path from the two-dimensional structured source distribution to the true bird's-eye view distribution is a straight line, and the current state, the estimated distribution, and the two-dimensional structured source distribution satisfy a linear relationship. Each correction substep does not require re-predicting the complete target distribution from the current state; instead, it only needs to predict the estimated distribution of the true bird's-eye view distribution using the two-dimensional discrete diffusion correction network. The difference between the estimated distribution and the two-dimensional structured source distribution is used as the correction direction, and the difference in time scheduling coefficients between adjacent correction substeps is used as the correction step size. Multiplying these two values ​​yields the state correction increment for the current correction substep, which is then added to the current state to complete the update for that substep. Thus, each correction substep proceeds along the same straight path from the two-dimensional structured source distribution to the true bird's-eye view distribution, with only the magnitude of the advancement determined by the difference in time scheduling coefficients, thereby ensuring that the entire iterative process steadily converges towards the true bird's-eye view distribution.

[0104] Specifically, since the update result of each correction substep is obtained by directly adding the current state and the state correction increment, it is a closed update. No additional multi-step state transition calculations are required within each substep, resulting in low computational cost per correction substep. Furthermore, after each correction substep update, a simplex projection is performed, truncating the negative components in the update result to zero and re-normalizing them. This ensures that each intermediate state always satisfies the non-negativity and normalization constraints of the semantic probability distribution and always lies within the probability simplex space. This avoids illegal distributions that do not satisfy probability constraints due to accumulated numerical errors from iterative updates.

[0105] Specifically, since the forward path is a straight line, when the correction along this path accumulates over multiple correction sub-steps, its sampling error is controlled by the upper bound of the prediction error of each correction sub-step, and will not be gradually amplified as the number of correction sub-steps increases. Multi-step iterative deployment can further approximate the true value bird's-eye view distribution and obtain higher correction accuracy as the number of correction sub-steps increases, while maintaining stability in the correction process. In contrast, single-step inference deployment takes the number of correction sub-steps to one and completes the correction in only one forward computation, trading slightly lower accuracy for lower inference cost. Thus, multi-step iterative deployment and single-step inference deployment are based on the same two-dimensional discrete diffusion correction network and the same forward path, and the number of correction sub-steps can be flexibly selected according to different requirements for accuracy and inference cost during deployment, without the need to retrain the two-dimensional discrete diffusion correction network.

[0106] For example, in a multi-step iterative deployment, a two-dimensional structured source distribution is used as the starting point for iteration, and updates are performed step by step according to a closed-form update formula, where the closed-form update formula is expressed as x t-1 ² = x t ² +( t-1 t )·( 0² x src ), where x t ² Let x be the current state of the current correction substep. t-1 ² To update the results, 0² x represents the predicted (i.e., estimated) semantic probability distribution of the bird's-eye view at the current diffusion time step by the two-dimensional discrete diffusion correction network. src For two-dimensional structured source distribution, t and t-1 These are the time scheduling coefficients corresponding to the current correction substep and the next correction substep, respectively. Wherein, ( 0² x src That is, the difference in distribution. t-1 t That is, the coefficient difference. Multiplying the two yields the state correction increment, which is then multiplied by the current state x. t ² Adding them together gives the updated result x t-1 ² .

[0107] For example, the simplex projection includes: truncating the negative values ​​in the update result to 0, and then normalizing the result according to the L1 norm of the update result. The projected state satisfies the non-negativity and normalization constraints of the semantic probability distribution. Specifically, the negative components in the update result are truncated to 0, and then normalized according to the L1 norm of the truncated result, so that all components of the projected state are non-negative and their sum is one. This ensures that the intermediate state obtained in each correction substep is still within the probabilistic simplex space, avoiding illegal distributions that do not satisfy the probability constraints due to iterative updates.

[0108] For example, the preset number of correction substeps can be 2, meaning two correction substeps are executed during the diffusion correction process. The number of correction substeps can also be set to other values ​​depending on the accuracy and inference cost requirements during deployment. In offline scenarios with ample computing budgets, the number of correction substeps can be appropriately increased to further improve accuracy. In scenarios with high real-time requirements, single-step inference deployment can be used.

[0109] As an optional embodiment, the forward process adopted by the two-dimensional discrete diffusion correction network during the training phase includes: obtaining the one-hot encoding of the ground truth view semantic graph corresponding to the training samples, and the two-dimensional structured source distribution obtained by column projection of the training samples; determining the interpolation coefficients corresponding to each diffusion time step in the diffusion time step; the interpolation coefficients changing between 0 and 1 as the diffusion time step progresses; multiplying the one-hot encoding of the ground truth view semantic graph with the interpolation coefficients to obtain a first weighted distribution; and subtracting the interpolation coefficients from 1. The difference is multiplied by the two-dimensional structured source distribution to obtain a second weighted distribution; the first weighted distribution is added to the second weighted distribution to obtain the diffusion intermediate state corresponding to the current diffusion time step; the diffusion intermediate state is located in the probability simplex space between the truth view semantic graph and the two-dimensional structured source distribution; the diffusion intermediate state, the current diffusion time step, and the conditional information are input into the two-dimensional discrete diffusion correction network, so that the two-dimensional discrete diffusion correction network learns the correction direction from the two-dimensional structured source distribution to the truth view semantic graph.

[0110] When performing the forward pass for each training sample, two basic elements are first obtained: First, the one-hot encoding of the ground truth bird's-eye view semantic graph corresponding to the training sample. This one-hot encoding is the vertex representation of the ground truth bird's-eye view semantic graph of the target scene in the probabilistic simplex space, where only the component corresponding to the true class is 1 at each two-dimensional grid position, and the rest are 0. Second, the two-dimensional structured source distribution obtained by projecting the training sample into the column direction. This two-dimensional structured source distribution is the one-hot encoding of the baseline bird's-eye view semantic graph obtained by predicting the sparse three-dimensional point cloud of the training sample using the basis model and then projecting it into the column direction. It resides in the same probabilistic simplex space as the one-hot encoding of the ground truth bird's-eye view semantic graph.

[0111] For each diffusion time step, interpolation coefficients are determined. These interpolation coefficients monotonically change between zero and one as the diffusion time step progresses: when the diffusion time step is zero, the interpolation coefficient is one, and the intermediate diffusion state is exactly equal to the one-hot encoding of the truth view semantic graph. When the diffusion time step reaches its maximum value, the interpolation coefficient is zero, and the intermediate diffusion state is exactly equal to the two-dimensional structured source distribution. At intermediate diffusion time steps, the interpolation coefficients take a midpoint between 0 and 1, ensuring that the intermediate diffusion state lies within the probability simplex space between the truth view semantic graph and the two-dimensional structured source distribution.

[0112] For example, the diffusion intermediate state is represented as x t ² = t ·x0² + (1 t )·x src , where x0² For the one-hot encoding of the truth-view semantic graph, x src For the structured source distribution, t For the interpolation coefficients corresponding to the diffusion time step t, and t It monotonically decreases as t increases, and satisfies the following at the boundary. 0=1 T=0. The above interpolation is performed in the probability simplex space: since x0² Both x_src and x_t are points in this space, and their convex combination xt² Located within this space, the intermediate states of diffusion satisfy the non-negativity and normalization constraints of the semantic probability distribution at any diffusion time step.

[0113] The diffusion intermediate state x t ² The current diffusion time step t and the conditional information are input into the two-dimensional discrete diffusion correction network, which predicts the estimated distribution of the ground truth view semantic map. The network calculates the loss based on the difference between the estimated distribution and the ground truth view semantic map, and updates the network parameters through backpropagation, so that the two-dimensional discrete diffusion correction network learns the correction direction from the two-dimensional structured source distribution to the ground truth view semantic map.

[0114] Because in the aforementioned forward process, the diffusion intermediate state x t ² Always located at x src With x0² On the connection line, the learning objective of the two-dimensional discrete diffusion correction network at any diffusion time step is always x_src pointing to x0². The correction direction is clear, thus the training signal is well-defined and converges stably. Furthermore, since the two-dimensional structured source distribution is already close to the true value, compared to the forward process starting with random noise or a full mask, the correction magnitude corresponding to the correction direction is smaller. The network's learning task is to make residual corrections to the initial predictions close to the true value, rather than reconstructing the complete semantic distribution from zero.

[0115] For example, during training, the exponential moving average weights of the two-dimensional discrete diffusion correction network parameters can be maintained simultaneously, and these exponential moving average weights can be used to update the weights during the deployment phase instead of the gradient update in the last step, in order to obtain a more stable correction effect. The two-dimensional discrete diffusion correction network trained as described above is the same network used in both the single-step inference deployment in Case 1 and the multi-step iterative deployment in Case 2, and the two do not need to be trained separately.

[0116] Optionally, the two-dimensional discrete diffusion correction network is trained using a synthetic training pair, which consists of a synthetic sparse 3D point cloud and paired scene voxel ground truth values. Based on this, the synthetic sparse 3D point cloud is projected onto the base model and the column direction to obtain a synthetic baseline bird's-eye view semantic map; the ground truth bird's-eye view semantic map obtained by projecting the scene voxel ground truth values ​​onto the column direction is used as supervision to train the two-dimensional discrete diffusion correction network.

[0117] The two-dimensional discrete diffusion correction network is trained using a joint loss mechanism. This joint loss includes the KL posterior loss computed on the bird's-eye view domain, the Lovász-Softmax two-dimensional surface loss, and the bottleneck layer auxiliary classification loss. For example, the weight of the Lovász-Softmax two-dimensional surface loss is 0.3, and the weight of the bottleneck layer auxiliary classification loss is 5 × 10⁻⁶. -4The above weight configuration makes the KL posterior loss the primary training objective, while the boundary loss and auxiliary loss serve as supplementary constraints. This ensures accurate prediction of the main semantic categories while improving the accuracy of category boundaries and the semantic representation capability of the bottleneck layer.

[0118] Further optionally, the synthetic training pair is obtained by performing multi-stage pyramid discrete diffusion generation on the original ground truth voxels, performing multi-level hard filtering and probability distribution divergence joint screening on the generated synthetic semantic scene, performing multi-echo ray tracing sparsification on the screened target synthetic scene, and performing object pasting enhancement based on a rare class object library.

[0119] The process involves several key steps. First, a multi-stage pyramid discrete diffusion generation method generates complete synthetic semantic scenes sequentially, from coarse to fine resolution, ensuring the scenes possess complete geometric structure and semantic attributes. Second, multi-level hard filtering and probability distribution divergence are used to jointly filter and eliminate samples with geometrically degraded or semantically dissimilar distributions, retaining only the target synthetic scenes that meet quality constraints. Third, multi-echo ray tracing sparsity is achieved by combining the hardware physical characteristics of the target radar sensor with sensor noise distribution to simulate virtual radar scanning, resulting in a synthetic sparse 3D point cloud that matches the target synthetic scene and conforms to real radar observation characteristics. Finally, object pasting enhancement uses a rare class object library to paste objects between the synthetic sparse 3D point cloud and the scene voxel ground truth values, supplementing and balancing rare long-tail categories while maintaining pairing consistency.

[0120] By using the above-mentioned synthetic training method to train the two-dimensional discrete diffusion correction network, the scale of training data can be expanded without relying on additional manual annotation. Furthermore, object pasting enhancement can improve the imbalance of rare long-tail categories in the training data, thereby enhancing the two-dimensional discrete diffusion correction network's ability to correct rare categories and correspondingly improving the accuracy of the final bird's-eye view semantic segmentation result on rare categories.

[0121] Step 105: Determine the semantic category of each two-dimensional grid position based on the semantic probability distribution correction result to obtain the bird's-eye view semantic segmentation result of the target scene.

[0122] The semantic probability distribution correction result is the bird's-eye view semantic probability distribution output by the two-dimensional discrete diffusion correction network in step 104 after correction, located in the probability simplex space. For each two-dimensional grid position on the bird's-eye view plane, it gives the probability that the position belongs to each candidate semantic category.

[0123] The semantic segmentation result of the bird's-eye view is a two-dimensional semantic map obtained by assigning a definite semantic category to each two-dimensional grid position on the bird's-eye view plane. That is, a bird's-eye view with semantic categories marked at each position from a top-down perspective, which expresses the spatial layout and semantic information of the target scene in two-dimensional form.

[0124] Specifically, for each two-dimensional grid location in the semantic probability distribution correction result, the semantic category with the highest probability in its semantic probability distribution is taken as the semantic category of that location. This process is repeated for all two-dimensional grid locations, transforming the semantic probability distribution correction result from the semantic probability distribution of each location into a definite semantic category for each location. This yields the bird's-eye view semantic segmentation result of the target scene. This bird's-eye view semantic segmentation result can be used as input for downstream tasks such as path planning and obstacle avoidance decision-making.

[0125] For example, in the perception scenario of autonomous vehicles, sparse 3D point clouds are acquired by vehicle-mounted LiDAR, and the reference bird's-eye view semantic map and the bird's-eye view semantic segmentation result are vehicle-side bird's-eye views based on the vehicle coordinate system. The bird's-eye view semantic segmentation result is used for path planning or decision-making of the autonomous driving perception system.

[0126] For example, in the perception scenario of embodied intelligence or indoor mobile robot, sparse 3D point cloud is acquired by solid-state LiDAR or depth camera, the reference bird's-eye view semantic map and the bird's-eye view semantic segmentation result are near-field bird's-eye view based on the robot's body coordinate system, the semantic category is replaced with the indoor scene semantic category, and the bird's-eye view semantic segmentation result is used for the robot's path planning or obstacle avoidance decision.

[0127] Reference Figure 3 , Figure 3 This is a schematic diagram of a bird's-eye view semantic segmentation task on SemanticKITTI validation set sequence 08, as an embodiment of this application. Figure 3 The left side shows the projection of the sparse lidar point cloud onto the bird's-eye view plane, which is the model input of the method. It can be seen that the sparse lidar point cloud has only a limited area of ​​effective observation on the bird's-eye view plane, with most areas being empty. The scan lines are distributed in an arc shape, and the overall observation information is sparse and uneven, making it very difficult to directly perform position-by-position semantic judgment. Figure 3 The right side shows the corresponding ground truth bird's-eye view semantic map, which is the supervision signal used during the training of the method. The ground truth bird's-eye view semantic map assigns a specific semantic category to each two-dimensional grid position on the bird's-eye view plane. Rendered using a two-dimensional semantic color map, it clearly distinguishes the spatial distribution of various semantic categories such as roads, buildings, vegetation, pedestrians, and cyclists. Both images are rendered using the same two-dimensional semantic color map to visually compare the differences between the input observation coverage and the ground truth semantic distribution.

[0128] For example, the embodiments of this application achieve 36.09% mIoU on the SemanticKITTI validation set, which is 9.06 percentage points higher than the previous strongest pure LiDAR 2D dedicated method 2D S3CNet (27.03%). This shows that the method can effectively improve the semantic segmentation accuracy of bird's-eye view under the condition of relying solely on LiDAR and taking into account processing efficiency.

[0129] It is worth noting that, regarding the number of correction steps, in offline evaluation or deployment scenarios that allow for higher inference latency, the number of correction sub-steps can be increased to 2 to 100 steps. For example, when the number of correction sub-steps is 2, the semantic segmentation accuracy of the bird's-eye view is slightly higher than that of a single-step deployment by about 0.05 percentage points. After increasing the number of correction sub-steps to 50 steps or more, the accuracy hardly changes. Regarding the selection of the base model, the base model can be replaced by any 3D semantic scene completion model capable of outputting a voxel-by-voxel semantic category distribution, such as LMSCNet, JS3C-Net, SCPNet, etc. Regarding the column projection method, in addition to taking the semantic category corresponding to the highest non-empty voxel, alternative projection methods can be adopted, such as weighting by category height distribution or averaging the most frequently occurring categories. Regarding the composition of the conditional information, in addition to the lidar bird's-eye view occupancy map, the 3D voxel features of the base model can be pooled along the vertical direction and input as an additional conditional channel to the 2D discrete diffusion correction network. Regarding the network structure, the two-dimensional discrete diffusion correction network defaults to a U-Net structure with four downsampling layers, a bottleneck layer, and four upsampling layers. This can be replaced with a lightweight variant with three downsampling layers, a bottleneck layer, and three upsampling layers, reducing the number of parameters by approximately 40% and decreasing the bird's-eye view semantic segmentation accuracy by approximately 1 percentage point. This is suitable for deployment scenarios with extremely high inference latency requirements. Regarding the bird's-eye view grid configuration, this application defaults to a 256×256 bird's-eye view grid covering a 51.2m × 51.2m area with the vehicle's rear axle as the origin. In embodied intelligent agent or indoor mobile robot perception scenarios, this can be changed to a near-field bird's-eye view grid covering a 10m × 10m area with a resolution of 2cm per pixel, based on the robot's body coordinate system. In multi-rotor UAV perception scenarios, this can be changed to a downward-looking projection grid based on an inertial measurement unit or altimeter. These replacements only require reconfiguring the input dimensions of the two-dimensional discrete diffusion correction network; the training code and backsampling closed-loop update rules remain unchanged. Regarding the semantic category table, this application defaults to using a 20-category semantic category table from the vehicle point cloud dataset. In embodied intelligence or indoor perception scenarios, this can be replaced with an indoor scene semantic category table, including walls, floors, furniture, doors, windows, and appliances. In industrial inspection or logistics scenarios, it can be replaced with an asset semantic category table specific to that industry, including shelves, automated guided vehicle tracks, cables, and personnel boundaries. These replacements only require adjusting the number of output channels of the two-dimensional discrete diffusion correction network and the training ground truth category table accordingly.

[0130] In this embodiment, a three-dimensional source prediction distribution is obtained by first completing the sparse three-dimensional point cloud with a three-dimensional semantic scene using a base model. Then, the three-dimensional source prediction distribution is projected along the vertical column direction to obtain a reference bird's-eye view semantic map. The one-hot encoding of the reference bird's-eye view semantic map is used as a two-dimensional structured source distribution, which is then corrected by a two-dimensional discrete diffusion correction network in the probabilistic simplex space to obtain the bird's-eye view semantic segmentation result. On the one hand, the reference bird's-eye view semantic map is obtained by projecting the three-dimensional source prediction distribution along the column direction, inheriting the semantic information learned by the base model in the vertical direction. This is equivalent to directly transferring the result of the three-dimensional semantic scene completion as the starting point for two-dimensional bird's-eye view segmentation, so that the two-dimensional discrete diffusion correction network does not need to repeatedly learn the complete mapping from sparse observations to dense bird's-eye view semantics, reducing the learning burden of the two-dimensional network. On the other hand, by using this near-true reference bird's-eye view semantic map instead of random noise as the starting point of the diffusion correction process, the two-dimensional discrete diffusion correction network only needs to learn the correction direction pointing to the true bird's-eye view distribution. Therefore, it can complete the segmentation with fewer correction steps or even a single step, reducing the inference cost. Furthermore, in the above process, both the baseline bird's-eye view semantic map and the conditional information are obtained from radar point clouds, without relying on camera input. Therefore, the embodiments of this application can obtain more accurate bird's-eye view semantic segmentation results while relying solely on radar point clouds and considering processing efficiency, thereby improving the perception reliability and operational security of the embodied intelligent system.

[0131] Compared to related technologies, in the embodiments of this application, firstly, under the condition of relying solely on LiDAR and without requiring camera input, the embodiments of this application achieve 36.09% mIoU on the SemanticKITTI validation set, which is 9.06 percentage points higher than the previous strongest pure LiDAR 2D dedicated method 2D S3CNet (27.03%), and also 1.82 percentage points higher than the baseline of the bird's-eye view projection (34.27%).

[0132] Second, the embodiments of this application support single-step inference deployment, requiring only one additional two-dimensional U-Net forward computation to complete the correction, without the need for distillation-assisted training. Compared with diffusion correction methods such as SegRefiner and IDBBR, which require 10 to 100 steps of backsampling, the inference latency is only about 1 / N, which can meet the processing efficiency requirements of real-time deployment.

[0133] Third, the embodiments of this application use deterministic linear interpolation on the probabilistic simplex instead of random noise as the forward process for diffusion correction, which differs from the Bernoulli state transition of SegRefiner and the Gaussian noise diffusion of BEVDiffuser. Since the training signal is a residual correction rather than generated from zero, the embodiments of this application have stable convergence and strong robustness to conditional noise.

[0134] Fourth, the embodiments of this application directly inherit the learned vertical semantic distribution of the base model through the highest non-empty voxel projection, using it as the structured starting point for two-dimensional discrete diffusion correction, which significantly reduces the learning burden of the two-dimensional network. This approach is also applicable to other existing three-dimensional predictors.

[0135] Fifth, the embodiments of this application do not rely on truth value guidance during the inference stage, but only rely on the prediction of the base model and the occupancy map of the LiDAR bird's-eye view. They do not assume that any truth value is available, which is in line with the actual deployment conditions and is different from methods such as BEVDiffuser that rely on truth value layout during the training period.

[0136] Sixth, the embodiments of this application work in conjunction with the structured source distribution discrete diffusion model. The same set of structured source distribution discrete diffusion weights can achieve better performance in both the three-dimensional semantic scene completion and the two-dimensional bird's-eye view semantic segmentation tasks, indicating that the structured source distribution discrete diffusion paradigm has strong versatility.

[0137] Seventh, the bird's-eye view semantic segmentation results output by the embodiments of this application are structurally consistent with the two-dimensional input conventions of the path planning, decision-making, and high-precision map matching modules in the L3 or L4 perception stack of autonomous driving, and also consistent with the path planning and obstacle avoidance input conventions of platforms such as embodied intelligent agent top-down world models, inspection robots, cleaning robots, logistics autonomous mobile robots, and multi-rotor drones. The single-step, non-distillation inference characteristic makes it compatible with the real-time budget of the aforementioned systems at the methodological level.

[0138] The above describes a radar bird's-eye view semantic segmentation method for embodied intelligence in the embodiments of this application. The following describes the radar bird's-eye view semantic segmentation device for embodied intelligence that performs the above-described radar bird's-eye view semantic segmentation method for embodied intelligence.

[0139] See Figure 4 ,like Figure 4 The diagram shows a structural schematic of a radar bird's-eye view semantic segmentation device for embodied intelligence. The radar bird's-eye view semantic segmentation device for embodied intelligence in this embodiment can achieve the above-mentioned... Figure 1 The steps of the radar bird's-eye view semantic segmentation method for embodied intelligence executed in the corresponding embodiments are described above. The functions implemented by the radar bird's-eye view semantic segmentation device for embodied intelligence can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and these modules can be software and / or hardware. The radar bird's-eye view semantic segmentation device for embodied intelligence may include the following modules, and the functional implementation of each module can be referred to... Figure 1 The operations performed in the corresponding embodiments will not be described in detail here.

[0140] The preprocessing module 401 is configured to perform voxelization on the environmental observation data of the target scene to obtain a sparse 3D point cloud to be processed. The prediction module 402 is configured to input the sparse 3D point cloud into the base model, perform feature extraction and preliminary prediction on the sparse 3D point cloud, and obtain a 3D source prediction distribution; the 3D source prediction distribution is used to characterize the semantic probability distribution of each voxel in the 3D mesh of the target scene. The projection module 403 is configured to project the three-dimensional source prediction distribution along the vertical direction in the column direction, and take the semantic category corresponding to the highest non-empty voxel in the vertical direction of each two-dimensional grid position to obtain the reference bird's-eye view semantic map. The correction module 404 is configured to use the one-hot encoding of the reference bird's-eye view semantic graph as a two-dimensional structured source distribution, input the two-dimensional structured source distribution, diffusion time step, and conditional information into a two-dimensional discrete diffusion correction network, and use the two-dimensional structured source distribution as the starting point of the diffusion correction process to predict the semantic probability distribution correction result; wherein, the two-dimensional discrete diffusion correction network performs the correction from the two-dimensional structured source distribution to the true bird's-eye view distribution in the probability simplex space; The segmentation module 405 is configured to determine the semantic category of each two-dimensional grid position based on the semantic probability distribution correction result, and obtain the bird's-eye view semantic segmentation result of the target scene.

[0141] In this embodiment, the above-mentioned multiple modules work together in sequence to realize a complete processing flow from environmental observation data to bird's-eye view semantic segmentation results. The preprocessing module 401 organizes the original environmental observation data into a regular three-dimensional grid representation, providing a unified input format for the prediction module 402. The prediction module 402 uses the base model to complete the three-dimensional semantic scene of the sparse three-dimensional point cloud, obtaining a three-dimensional source prediction distribution that already contains vertical semantic information, providing a high-quality three-dimensional semantic prior for subsequent column projection. The projection module 403 performs column projection on the three-dimensional source prediction distribution, compressing the three-dimensional semantic information into a reference bird's-eye view semantic map from a top-down perspective, thereby providing the correction module 404 with a structured starting point that is close to the truth rather than random. The correction module 404 uses the one-hot encoding of the reference bird's-eye view semantic map as a two-dimensional structured source distribution, driving the two-dimensional discrete diffusion correction network to perform correction from the two-dimensional structured source distribution to the truth bird's-eye view distribution in the probability simplex space, so that the network only needs to learn the residual correction direction, and can complete the correction in fewer correction steps or even a single step, reducing the inference cost. The segmentation module 405 selects the semantic category with the highest probability at each position based on the correction result, and obtains the final bird's-eye view semantic segmentation result. Therefore, this device can obtain more accurate bird's-eye view semantic segmentation results with lower inference cost by relying only on radar point clouds and without camera input, thus adapting to the various application requirements of platforms such as autonomous driving perception systems, embodied intelligent agents, and mobile robots for semantic segmentation accuracy and processing efficiency.

[0142] The above describes the radar bird's-eye view semantic segmentation device for embodied intelligence in the embodiments of this application from the perspective of modular functional entities. The following describes the radar bird's-eye view semantic segmentation device for embodied intelligence in the embodiments of this application from the perspective of hardware processing.

[0143] It should be noted that, Figure 4 The preprocessing module 401, prediction module 402, projection module 403, correction module 404 and segmentation module 405 shown can all be deployed in the corresponding physical devices, such as in the processor.

[0144] Figure 4 The devices shown can all have the following characteristics: Figure 5 The computing device structure shown, when Figure 4 The radar bird's-eye view semantic segmentation device shown has the following characteristics: Figure 5 When the structure shown is used, Figure 5 The processor and transceiver in the device can perform the same or similar functions as the modules provided in the aforementioned embodiments of the corresponding device. Figure 5 The memory stores the computer programs that the processor needs to call when executing the aforementioned radar bird's-eye view semantic segmentation method for embodied intelligence. Figure 5The bus in the system is used to send and receive data from various modules.

[0145] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0146] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0147] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0148] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0150] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)). The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. A radar overhead map semantic segmentation method for embodied intelligence, characterized in that, The method includes: The environmental observation data of the target scene is voxelized to obtain a sparse 3D point cloud to be processed. The sparse 3D point cloud is input into the base model, and features are extracted and preliminary predictions are made on the sparse 3D point cloud to obtain the 3D source prediction distribution; the 3D source prediction distribution is used to characterize the semantic probability distribution of each voxel in the 3D mesh of the target scene. The three-dimensional source prediction distribution is projected vertically, and the semantic category corresponding to the highest non-empty voxel in the vertical direction of each two-dimensional grid position is taken to obtain the baseline bird's-eye view semantic map. Using the one-hot encoding of the reference bird's-eye view semantic graph as a two-dimensional structured source distribution, the two-dimensional structured source distribution, diffusion time step, and conditional information are input into a two-dimensional discrete diffusion correction network. The two-dimensional structured source distribution is used as the starting point of the diffusion correction process to predict the semantic probability distribution correction result. The two-dimensional discrete diffusion correction network performs the correction from the two-dimensional structured source distribution to the true bird's-eye view distribution in the probability simplex space. Based on the semantic probability distribution correction results, the semantic category of each two-dimensional grid position is determined, and the semantic segmentation result of the bird's-eye view of the target scene is obtained.

2. The method of claim 1, wherein, The conditional information includes a bird's-eye view of the lidar occupancy map; before inputting the two-dimensional structured source distribution, diffusion time step, and conditional information into the pre-trained two-dimensional discrete diffusion correction network, the method further includes: The sparse 3D point cloud is aggregated column by column along the vertical direction onto the bird's-eye view plane to obtain the bird's-eye view occupancy map, and the bird's-eye view occupancy map is used as the condition information; The input of the two-dimensional discrete diffusion correction network is obtained by channel splicing of the two-dimensional structured source distribution, the time step features obtained by mapping the diffusion time step, and the bird's-eye view occupancy map.

3. The method of claim 1, wherein, The process of projecting the 3D source prediction distribution vertically along the column direction and taking the semantic category corresponding to the highest non-empty voxel in the vertical direction at each 2D grid position to obtain the baseline bird's-eye view semantic map includes: Traverse all 2D grid locations and perform the following processing procedure on each 2D grid location to obtain the semantic map of the baseline bird's-eye view: For each two-dimensional grid position, traverse each voxel in the voxel column corresponding to the current position along the vertical direction, and determine the voxel with the largest vertical coordinate and non-empty semantic category in the voxel column as the highest non-empty voxel of the current position. The process of obtaining the semantic probability distribution of the highest non-empty voxel in the three-dimensional source prediction distribution, determining the semantic category with the highest probability in the semantic probability distribution as the semantic category of the baseline bird's-eye view semantic map at the current position, and jumping to the next position. In this process, the column projection is implemented as a differentiable projection through soft maximum value operation during the training phase of the two-dimensional discrete diffusion correction network, and the semantic category corresponding to the highest non-empty voxel is determined through hard maximum value operation during the deployment phase.

4. The method of claim 1, wherein, The method of using the one-hot encoding of the reference bird's-eye view semantic graph as a two-dimensional structured source distribution includes: Obtain the semantic category at each two-dimensional grid position in the semantic map of the baseline bird's-eye view; For each two-dimensional grid position, a probability vector with a dimension equal to the total number of semantic categories is constructed. The components in the probability vector that match the semantic category of the current position are set to 1, and the remaining components that do not match the semantic category of the current position are set to 0, thus obtaining the one-hot encoding vector of the current position; the one-hot encoding vector is the vertex of the probabilistic simplex space. The one-hot encoded vectors at each two-dimensional grid position are arranged according to the two-dimensional grid position to obtain the two-dimensional structured source distribution; the two-dimensional structured source distribution and the ground truth bird's-eye view distribution are located in the same probability simplex space and serve as the starting point of the diffusion correction process.

5. The method of claim 4, wherein, The step of inputting the two-dimensional structured source distribution, diffusion time step, and conditional information into a two-dimensional discrete diffusion correction network, using the two-dimensional structured source distribution as the starting point of the diffusion correction process, and predicting the semantic probability distribution correction result includes: Under single-step inference deployment, the total number of steps in the diffusion correction process is set to 1, and the diffusion time step is set to the maximum diffusion time step; The two-dimensional structured source distribution, the maximum diffusion time step, and the conditional information are input into the two-dimensional discrete diffusion correction network, and the intermediate result output by the two-dimensional discrete diffusion correction network is obtained through a single forward calculation. The intermediate results are normalized to obtain an estimated distribution located in the probability simplex space. The estimated distribution is the prediction of the ground truth bird's-eye view distribution by the two-dimensional discrete diffusion correction network. The estimated distribution is directly determined as the corrected semantic probability distribution; The two-dimensional discrete diffusion correction network is a network obtained without distillation training, and the output of the single forward computation is the average correction direction from the two-dimensional structured source distribution to the ground truth bird's-eye view distribution.

6. The method of claim 4, wherein, The step of inputting the two-dimensional structured source distribution, diffusion time step, and conditional information into a two-dimensional discrete diffusion correction network, using the two-dimensional structured source distribution as the starting point of the diffusion correction process, and predicting the semantic probability distribution correction result includes: Under multi-step iterative deployment, the diffusion correction process is divided into multiple consecutive correction sub-steps, the time scheduling coefficients corresponding to each correction sub-step are determined, and the two-dimensional structured source distribution is used as the current state of the first correction sub-step. For each correction substep, the current state of the current correction substep, the diffusion time step corresponding to the current correction substep, and the condition information are input into the two-dimensional discrete diffusion correction network. The estimated distribution of the current correction substep is obtained through forward calculation. The estimated distribution is a prediction of the true bird's-eye view distribution. Calculate the distribution difference between the estimated distribution and the two-dimensional structured source distribution, the distribution difference being used to indicate the correction direction from the two-dimensional structured source distribution to the true bird's-eye view distribution; Obtain the time scheduling coefficient corresponding to the current correction substep and the time scheduling coefficient corresponding to the next correction substep adjacent in the time shift direction, and calculate the coefficient difference between the time scheduling coefficient corresponding to the next correction substep and the time scheduling coefficient corresponding to the current correction substep. Multiply the distribution difference by the coefficient difference to obtain the state correction increment of the current correction sub-step, and add the current state to the state correction increment to obtain the update result of the current correction sub-step; Perform simplex projection on the updated result to obtain the current corrected substep projection state; The projected state is used as the current state of the next correction sub-step. After iterating to a preset number of correction sub-steps, the projected state of the last correction sub-step is determined as the semantic probability distribution correction result.

7. The method of claim 6, wherein, The simplex projection includes: truncating the negative values ​​in the update result to 0, and then normalizing them according to the L1 norm of the update result. The projected state satisfies the non-negativity and normalization constraints of the semantic probability distribution.

8. The method according to claim 1, characterized in that, The forward process used by the two-dimensional discrete diffusion correction network during the training phase includes: Obtain the one-hot encoding of the ground truth bird's-eye view semantic graph corresponding to the training samples, and the two-dimensional structured source distribution obtained by column projection of the training samples; For each diffusion time step in the diffusion time step, the interpolation coefficient corresponding to the current diffusion time step is determined; the interpolation coefficient changes between 0 and 1 as the diffusion time step progresses; The one-hot encoding of the truth bird's-eye view semantic graph is multiplied by the interpolation coefficients to obtain the first weighted distribution; The difference between 1 and the interpolation coefficients is multiplied by the two-dimensional structured source distribution to obtain the second weighted distribution; The first weighted distribution and the second weighted distribution are added together to obtain the diffusion intermediate state corresponding to the current diffusion time step; the diffusion intermediate state is located in the probability simplex space between the truth bird's-eye view semantic graph and the two-dimensional structured source distribution; The intermediate diffusion state, the current diffusion time step, and the conditional information are input into the two-dimensional discrete diffusion correction network, enabling the two-dimensional discrete diffusion correction network to learn the correction direction from the two-dimensional structured source distribution to the truth bird's-eye view semantic graph.

9. The method according to claim 6, characterized in that, The two-dimensional discrete diffusion correction network is trained using synthetic training pairs, which consist of synthetic sparse 3D point clouds and paired scene voxel ground truth values; the method further includes: The synthesized sparse 3D point cloud is projected onto the base model and the column direction to obtain a synthesized baseline bird's-eye view semantic map; The two-dimensional discrete diffusion correction network is trained using the truth bird's-eye view semantic graph obtained by projecting the truth values ​​of the scene voxels through the column direction as supervision.

10. A semantic segmentation device for radar bird's-eye view images for embodied intelligence, characterized in that, The device includes: The preprocessing module is configured to perform voxelization on the environmental observation data of the target scene to obtain a sparse 3D point cloud to be processed. The prediction module is configured to input the sparse 3D point cloud into the base model, perform feature extraction and preliminary prediction on the sparse 3D point cloud, and obtain a 3D source prediction distribution; the 3D source prediction distribution is used to characterize the semantic probability distribution of each voxel in the 3D mesh of the target scene. The projection module is configured to project the three-dimensional source prediction distribution along the vertical direction in a column direction, and take the semantic category corresponding to the highest non-empty voxel in the vertical direction of each two-dimensional grid position to obtain the reference bird's-eye view semantic map. The correction module is configured to use the one-hot encoding of the reference bird's-eye view semantic graph as a two-dimensional structured source distribution, input the two-dimensional structured source distribution, diffusion time step, and conditional information into a two-dimensional discrete diffusion correction network, and use the two-dimensional structured source distribution as the starting point of the diffusion correction process to predict the semantic probability distribution correction result; wherein, the two-dimensional discrete diffusion correction network performs the correction from the two-dimensional structured source distribution to the true bird's-eye view distribution in the probability simplex space; The segmentation module is configured to determine the semantic category of each two-dimensional grid position based on the semantic probability distribution correction result, and obtain the bird's-eye view semantic segmentation result of the target scene.

Images