Method for generating simulated point cloud data and electronic device

Abstract

The disclosure provides a simulation-based point cloud data generation method and electronic equipment, and relates to the technical field of automatic driving and unmanned vehicles. The simulation-based point cloud data generation method comprises the following steps: obtaining simulation synthesis point cloud data, wherein the simulation synthesis point cloud data comprises scene point cloud data of a target scene and first point cloud data of a target object, and the first point cloud data does not comprise intensity information; performing spatial occupancy prediction on the simulation synthesis point cloud data to obtain spatial occupancy information; performing prediction based on the spatial occupancy information through a diffusion model to obtain second point cloud data of the target object, wherein the second point cloud data comprises intensity information; and fusing the second point cloud data and the scene point cloud data to obtain target point cloud data under the target scene.

Description

Technical Field

[0001] This disclosure relates to the fields of autonomous driving and driverless vehicle technology, specifically to a method and electronic device for generating point cloud data based on simulation. Background Technology

[0002] In the field of autonomous driving, simulated point cloud data is widely used in scene construction, algorithm verification and virtual testing. Especially in complex and long-tailed scenarios such as mines, it can generate point cloud data with real physical characteristics, which is crucial for enhancing the generalization ability and robustness of perception algorithms.

[0003] Currently, related technologies are based on the principle of geometric ray tracing. They generate spatial distribution information of point clouds by emitting laser rays in a virtual environment and calculating their intersections with 3D terrain, vehicles, and other facilities. This method has advantages such as simplicity and high computational efficiency. However, the simulated point cloud data generated by this method has certain limitations in modeling reflection intensity information, affecting the generalization ability of the perception algorithm. Summary of the Invention

[0004] In view of this, embodiments of the present disclosure provide a method and electronic device for generating point cloud data based on simulation.

[0005] Firstly, a method for generating point cloud data based on simulation is provided, comprising: acquiring simulated synthetic point cloud data, which includes scene point cloud data of a target scene and first point cloud data of a target object, wherein the first point cloud data does not include intensity information; performing space occupancy prediction on the simulated synthetic point cloud data to obtain space occupancy information; using a diffusion model, performing prediction based on the space occupancy information to obtain second point cloud data of the target object, wherein the second point cloud data includes intensity information; and fusing the second point cloud data with the scene point cloud data to obtain target point cloud data under the target scene.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, the target scene includes a static background and dynamic targets. Space occupancy prediction is performed on the simulated synthetic point cloud data to obtain space occupancy information, including: performing semantic segmentation on the scene point cloud data to obtain multiple frames of static background point clouds corresponding to the static background and multiple frames of dynamic target point clouds corresponding to the dynamic targets; performing frame stacking processing on the multiple frames of static background point clouds to generate a dense background point cloud; and performing space occupancy prediction on the dense background point cloud, the multiple frames of dynamic target point clouds, and the first point cloud data to obtain space occupancy information.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the voxel grid resolution of the space occupancy information is greater than the voxel grid resolution in the height direction than in the length and width directions.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, obtaining simulated synthetic point cloud data includes: obtaining scene point cloud data of the target scene, which is obtained by data collection of the target scene; obtaining the target 3D model corresponding to the target object; rendering the target 3D model to obtain the first point cloud data of the target object; and fusing the first point cloud data and the scene point cloud data to obtain simulated synthetic point cloud data.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the first point cloud data and the scene point cloud data are fused, including: obtaining a preset trajectory corresponding to the scene point cloud data, the preset trajectory representing the movement trajectory of the target object in the target scene; and fusing the first point cloud data and the scene point cloud data according to the preset trajectory.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the second point cloud data of the target object is obtained by predicting the spatial occupancy information through a diffusion model, including: obtaining the point cloud mask of the target object; inputting the spatial occupancy information into the diffusion model to obtain the third point cloud data corresponding to the simulated synthetic point cloud data; and extracting the second point cloud data from the third point cloud data using the point cloud mask.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, spatial occupancy information is input into a diffusion model to obtain third point cloud data corresponding to the simulated synthetic point cloud data, including: generating spatial features corresponding to the spatial occupancy information through the diffusion model; performing ray sampling on the spatial features and decoding the sampling results of the ray sampling operation to generate point cloud coordinate information and point cloud intensity information corresponding to the spatial occupancy information; and determining the third point cloud data based on the point cloud coordinate information and point cloud intensity information.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, spatial occupancy prediction is performed on the simulated synthetic point cloud data to obtain spatial occupancy information, including: obtaining the point cloud mask of the target object; processing the simulated synthetic point cloud data using the point cloud mask; performing spatial occupancy prediction on the processed simulated synthetic point cloud data to obtain spatial occupancy information corresponding to the target object; and predicting the spatial occupancy information through a diffusion model to obtain second point cloud data of the target object, including: inputting the spatial occupancy information corresponding to the target object into the diffusion model to obtain second point cloud data.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the second point cloud data of the target object is obtained by predicting the spatial occupancy information through a diffusion model, including: obtaining the point cloud mask of the target object; processing the spatial occupancy information using the point cloud mask to obtain the spatial occupancy information corresponding to the target object; and inputting the spatial occupancy information corresponding to the target object into the diffusion model to obtain the second point cloud data.

[0014] In a second aspect, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method provided in the first aspect by executing the executable instructions.

[0015] The simulation-based point cloud data generation method disclosed herein captures the structure of the target object by predicting the spatial occupancy of the simulated synthetic point cloud data, providing geometric constraints for the point cloud data generation process, and using a diffusion model to predict point cloud data with intensity information. Finally, it is fused with scene point cloud data to generate target point cloud data, providing high-fidelity, controllable point cloud data that is adaptable to complex meteorological environments for autonomous driving technology in long-tail scenarios. Attached Figure Description

[0016] Figure 1 The diagram shown is a system architecture diagram of a simulation-based point cloud data generation method provided in an embodiment of this disclosure.

[0017] Figure 2 The diagram shown is a flowchart illustrating a simulation-based point cloud data generation method according to an embodiment of this disclosure.

[0018] Figure 3 The diagram shown is a flowchart illustrating the steps for obtaining simulated synthetic point cloud data according to an embodiment of this disclosure.

[0019] Figure 4 The diagram shown is a flowchart illustrating the steps of fusing first point cloud data and scene point cloud data according to an embodiment of this disclosure.

[0020] Figure 5 The diagram shown is a flowchart illustrating the steps of predicting space occupancy from simulated synthetic point cloud data to obtain space occupancy information, according to an embodiment of this disclosure.

[0021] Figure 6 The diagram shown is a flowchart illustrating the step of predicting spatial occupancy information and obtaining the second point cloud data of the target object through a diffusion model, according to an embodiment of this disclosure.

[0022] Figure 7 The diagram shown is a flowchart illustrating the steps of inputting spatial occupancy information into a diffusion model to obtain the third point cloud data corresponding to the simulated synthetic point cloud data, according to an embodiment of this disclosure.

[0023] Figure 8 The diagram shown is a flowchart illustrating the steps of predicting space occupancy from simulated synthetic point cloud data to obtain space occupancy information, according to an embodiment of this disclosure.

[0024] Figure 9The diagram shown is a flowchart illustrating the step of predicting spatial occupancy information and obtaining the second point cloud data of the target object through a diffusion model, according to an embodiment of this disclosure.

[0025] Figure 10 The diagram shown is a flowchart illustrating a simulation-based point cloud data generation method provided in another embodiment of this disclosure.

[0026] Figure 11 The diagram shown is a schematic representation of a simulation-based point cloud data generation device provided in an embodiment of this disclosure.

[0027] Figure 12 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0028] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0029] In complex, long-tailed scenarios like mines, large-scale, highly realistic LiDAR point cloud data is typically required for training and validating environmental perception algorithms. Among related technologies, a physical rule-based simulation method is primarily employed. This method involves geometric ray tracing within a virtual 3D scene to calculate the intersections of laser rays with 3D terrain, mining vehicles, and other facilities, thereby generating spatial distribution information of the point cloud. This approach is simple to implement, computationally efficient, highly controllable, and can rapidly generate large-scale simulated point cloud data.

[0030] However, this approach has significant limitations in modeling reflection intensity information, typically using only distance-squared attenuation or fixed material reflectivity to approximate laser echoes. This simplified method struggles to accurately describe the diverse reflection characteristics of lasers on complex surfaces, rocks, mineral sands, and vehicle metal surfaces, resulting in a large discrepancy between the intensity information in simulated point cloud data and real sensor data, thus affecting the generalization ability of perception algorithms in real-world environments.

[0031] This problem is even more pronounced in mining environments. Mining environments are typically characterized by high levels of dust, slippery surfaces, and are prone to severe weather conditions such as rain, snow, and fog. These environmental factors cause scattering, absorption, and multipath interference of laser signals, making it impossible for the intensity information generated by the above-mentioned schemes to accurately reflect the characteristics of the laser echo. This results in significant deviations in visual consistency and reflected energy distribution in the simulated point cloud data.

[0032] To address the aforementioned issues, a deep learning-based method for generating intensity information has been proposed. This method utilizes intensity values ​​from real point cloud data for supervised training to learn the laser reflection patterns under complex scenes and meteorological conditions, thereby generating realistic and physically consistent intensity information. However, this approach is prone to producing illusions that violate physical laws and is computationally expensive.

[0033] To address the aforementioned technical problems, this disclosure provides a method for generating point cloud data based on simulation, comprising: acquiring simulated synthetic point cloud data, the simulated synthetic point cloud data including scene point cloud data of the target scene and first point cloud data of the target object, the first point cloud data excluding intensity information; performing space occupancy prediction on the simulated synthetic point cloud data to obtain space occupancy information; using a diffusion model, performing prediction based on the space occupancy information to obtain second point cloud data of the target object, the second point cloud data including intensity information; and fusing the second point cloud data with the scene point cloud data to obtain target point cloud data under the target scene.

[0034] The following is a combination of... Figure 1 This disclosure introduces a system architecture provided by one embodiment.

[0035] Figure 1 The diagram shown is a system architecture schematic of a simulation-based point cloud data generation method according to an embodiment of this disclosure. Figure 1 As shown, the system architecture includes a terminal 110 and a server 120. The terminal 110 and the server 120 are communicatively connected, for example, via a wired or wireless network.

[0036] Terminal 110 may include electronic devices such as vehicles, mobile phones, laptops, or personal computers, or it may include vehicle-mounted or fixed data acquisition devices. Terminal 110 is used for data acquisition and preprocessing operations in the target scene. For example, terminal 110 can generate simulated synthetic point cloud data in the target scene; or, the user can directly upload the simulated synthetic point cloud data to terminal 110 through the interactive interface of terminal 110. Then, terminal 110 sends the simulated synthetic point cloud data to server 120.

[0037] Server 120 receives simulated synthetic point cloud data from terminal 110, processes the simulated synthetic point cloud data, and generates target point cloud data for the target scene. Finally, the target point cloud data is transmitted back to terminal 110 for user use. Server 120 can be a single server, a server cluster or cloud server composed of multiple servers, or server 120 can be integrated into terminal 110.

[0038] Those skilled in the art will know that Figure 1The number of terminals and servers shown is merely illustrative. Depending on actual needs, there may be any number of terminals and servers, and this disclosure does not impose any limitation on this.

[0039] The following is combined Figures 2 to 10 This disclosure describes in detail the method for generating point cloud data based on simulation, as provided in the embodiments of this disclosure.

[0040] Figure 2 The diagram shows a flowchart illustrating a simulation-based point cloud data generation method according to an embodiment of this disclosure. Exemplarily, the simulation-based point cloud data generation method provided in this embodiment consists of... Figure 1 The server 120 shown is executing.

[0041] like Figure 2 As shown in the embodiments of this disclosure, the method for generating simulation-based point cloud data includes the following steps.

[0042] S210, acquire simulated point cloud data.

[0043] The simulated point cloud data includes the scene point cloud data of the target scene and the first point cloud data of the target object.

[0044] The target scene refers to the specific environment from which point cloud data is to be generated. For example, target scenes include non-standard road scenes such as mines and construction sites. The scene point cloud data is obtained by collecting real data from the target scene, for example, by using a vehicle-mounted LiDAR system while driving in an actual mining area. The scene point cloud data reflects the true geometry and intensity distribution of the target scene.

[0045] The target object is a long-tail target located in the target scene, such as excavators, refueling trucks, command vehicles, pedestrians, and other objects that appear infrequently or exhibit unique behaviors. The first point cloud data of the target object can be obtained through 3D model rendering; alternatively, the first point cloud data can be pre-generated or extracted from a pre-established asset library. If the obtained point cloud data carries intensity information generated by simple physical rules, this intensity information can be removed to obtain first point cloud data that does not include intensity information.

[0046] The simulated point cloud data is obtained by fusing the scene point cloud data and the first point cloud data. Optionally, the fusion operation is essentially to add the point set representing the target object to the point set representing the target scene based on its pose in the world coordinate system.

[0047] S220 performs space occupancy prediction on the simulated synthetic point cloud data to obtain space occupancy information.

[0048] Space occupancy prediction can be achieved through various algorithms. For example, point cloud data from simulated synthetic point cloud data can be voxels, and a prediction model can be used to predict the probability of each voxel being occupied, thus obtaining space occupancy information.

[0049] Space occupancy information is used to represent the probability information of each voxel being occupied in the three-dimensional space corresponding to the target scene. Specifically, the three-dimensional space of the target scene is divided into a regular voxel grid, and each voxel is assigned an occupancy probability value, representing the likelihood that the voxel will be occupied by any entity.

[0050] In some embodiments, the voxel grid resolution of the space occupancy information is greater than the voxel grid resolution in the height direction than in the length and width directions.

[0051] Mine roads are uneven and have inclines and declines. To improve prediction accuracy, the voxel grid resolution for spatial occupancy information can be specifically adjusted.

[0052] Optionally, the space occupancy information can be set such that the voxel grid resolution in both the length (X-axis) and width (Y-axis) directions is greater than the voxel grid resolution in the height (Z-axis) direction.

[0053] For example, the height range of the three-dimensional space is [-5 meters, 9 meters]. The resolution in the height direction can be set to 0.1 meters, and the resolution in both the length and width directions can be set to 0.5 meters. Alternatively, the resolution in the height direction can be set to 0.15 meters, and the resolution in both the length and width directions can be set to 0.6 meters. Or, the resolution in the height direction can be set to 0.2 meters, the resolution in the length direction can be set to 0.5 meters, and the resolution in the width direction can be set to 0.6 meters.

[0054] In scenarios with frequent uphill and downhill sections, a finer height resolution helps to more accurately model road surface inclination and the relative position of the target object to the ground. This non-uniform resolution setting can improve prediction accuracy without significantly increasing computational load.

[0055] In a specific example, the simulated synthetic point cloud data obtained in the above steps is normalized into a preset 3D bounding box, which covers the region of interest in length, width, and height. Then, the region of interest is divided using a voxel mesh with a resolution of [0.5m, 0.5m, 0.1m]. For each voxel, it is predicted whether it is occupied; if so, the voxel is marked as "occupied" (value 1); if not, it is marked as "unoccupied" (value 0), thus obtaining the spatial occupancy information.

[0056] It is understood that the specific resolution value can be adjusted according to the scene size and accuracy requirements. For example, the length and width resolution can be selected between 0.4 meters and 1.0 meters, such as 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1.0 meters, and the height resolution can be selected between 0.1 meters and 0.4 meters, such as 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, and 0.4 meters. Furthermore, occupancy prediction can be achieved based on the density and number of points in the voxel grid or through neural networks, etc., and this disclosure does not impose specific limitations in this regard.

[0057] S230 uses a diffusion model to predict based on spatial occupancy information to obtain the second point cloud data of the target object.

[0058] The diffusion model is a generative model based on the diffusion process. The diffusion process mainly consists of two steps: the forward diffusion process gradually adds noise to the data until it becomes pure noise, and the backward denoising process learns to gradually reconstruct the original data from the noise.

[0059] The diffusion model generates second point cloud data of the target object with intensity information based on spatial occupancy information. A point in the second point cloud data can be represented as {x, y, z, i}, where x, y, and z represent the three-dimensional coordinates of the point, and i represents the intensity value of the point.

[0060] The diffusion model is trained under supervised supervision using multiple sets of real point cloud data collected at different times and under different meteorological conditions in the target scene to learn the laser reflection patterns under complex scenes and meteorological conditions. The trained diffusion model can accurately predict intensity information that conforms to the physical characteristics of the real environment. The generated second point cloud data not only has a high degree of geometric matching with the target object, but its intensity information can also accurately reflect the diverse reflection characteristics of the target object to laser.

[0061] S240, the second point cloud data is fused with the scene point cloud data to obtain the target point cloud data under the target scene.

[0062] This step is used to integrate the second point cloud data with the scene point cloud data to generate high-fidelity point cloud data that can ultimately be used for downstream tasks.

[0063] Specifically, based on the location of the target object in the target scene, the second point cloud data is located into the scene point cloud data and fused to obtain the target point cloud data. The fused target point cloud data retains the real-collected point cloud containing real intensity information in the background, while the foreground target object part is a point cloud whose shape is ensured by geometric simulation and whose intensity is ensured by diffusion model, thus obtaining high-fidelity synthetic point cloud data as a whole.

[0064] In this embodiment of the disclosure, spatial occupancy prediction of simulated synthetic point cloud data is performed to capture the structure of the target object, thereby providing geometric constraints for the point cloud data generation process. A diffusion model is used to predict point cloud data with intensity information, which is then fused with scene point cloud data to generate target point cloud data. This provides high-fidelity, controllable point cloud data that is adaptable to complex weather environments for autonomous driving technology in long-tail scenarios.

[0065] The specific implementation methods of each step in the above embodiments will be described below.

[0066] Figure 3 The diagram shown is a flowchart illustrating the steps for acquiring simulated synthetic point cloud data according to an embodiment of this disclosure. Figure 3 As shown in the embodiments of this disclosure, the steps for obtaining simulated synthetic point cloud data include the following steps.

[0067] S310 acquires the scene point cloud data of the target scene.

[0068] Scene point cloud data is obtained by collecting data from a target scene, such as by using a vehicle-mounted LiDAR device in a real target scene. Scene point cloud data reflects the true geometry and intensity distribution of the target scene.

[0069] Scene point cloud data can also be retrieved from a pre-established static asset library. The static asset library is used to store scene point cloud data collected by actual vehicles in target scenes (such as specific mining areas). For example, the static asset library includes scene point clouds of specific scenes such as mining transportation sections, unloading areas, and mining operation areas, as well as corresponding labels; during the retrieval process, scene point cloud data with corresponding labels can be searched according to usage needs.

[0070] S320: Obtain the target 3D model corresponding to the target object.

[0071] The target 3D model can be retrieved from a pre-established dynamic asset library or generated through a modeling process such as 3D Gaussian sputtering.

[0072] The dynamic asset library stores standardized 3D models of various common target objects, such as mining trucks, excavators, refueling trucks, command vehicles, and other engineering equipment commonly found in mining scenes, as well as personnel and roadblocks. When modeling using the 3D Gaussian sputtering method, multi-view images or basic point cloud data of the target object can be collected first, and then the geometric features of the target object's surface can be fitted using a Gaussian distribution to gradually construct a high-precision 3D model. This method is suitable for specific long-tail targets not included in the dynamic asset library.

[0073] S330 renders the target 3D model to obtain the first point cloud data of the target object.

[0074] The target 3D model is virtually rendered. By simulating the scanning principle of LiDAR, samples are taken from the model surface to generate the first point cloud data corresponding to the target object. The first point cloud data preserves the precise geometry of the target object.

[0075] S340 fuses the first point cloud data and the scene point cloud data to obtain the simulated synthetic point cloud data.

[0076] Spatially align and fuse the first point cloud data with the scene point cloud data. That is, based on the position of the target object in the target scene, insert or replace the point cloud of the target object into the specified position in the scene, and finally obtain the simulated synthetic point cloud data that fuses the point cloud of the virtual target object and the point cloud of the real background.

[0077] In this embodiment, the first point cloud data generated based on the 3D model fully preserves the precise geometry of the target object. By combining this data with real-world scene point cloud data through spatial alignment and fusion techniques, the geometric accuracy of the target point cloud is ensured, while also giving the simulated point cloud data the environmental characteristics of a real scene. This guarantees the realism of the background while ensuring the diversity and controllability of the target objects within the target scene.

[0078] Furthermore, by flexibly adjusting the position and pose of the target object in the scene, diverse simulated synthetic point cloud data can be quickly generated, meeting the needs of perception algorithms for massive and diverse training data. This will be discussed in detail below.

[0079] Figure 4 The diagram shown is a flowchart illustrating the steps of fusing first point cloud data and scene point cloud data according to an embodiment of this disclosure. Figure 4 As shown in the embodiments of this disclosure, the step of fusing the first point cloud data and the scene point cloud data includes the following steps.

[0080] S410, acquire the preset trajectory corresponding to the scene point cloud data.

[0081] A preset trajectory is a sequence of movement trajectories and pose changes of a target object within a target scene. The preset trajectory can be a keyframe path designed based on the target scene, or a real motion trajectory extracted from data collected from actual vehicles. The preset trajectory can be generated using rule-based methods, physics-based trajectory generation, deep learning model-based trajectory generation, reinforcement learning-based trajectory generation, etc. The appropriate generation method can be selected based on the required agent behavior to pre-generate the preset trajectory sequence.

[0082] In some embodiments, the scene point cloud data may include not only static backgrounds but also dynamic targets, such as pedestrians and vehicles. It is understood that dynamic targets are not target objects added to the target scene through the fusion process, but rather dynamic elements that already exist in the target scene. Their motion states and positional changes are fully preserved in the scene point cloud data, together with the target objects subsequently fused, to form a point cloud scene that more closely resembles the complex environment of the real world.

[0083] Due to the presence of dynamic targets, the preset trajectory must not only consider the geometric direction of the road but also cooperate with the dynamic targets to meet physical rules such as obstacle avoidance and following. For example, in a mining scenario, the preset trajectory includes the round-trip route of vehicles from the loading area to the unloading area in a specific road section; when the target object encounters oncoming vehicles at a narrow bend, the preset trajectory includes a passing strategy of slowing down and moving to the right; when personnel or auxiliary vehicles are passing near the loading area, the preset trajectory includes a temporary stopping action to give way, in order to simulate a real dynamic interaction scenario.

[0084] S420, based on a preset trajectory, fuses the first point cloud data and the scene point cloud data.

[0085] In this step, each frame of the preset trajectory is mapped to the corresponding coordinate position of the target scene, and the first point cloud data of the target object is fused frame by frame along the preset trajectory to generate a simulated point cloud sequence of the target object moving in the target scene over a continuous period of time.

[0086] For example, based on a preset trajectory, each point in the first point cloud data is mapped to the correct coordinate position in the global coordinate system of the scene point cloud data through coordinate transformation, thereby obtaining simulated synthetic point cloud data containing the target object.

[0087] In this embodiment of the disclosure, the first point cloud data and the scene point cloud data are fused by a preset trajectory to simulate the dynamic movement process of the target object in the target scene. This improves the temporal realism of the simulated synthetic point cloud data, makes the motion state of the target object more consistent with the physical constraints of the actual scene, and enhances the rationality and practicality of the synthetic point cloud.

[0088] After generating the simulated synthetic point cloud data, it is necessary to further generate its corresponding spatial occupancy information. As described in the above embodiments, the target scene includes static background and dynamic targets. Static background refers to objects in the target scene whose positions do not change over time, such as roads, buildings, and mountains; dynamic targets refer to objects in the target scene whose positions or postures change over time, such as moving vehicles and walking people.

[0089] Dynamic targets occupy different spatial positions at each moment, and their point cloud shape and occupied area change with the frame. If the same processing strategy is used as for static backgrounds, it will result in motion blur or ghosting. Therefore, it is necessary to process static backgrounds and dynamic targets separately, which will be introduced in detail below.

[0090] Figure 5 The diagram shown is a flowchart illustrating the steps of predicting space occupancy from simulated synthetic point cloud data to obtain space occupancy information, according to an embodiment of this disclosure. Figure 5 As shown in the embodiments of this disclosure, the steps for predicting space occupancy of simulated synthetic point cloud data to obtain space occupancy information include the following steps.

[0091] S510 performs semantic segmentation on scene point cloud data to obtain multiple frames of static background point cloud corresponding to static background and multiple frames of dynamic target point cloud corresponding to dynamic target.

[0092] Semantic segmentation is a process of classifying each point in a scene point cloud data to distinguish its semantic category.

[0093] In scene point cloud data, each point contains category information. Using a pre-trained point cloud semantic segmentation model, the system automatically identifies and segments the static background point cloud corresponding to the static background, and the dynamic target point cloud corresponding to various dynamic targets, based on the category information. Since scene point cloud data is typically a multi-frame sequence, segmenting each frame of the scene point cloud separately yields multiple frames of static background point clouds and multiple frames of dynamic target point clouds.

[0094] It is understandable that, in addition to using large point cloud semantic segmentation models for semantic segmentation, rule-based semantic segmentation methods can also be employed, or semantic segmentation can be achieved by pre-setting dynamic target category ranges in conjunction with prior knowledge of the target scene. In scenes with simple scene structures and fixed dynamic target motion patterns, efficient segmentation can be achieved with lower computational costs. This disclosure does not impose specific restrictions on the implementation method of semantic segmentation operations.

[0095] The S520 performs frame stacking processing on multiple static background point clouds to generate a dense background point cloud.

[0096] Frame stacking refers to the process of fusing multiple frames of point cloud data into a single dense point cloud frame. By utilizing the temporal continuity of the static background, the density and integrity of the background point cloud are enhanced.

[0097] Optionally, based on the extrinsic data of the static background point cloud, multiple frames of static background point clouds are aligned through rotation and translation transformations and merged into the same coordinate system for accumulation, thereby generating a denser background point cloud than a single frame of static background point cloud.

[0098] S530 performs space occupancy prediction on dense background point cloud, multi-frame dynamic target point cloud, and first point cloud data to obtain space occupancy information.

[0099] For any frame in the simulated synthetic point cloud data, the dense background point cloud of the static background, the multi-frame dynamic target point cloud of the dynamic target, and the first point cloud data of the target object in the current frame are voxelized together to generate the spatial occupancy information of the current frame. The spatial occupancy information of multiple frames together constitutes the spatial occupancy information set in the time series. The specific implementation method of spatial occupancy prediction can be referred to the above embodiment, and will not be repeated here.

[0100] In this embodiment of the disclosure, the static background and dynamic target in the target scene are first divided. The static background point cloud of the static background is enhanced by frame overlay, while the dynamic target point cloud of the dynamic target retains its complete motion state in time series. This allows the spatial occupancy information to accurately reflect the scene structure and the instantaneous state of dynamic elements, which is more in line with the perception input of the real world, thus providing an accurate context for the subsequent intensity information prediction process.

[0101] In some embodiments, if the target scene does not include dynamic targets, semantic segmentation is not required. Instead, multiple frames of static background point clouds are overlaid, and spatial occupancy information is generated based on the dense background point cloud and the first point cloud data. The specific implementation is similar to the embodiments described above and will not be repeated here.

[0102] The above describes the implementation method for generating global space occupancy information including static background, dynamic target, and target object. The following describes the specific implementation method for generating second point cloud data based on space occupancy information.

[0103] Figure 6 The diagram illustrates a step in predicting spatial occupancy information using a diffusion model to obtain the second point cloud data of a target object, according to an embodiment of this disclosure. Figure 6 As shown in this embodiment, the step of predicting spatial occupancy information and obtaining the second point cloud data of the target object through a diffusion model includes the following steps.

[0104] S610, obtain the point cloud mask of the target object.

[0105] The point cloud mask of a target object refers to the binary template or label information used to distinguish the target object from the simulated synthetic point cloud data.

[0106] Optionally, in the simulated synthetic point cloud data, the point corresponding to the target object can be represented as {x, y, z, c, id}, where x, y, and z are the three-dimensional coordinates of the point, c is the category information of the target object, and id is the unique identifier of the target object. In this step, a point cloud mask can be generated based on the category information of the points in the simulated synthetic point cloud data.

[0107] S620 inputs the space occupancy information into the diffusion model to obtain the third point cloud data corresponding to the simulated synthetic point cloud data.

[0108] The spatial occupancy information includes the geometry and specific location of the target object, as well as the global context of the target scene. Based on the complete contextual information, the diffusion model generates third point cloud data, carrying intensity information, corresponding to the entire simulated synthetic point cloud data.

[0109] The S630 uses a point cloud mask to extract the second point cloud data from the third point cloud data.

[0110] Using point cloud masks, the point set belonging to the target object is cropped from the third point cloud data to form the second point cloud data.

[0111] For example, the cropping operation can be performed as follows: The third point cloud data is matched point-by-point with the point cloud mask. For points with a mask value of 1, their 3D coordinates and intensity information are retained; for points with a mask value of 0, they are discarded, thus segmenting a set of points belonging only to the target object from the third point cloud data. These points are then combined to form the second point cloud data. The second point cloud data accurately reflects the geometric contour, position, and intensity information of the target object in the target scene, facilitating the fusion of the second point cloud data with the scene point cloud data to obtain high-precision target point cloud data.

[0112] In this embodiment, a diffusion model is first used to generate global third-point cloud data based on spatial occupancy information containing complete scene context. During the generation process, the diffusion model can incorporate richer contextual information, making full use of the spatial correlation, occlusion relationship, and semantic consistency between the target object and its surrounding environment, effectively improving the reliability and consistency of intensity prediction.

[0113] Figure 7 The diagram shows a flowchart illustrating the steps of inputting spatial occupancy information into a diffusion model to obtain third point cloud data corresponding to simulated synthetic point cloud data, according to an embodiment of this disclosure. Figure 7 As shown in the embodiment of this disclosure, the step of inputting spatial occupancy information into the diffusion model to obtain the third point cloud data corresponding to the simulated synthetic point cloud data includes the following steps.

[0114] S710 generates spatial features corresponding to spatial occupancy information through a diffusion model.

[0115] Spatial occupancy information is used as a conditional input to a pre-trained diffusion model. The diffusion model employs an encoder-decoder structure, learning the distribution characteristics of spatial occupancy information in the latent space through a progressive denoising process, generating spatial feature tensors corresponding to the spatial occupancy information. These spatial features encode the geometry, surface normals, and local context information of each voxel location, providing rich feature representations for subsequent steps.

[0116] S720 performs ray sampling on spatial features and decodes the sampling results to generate point cloud coordinates and point cloud intensity information corresponding to the spatial occupancy information.

[0117] The position and orientation of a virtual sensor are preset in the target scene. Ray sampling is performed from the perspective of the virtual sensor along preset intervals. Discrete sampling is performed along the ray extension direction with a fixed step size to form a ray feature sequence composed of multiple sampling points.

[0118] The sampling results are input into the decoder network, which contains two parallel prediction branches: a pose prediction head and an intensity prediction head. The pose prediction head predicts the 3D coordinates of the intersection point between the ray and the target object's surface to generate point cloud coordinate information. The intensity prediction head predicts the reflection intensity value corresponding to that intersection point to generate point cloud intensity information.

[0119] Optionally, to ensure the performance of the diffusion model, unoccupied meshes can be masked to significantly reduce the computational cost of the network.

[0120] S730 determines the third point cloud data based on point cloud coordinate information and point cloud intensity information.

[0121] Based on the point cloud coordinate information and point cloud intensity information generated by decoding, the third point cloud data is determined.

[0122] Optionally, to improve the accuracy of the third-point cloud data and avoid the illusionary effects that do not conform to physical laws during the generation process, it is also necessary to process the third-point cloud data according to rules, removing points that do not conform to physical laws. For example, for a target object, filtering rules can be set based on the prior geometric features and physical attributes of its category, and points that do not meet the category constraints can be removed by rule-based elimination.

[0123] For example, for points belonging to the "vehicle" category in the third point cloud data, if their spatial position is suspended above the ground at a certain height (e.g., more than 0.5 meters) without any supporting structure, they are determined to be an anomaly that does not conform to physical laws and are removed; or, if a point belonging to the "vehicle" category is a discrete point and is far away from other points belonging to the "vehicle" category, it is determined to be an anomaly and removed. Through the above rule-based processing, geometric distortion and semantic errors in the generation process can be effectively suppressed, improving the credibility and usability of the generated point cloud.

[0124] In this embodiment, a diffusion model is used to generate spatial features, and a third point cloud data is generated through ray sampling and decoding. By using a parallel prediction process involving both a pose prediction head and an intensity prediction head, decoupled modeling of the geometric structure and reflection intensity is achieved. The pose prediction head is used to recover the precise three-dimensional position of the target object's surface, ensuring the spatial and structural accuracy of the generated point cloud. The intensity prediction head is used to predict the reflection characteristics of different materials to obtain more realistic point cloud intensity information, which better matches the actual acquisition characteristics of the lidar.

[0125] In the above embodiments, it is necessary to predict the global point cloud data and extract the second point cloud data from it, which involves high computational complexity and leads to a relative increase in overall time consumption. Therefore, this disclosure also provides two methods for directly generating the second point cloud data to reduce computational complexity.

[0126] Figure 8 The diagram shown is a flowchart illustrating the steps of predicting space occupancy from simulated synthetic point cloud data to obtain space occupancy information, according to an embodiment of this disclosure. Figure 8 As shown in the embodiments of this disclosure, the steps for predicting space occupancy of simulated synthetic point cloud data to obtain space occupancy information include the following steps.

[0127] S810, obtain the point cloud mask of the target object.

[0128] The S820 uses point cloud masks to process simulated synthetic point cloud data.

[0129] Target extraction is performed on simulated synthetic point cloud data using point cloud masks. For each point in the simulated synthetic point cloud data, its corresponding value in the point cloud mask is queried, and only the point cloud corresponding to the target object is retained, while the remaining points are removed.

[0130] S830 performs space occupancy prediction on the processed simulated synthetic point cloud data to obtain the space occupancy information corresponding to the target object.

[0131] Through the above processing, the complete simulated synthetic point cloud data is cropped into point cloud data that only retains the target object, and the static background point cloud and dynamic target point cloud corresponding to the target scene are removed.

[0132] In this step, space occupancy prediction is performed only on the point cloud data corresponding to the target object, and the corresponding space occupancy information of the target object is obtained accordingly. The specific implementation method of space occupancy prediction is similar to that in the above embodiment, and will not be repeated here.

[0133] Accordingly, the step of predicting the spatial occupancy information through the diffusion model to obtain the second point cloud data of the target object includes: inputting the spatial occupancy information corresponding to the target object into the diffusion model to obtain the second point cloud data.

[0134] In this step, the diffusion model processes only the spatial occupancy information of the target object, directly obtaining the second point cloud data of the target object. Compared to the global point cloud data generation method in the above embodiments, this embodiment significantly reduces the input data, lowers computational overhead, and increases generation efficiency.

[0135] Next, we will introduce another way to directly generate second point cloud data.

[0136] Figure 9 The diagram illustrates a step in predicting spatial occupancy information using a diffusion model to obtain the second point cloud data of a target object, according to an embodiment of this disclosure. Figure 9 As shown in this embodiment, the step of predicting spatial occupancy information and obtaining the second point cloud data of the target object through a diffusion model includes the following steps.

[0137] S910, obtain the point cloud mask of the target object.

[0138] The S920 uses point cloud masks to process spatial occupancy information and obtains the spatial occupancy information corresponding to the target object.

[0139] The spatial occupancy information includes global spatial occupancy information for both the target scene and the target object. Using point cloud masks, target extraction is performed on this spatial occupancy information. From the global spatial occupancy information covering the entire scene, the local spatial occupancy information corresponding to the target object is extracted. This local spatial occupancy information accurately describes the geometric contour of the target object itself.

[0140] S930 inputs the spatial occupancy information corresponding to the target object into the diffusion model to obtain the second point cloud data.

[0141] In this step, the diffusion model generates second point cloud data corresponding to the target object based on local space occupancy information. Point cloud masking significantly reduces the input data to the diffusion model, thus greatly improving generation efficiency. The specific implementation of this step is similar to the above embodiment and will not be repeated here.

[0142] The above embodiments describe in detail the specific implementation of the simulation-based point cloud data generation method in stages. To make the technical solution of this disclosure clearer, a general description is provided below from an overall perspective; parts not described in detail can be found in the above embodiments.

[0143] Figure 10 The diagram shown is a flowchart illustrating a simulation-based point cloud data generation method provided in another embodiment of this disclosure.

[0144] like Figure 10 As shown, firstly, the specific requirements for obtaining the point cloud data to be generated are determined. Based on the target object category specified in the specific requirements, the 3D model of the target object is retrieved from the pre-built dynamic asset library as the target 3D model; based on the target area specified in the specific requirements, the scene point cloud data corresponding to the target scene is retrieved from the static asset library collected from the actual vehicle as the background point cloud.

[0145] The retrieved 3D model of the target object is rendered to generate the first point cloud data of the target object. Based on the preset trajectory of the target object in the target scene, the first point cloud data is fused with the scene point cloud data to obtain simulated synthetic point cloud data. The simulated synthetic point cloud data consists of scene point cloud data containing real intensity information obtained from actual data collection, and the first point cloud data obtained through rendering that does not contain intensity information. In the following steps, the intensity information of the target object is predicted to obtain the second point cloud data of the target object. Finally, the second point cloud data is fused with the scene point cloud data to obtain the target point cloud data.

[0146] The following describes two paths for generating second point cloud data.

[0147] Path 1: Predict space occupancy of the simulated synthetic point cloud data to obtain global space occupancy information. Using a diffusion model, predict based on the global space occupancy information to obtain third point cloud data including the background and target object. Then, using a pre-generated point cloud mask of the target object, extract the second point cloud data corresponding to the target object from the third point cloud data.

[0148] This approach utilizes global space occupancy information containing the complete scene context, resulting in high generation accuracy, but also in high computational cost.

[0149] Path 2: First, use point cloud masks to obtain local spatial occupancy information containing only the target object, and then use a diffusion model to generate second point cloud data containing only the target object based on the local spatial occupancy information.

[0150] This path only processes the space occupancy information of the target object, which greatly reduces the input data, reduces computational overhead, and increases generation efficiency.

[0151] The above text combined Figures 1 to 10 The method embodiments of this disclosure have been described in detail below, in conjunction with... Figure 11 The apparatus embodiments of this disclosure are described in detail below. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the foregoing method embodiments.

[0152] Figure 11 The diagram shown is a structural schematic of a simulation-based point cloud data generation device provided in an embodiment of this disclosure. Figure 11 As shown, the simulation-based point cloud data generation device 1100 of this embodiment includes: an acquisition module 1110, a first prediction module 1120, a second prediction module 1130, and a fusion module 1140.

[0153] The acquisition module 1110 is configured to acquire simulated synthetic point cloud data, which includes scene point cloud data of the target scene and first point cloud data of the target object. The first point cloud data does not include intensity information.

[0154] The first prediction module 1120 is configured to predict the space occupancy of the simulated synthetic point cloud data to obtain space occupancy information.

[0155] The second prediction module 1130 is configured to predict the second point cloud data of the target object based on the spatial occupancy information using a diffusion model. The second point cloud data includes intensity information.

[0156] The fusion module 1140 is configured to fuse the second point cloud data with the scene point cloud data to obtain the target point cloud data in the target scene.

[0157] Below, for reference Figure 12 To describe an electronic device according to embodiments of the present disclosure.

[0158] Figure 12 The diagram shown is a structural schematic of an electronic device provided according to an embodiment of this disclosure. Figure 12 As shown, the electronic device 1200 includes one or more processors 1210 and memory 1220.

[0159] The processor 1210 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1200 to perform desired functions.

[0160] The memory 1220 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1210 may execute the program instructions to implement the simulation-based point cloud data generation methods of the various embodiments of this disclosure described above, and / or other desired functions.

[0161] In one example, the electronic device 1200 may also include an input device 1230 and an output device 1240, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0162] The input device 1230 may include, for example, a keyboard, a mouse, etc.

[0163] The output device 1240 can output various information to the outside, including target point cloud data. The output device 1240 may include, for example, a display, a projection device, a communication network and its connected remote output devices, etc.

[0164] Of course, for the sake of simplicity, Figure 12 Only some of the components of the electronic device 1200 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1200 may include any other suitable components depending on the specific application.

[0165] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the simulation-based point cloud data generation methods according to various embodiments of this disclosure described above.

[0166] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0167] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the simulation-based point cloud data generation method according to various embodiments of this disclosure described above.

[0168] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0169] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0170] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0171] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0172] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0173] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for generating point cloud data based on simulation, characterized in that, include: Acquire simulated synthetic point cloud data, which includes scene point cloud data of the target scene and first point cloud data of the target object, wherein the first point cloud data does not include intensity information; Space occupancy prediction is performed on the simulated synthetic point cloud data to obtain space occupancy information; By using a diffusion model and based on the spatial occupancy information, a second point cloud data of the target object is obtained, and the second point cloud data includes intensity information. The second point cloud data is fused with the scene point cloud data to obtain the target point cloud data under the target scene.

2. The method according to claim 1, characterized in that, The target scene includes a static background and dynamic targets; The step of predicting space occupancy in the simulated synthetic point cloud data to obtain space occupancy information includes: Perform semantic segmentation on the scene point cloud data to obtain multiple frames of static background point cloud corresponding to the static background and multiple frames of dynamic target point cloud corresponding to the dynamic target; The multiple frames of static background point cloud are overlaid to generate a dense background point cloud; Space occupancy prediction is performed on the dense background point cloud, the multi-frame dynamic target point cloud, and the first point cloud data to obtain the space occupancy information.

3. The method according to claim 1, characterized in that, The space occupancy information has a higher voxel grid resolution in both the length and width directions than in the height direction.

4. The method according to claim 1, characterized in that, The acquisition of simulated synthetic point cloud data includes: The scene point cloud data of the target scene is obtained by collecting data from the target scene. Obtain the target 3D model corresponding to the target object; Render the target 3D model to obtain the first point cloud data of the target object; The first point cloud data and the scene point cloud data are fused to obtain the simulated synthesized point cloud data.

5. The method according to claim 4, characterized in that, The step of fusing the first point cloud data and the scene point cloud data includes: Obtain a preset trajectory corresponding to the scene point cloud data, wherein the preset trajectory represents the movement trajectory of the target object in the target scene; Based on the preset trajectory, the first point cloud data and the scene point cloud data are fused together.

6. The method according to claim 1, characterized in that, The step of predicting the spatial occupancy information using a diffusion model to obtain the second point cloud data of the target object includes: Obtain the point cloud mask of the target object; The space occupancy information is input into the diffusion model to obtain the third point cloud data corresponding to the simulated synthetic point cloud data; The second point cloud data is extracted from the third point cloud data using the point cloud mask.

7. The method according to claim 6, characterized in that, The step of inputting the space occupancy information into the diffusion model to obtain the third point cloud data corresponding to the simulated synthetic point cloud data includes: The spatial features corresponding to the spatial occupancy information are generated using the diffusion model. Ray sampling is performed on the spatial features, and the sampling results of the ray sampling operation are decoded to generate point cloud coordinate information and point cloud intensity information corresponding to the spatial occupancy information; The third point cloud data is determined based on the point cloud coordinate information and the point cloud intensity information.

8. The method according to claim 1, characterized in that, The step of predicting space occupancy in the simulated synthetic point cloud data to obtain space occupancy information includes: Obtain the point cloud mask of the target object; The simulated synthetic point cloud data is processed using the point cloud mask. Space occupancy prediction is performed on the processed simulated synthetic point cloud data to obtain the space occupancy information corresponding to the target object; The step of predicting the spatial occupancy information using a diffusion model to obtain the second point cloud data of the target object includes: The spatial occupancy information corresponding to the target object is input into the diffusion model to obtain the second point cloud data.

9. The method according to claim 1, characterized in that, The step of predicting the spatial occupancy information using a diffusion model to obtain the second point cloud data of the target object includes: Obtain the point cloud mask of the target object; Using the point cloud mask, the space occupancy information is processed to obtain the space occupancy information corresponding to the target object; The spatial occupancy information corresponding to the target object is input into the diffusion model to obtain the second point cloud data.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 9 by executing the executable instructions.

Images