Point cloud data generation method and apparatus, electronic device, and storage medium
By generating target point cloud data using a diffusion model and a point cloud variational autoencoder, the problems of poor efficiency and controllability in LiDAR point cloud data generation are solved, achieving efficient and controllable point cloud data generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-14
- Publication Date
- 2026-07-24
Smart Images
Figure CN122453648A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a point cloud data generation method, apparatus, electronic device, and storage medium. Background Technology
[0002] Light Detection and Ranging (LiDAR) point cloud data generation is a core enabling technology for autonomous driving development. The generated LiDAR point cloud data can realistically simulate diverse road scenarios in the real world. By providing efficient, safe, and comprehensive data support, it significantly reduces R&D costs and risks while improving the reliability and safety of the system.
[0003] Current methods for generating LiDAR point cloud data typically rely on a single-frame point cloud autoencoder. However, this approach suffers from low efficiency due to a lack of efficient point cloud representation methods, and the fact that it processes raw point cloud data alone results in poor controllability. Summary of the Invention
[0004] In view of the above problems, this application proposes a point cloud data generation method, apparatus, electronic device and storage medium to solve the problems of poor efficiency and controllability in point cloud data generation.
[0005] According to one aspect of an embodiment of this application, a point cloud data generation method is provided, the method comprising: Using a diffusion model, target latent space features are generated based on the target noise sequence and target control conditions; Using a point cloud variational autoencoder, the target latent space features are decoded and voxel rendered to obtain a target point cloud data sequence under the target control conditions.
[0006] Optionally, the point cloud variational autoencoder is trained in the following manner: Using the first model to be trained, the voxel information of the first sample point cloud data sequence in cylindrical coordinate system is determined. The latent space features of the first sample are determined based on the first sample point cloud data sequence and the voxel information of the first sample. Decoding and voxel rendering are performed based on the latent space features of the first sample to obtain the reconstructed point cloud data sequence of the first sample. The result of the first loss function is obtained based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence. Training is considered complete after the result of the first loss function satisfies the first condition, and the first trained model is used as the point cloud variational autoencoder.
[0007] Optionally, determining the latent space features of the first sample based on the first sample point cloud data sequence and the first sample voxel information includes: Based on the first sample point cloud data sequence and the first sample voxel information, determine the first sample feature of the first sample point cloud data sequence; The first sample features are spatiotemporally compressed and encoded to obtain the latent space features of the first sample.
[0008] Optionally, the step of obtaining the result of the first loss function based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence includes: Multiple target losses are determined based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence; the multiple target losses include multiple of the following: relative entropy loss, voxel reconstruction loss, depth loss, reflection intensity loss, masking loss, and regularization loss. The first loss function is obtained by weighted summation of the multiple target losses.
[0009] Optionally, the diffusion model is trained in the following manner: Using the point cloud variational autoencoder, the voxel information of the second sample point cloud data sequence in cylindrical coordinates is determined, and the latent space features of the second sample are determined based on the second sample point cloud data sequence and the voxel information. The second sample noise sequence is obtained by adding noise to the latent space features of the second sample using the second training model. The second training model is then trained using the second sample noise sequence and sample control conditions. The trained second training model is then used as the diffusion model.
[0010] Optionally, the step of training the second training model using the second sample noise sequence and sample control conditions, and using the trained second training model as the diffusion model, includes: The predicted velocity characteristics are determined using the second sample noise sequence and sample control conditions, and the actual velocity characteristics are determined based on the second sample latent space characteristics. The result of the second loss function is obtained based on the predicted speed features and the actual speed features. After the result of the second loss function satisfies the second condition, the training is determined to be complete, and the second model to be trained after training is used as the diffusion model.
[0011] Optionally, the target control conditions include at least one of the following: target scene description information, target foreground object boundary information, and target road structure map information.
[0012] According to another aspect of the embodiments of this application, a point cloud data generation apparatus is provided, the apparatus comprising: The generation module is used to generate target latent space features based on the target noise sequence and target control conditions using a diffusion model. The decoding and rendering module is used to decode and render voxels based on the target latent space features using a point cloud variational autoencoder to obtain a target point cloud data sequence under the target control conditions.
[0013] Optionally, the point cloud variational autoencoder is trained using the following modules: The first training module is used to determine the voxel information of the first sample point cloud data sequence in cylindrical coordinates using the first model to be trained, determine the latent space features of the first sample based on the first sample point cloud data sequence and the voxel information of the first sample, and perform decoding and voxel rendering based on the latent space features of the first sample to obtain the reconstructed point cloud data sequence of the first sample. The first determining module is used to obtain the result of the first loss function based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence; after the result of the first loss function satisfies the first condition, it determines that the training is completed, and uses the first trained model as the point cloud variational autoencoder.
[0014] Optionally, the first training module is specifically used to determine the first sample feature of the first sample point cloud data sequence based on the first sample point cloud data sequence and the first sample voxel information; and to perform spatiotemporal compression encoding on the first sample feature to obtain the first sample latent space feature.
[0015] Optionally, the first determining module is specifically used to determine multiple target losses based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence; the multiple target losses include multiple of relative entropy loss, voxel reconstruction loss, depth loss, reflection intensity loss, masking loss, and regularization loss; and the result of the first loss function is obtained by weighted summation of the multiple target losses.
[0016] Optionally, the diffusion model is trained using the following modules: The second determining module is used to determine the second sample voxel information in the cylindrical coordinate system of the second sample point cloud data sequence using the point cloud variational autoencoder, and to determine the second sample latent space features based on the second sample point cloud data sequence and the second sample voxel information. The second training module is used to add noise to the latent space features of the second sample using the second model to be trained to obtain a second sample noise sequence, and to train the second model to be trained using the second sample noise sequence and sample control conditions, and to use the trained second model to be trained as the diffusion model.
[0017] Optionally, the second training module is specifically used to determine the predicted velocity features using the second sample noise sequence and sample control conditions, and to determine the actual velocity features based on the second sample latent space features; to obtain the result of the second loss function based on the predicted velocity features and the actual velocity features; to determine that the training is complete after the result of the second loss function satisfies the second condition; and to use the trained second model as the diffusion model.
[0018] Optionally, the target control conditions include at least one of the following: target scene description information, target foreground object boundary information, and target road structure map information.
[0019] According to another aspect of the embodiments of this application, an electronic device is provided, the electronic device including a processor and a computer-readable storage medium storing a computer program; when the computer program is executed by the processor, the processor causes the processor to perform the point cloud data generation method as described in any of the preceding claims.
[0020] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the point cloud data generation method as described in any of the preceding claims.
[0021] In this embodiment, a diffusion model is used to generate target latent space features based on the target noise sequence and target control conditions. A point cloud variational autoencoder is then used to decode and voxel-render based on these features, resulting in a target point cloud data sequence under the target control conditions. Therefore, this embodiment utilizes a diffusion model and a point cloud variational autoencoder to generate target point cloud data sequences based on the target noise sequence and target control conditions. On one hand, the point cloud variational autoencoder enables the generation of long-term target point cloud data sequences, improving efficiency compared to single-frame point cloud autoencoder processing. On the other hand, the addition of a target control condition injection mechanism enables the generation of target point cloud data sequences under target control conditions, improving controllability and achieving precise control over point cloud data generation compared to methods that rely solely on point cloud data processing.
[0022] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description
[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some drawings of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 This is a flowchart of a point cloud data generation method according to an embodiment of this application; Figure 2 This is a flowchart of a point cloud variational autoencoder training method according to an embodiment of this application; Figure 3 This is a schematic diagram of a horizontal cross-section of a point cloud voxel in cylindrical coordinate system according to an embodiment of this application; Figure 4 This is a schematic diagram of the encoder structure of a point cloud variational autoencoder according to an embodiment of this application; Figure 5 This is a schematic diagram of a decoder structure for a point cloud variational autoencoder according to an embodiment of this application; Figure 6 This is a flowchart of a diffusion model training method according to an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a diffusion model according to an embodiment of this application; Figure 8 This is a structural block diagram of a point cloud data generation device according to an embodiment of this application; Figure 9 This is a structural block diagram of an electronic device according to an embodiment of this application; Figure 10 This is a structural block diagram of a computer-readable storage medium according to an embodiment of this application. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0026] Example 1: Reference Figure 1 The diagram shows a flowchart of a point cloud data generation method according to an embodiment of this application.
[0027] like Figure 1 As shown, the point cloud data generation method may include the following steps: Step 101: Using a diffusion model, generate target latent space features based on the target noise sequence and target control conditions.
[0028] In this embodiment, the diffusion model is pre-trained using a second sample noise sequence and sample control conditions. The second sample noise sequence is obtained by adding noise to the second sample latent space features, which are generated by a point cloud variational autoencoder based on the second sample point cloud data sequence. The specific training process of the diffusion model will be described in detail in the embodiments below.
[0029] For example, the target noise sequence can be obtained by randomly sampling Gaussian distributed noise. It can be represented as ,in, Represents a standard Gaussian distribution. The covariance matrix represents the noise.
[0030] After obtaining the target noise sequence and target control conditions, the target noise sequence and target control conditions are used as input to the diffusion model. The diffusion model can then be used to generate the target latent space features based on the target noise sequence and target control conditions.
[0031] Step 102: Using a point cloud variational autoencoder, decode and voxel render the target latent space features to obtain a target point cloud data sequence under the target control conditions.
[0032] In this embodiment, the point cloud variational autoencoder is pre-trained using a first sample point cloud data sequence. The specific training process of the point cloud variational autoencoder will be described in detail in the embodiments below.
[0033] The point cloud variational autoencoder includes a decoding and rendering module. After generating target latent space features based on the target noise sequence and target control conditions using a diffusion model, the target latent space features are input into the decoding and rendering module of the point cloud variational autoencoder. The decoding and rendering module performs decoding and voxel rendering based on the target latent space features to obtain the target point cloud data sequence under the target control conditions.
[0034] For example, the target point cloud data sequence may include at least one consecutive frame of target point cloud data. Each frame of target point cloud data may include at least one of the following pieces of information for each target 3D point in that frame of point cloud: Cartesian coordinates, reflection intensity, depth, etc.
[0035] In this embodiment, a diffusion model and a point cloud variational autoencoder are used for processing. A target point cloud data sequence can be generated based on the target noise sequence and the target control conditions. On the one hand, the point cloud variational autoencoder enables the generation of long-term target point cloud data sequences, which improves the efficiency of point cloud data generation compared to the single-frame point cloud autoencoder processing method. On the other hand, a target control condition injection mechanism is added, which enables the generation of target point cloud data sequences under target control conditions. Compared to the method of processing point cloud data alone, this improves the controllability of point cloud data generation and achieves precise control of point cloud data generation.
[0036] Example 2: For example, the target control conditions can be set manually according to actual needs. The target control conditions may include, but are not limited to, at least one of the following: target scene description information, target foreground object boundary information, and target road structure map information. This approach allows for a more comprehensive and accurate description of the target control conditions. By incorporating an injection mechanism for at least one target control condition, the generation of target point cloud data sequences under target control conditions is achieved, improving the controllability of point cloud data generation compared to methods based on a single control condition.
[0037] For example, the target scene description information may include information such as the weather, time of day, scene type, and background content corresponding to the target scene. The target foreground object boundary information is three-dimensional information and may include the target foreground object's size information (such as length, width, and height), position information (such as center point coordinates), orientation information, and object category (such as vehicle, pedestrian, cyclist, traffic cone, etc.). The target road structure map information may include target road layout information and target foreground object boundary information, etc. The target foreground object boundary information included in the target road structure map information is two-dimensional information, obtained by mapping the three-dimensional target foreground object boundary information onto the two-dimensional target road structure map. The target road layout information may include target road boundary lines, target road lane dividers, etc.
[0038] Example 3: Reference Figure 2 The diagram shows a flowchart of a point cloud variational autoencoder training method according to an embodiment of this application.
[0039] like Figure 2 As shown, the point cloud variational autoencoder training method may include the following steps: Step 201: Using the first model to be trained, determine the voxel information of the first sample point cloud data sequence in cylindrical coordinates; determine the latent space features of the first sample based on the first sample point cloud data sequence and the first sample voxel information; and perform decoding and voxel rendering based on the first sample latent space features to obtain the reconstructed point cloud data sequence of the first sample.
[0040] In this embodiment of the application, multiple first sample point cloud data sequences for training point cloud variational autoencoders are obtained in advance, and the multiple first sample point cloud data sequences are used as training samples for training point cloud variational autoencoders, with each first sample point cloud data sequence serving as a training sample.
[0041] For example, at least one consecutive frame of first raw point cloud data (e.g., LiDAR point cloud data) can be acquired by a data acquisition vehicle as a first sample point cloud data sequence. Each frame of first raw point cloud data may include at least one of the following information for each first raw point in that frame: Cartesian coordinates, reflection intensity, depth, etc.
[0042] The first model to be trained refers to the point cloud variational autoencoder to be trained. After inputting the first sample point cloud data sequence into the first model to be trained, the first sample voxel information of the first sample point cloud data sequence in cylindrical coordinates is determined using the first model to be trained. The first sample latent space features are determined based on the first sample point cloud data sequence and the first sample voxel information. Decoding and voxel rendering are performed based on the first sample latent space features to obtain the first sample reconstructed point cloud data sequence.
[0043] Step 202: Obtain the result of the first loss function based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence. After the result of the first loss function satisfies the first condition, determine that the training is complete, and use the first trained model as the point cloud variational autoencoder.
[0044] The result of the first loss function can be obtained based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence. If the result of the first loss function does not meet the first condition, the first model to be trained is iteratively adjusted and training continues. Once the result of the first loss function meets the first condition, training is considered complete, and the trained first model to be trained is used as the point cloud variational autoencoder. The first condition can be set according to actual needs; this embodiment does not impose any restrictions on it.
[0045] First, each first sample point cloud data sequence is voxelized in cylindrical coordinates to determine the first sample voxel information of the first sample point cloud data sequence in cylindrical coordinates. A voxel is short for volume pixel, which is the smallest unit of digital data in three-dimensional space, similar to a pixel in two-dimensional space.
[0046] For example, the process of cylindrical coordinate system voxelization may include: Suppose the Cartesian coordinates of any first original point in the first sample point cloud data sequence are ( The cylindrical coordinates corresponding to the first original point can be calculated using the following formula. :
[0047] Where x represents the x-axis coordinate in Cartesian coordinates, y represents the y-axis coordinate in Cartesian coordinates, and r represents the radial distance. The x-axis represents the horizontal angle, and z represents the vertical distance relative to the horizontal plane formed by the x-axis and y-axis.
[0048] Assume the effective range of point cloud data in cylindrical coordinates is:
[0049] in, Indicates the minimum radial distance. Indicates the maximum radial distance. Indicates the minimum horizontal angle. Indicates the maximum horizontal angle. Indicates the minimum height. Indicates the maximum height.
[0050] Assuming the size of each voxel is , Radial step size, The angle step size is... Let the height step size be , This indicates that the number of voxels required for a single frame of point cloud data is... H represents the radial voxel count, W represents the angular voxel count, and C represents the height voxel count. Indicates rounding up. Calculates the first origin point. Corresponding voxel coordinates , Represents radial voxel coordinates, Represents the number of voxel coordinates of the angle. Represents the height voxel coordinates, The value indicates rounding down. The voxel coordinates of the first origin point represent the voxel position to which the first origin point belongs.
[0051] After converting the input first sample point cloud data sequence to cylindrical coordinates, a 4D (four-dimensional) first sample voxel raster can be obtained. The first sample voxel grid is the first sample voxel information of the first sample point cloud data sequence in cylindrical coordinates. Here, T represents the frame number, and the value of each voxel in V is 0 or 1. 1 indicates that the voxel contains the first original point in the first sample point cloud data sequence, and 0 indicates that the voxel does not contain the first original point in the first sample point cloud data sequence.
[0052] In this embodiment of the application, a method for voxelizing point cloud data into cylindrical coordinates is proposed. Figure 3 This is a schematic diagram of a horizontal cross-section of a point cloud voxel in a cylindrical coordinate system according to an embodiment of this application. Compared with point cloud voxels in a Cartesian coordinate system, point cloud voxelization in a cylindrical coordinate system can avoid the waste of voxels in a Cartesian coordinate system, and its voxel granularity decreases with decreasing distance, which is more in line with the distribution characteristics of point cloud that is denser in the near and sparser in the far, and can more efficiently characterize point cloud.
[0053] Then, based on the first sample point cloud data sequence and the first sample voxel information, voxel feature aggregation and encoding are performed to determine the latent space features of the first sample.
[0054] The first model to be trained may include a feature aggregation module. This module performs voxel feature aggregation based on the first sample point cloud data sequence and the voxel information of the first sample, thereby determining the first sample features of the first sample point cloud data sequence. Voxel feature aggregation may include in-voxel point feature aggregation and bird's-eye view column feature aggregation. In-voxel point feature aggregation uses a learnable MLP (Multiple Layer Perceptron) to aggregate the information of all first original points within each voxel in the first sample point cloud data sequence (each original point's information may include at least one of Cartesian coordinates, reflection intensity, and depth) into a C-dimensional voxel feature. Bird's-eye view column feature aggregation uses a learnable MLP to aggregate the voxel features of C voxels with the same bird's-eye view projection coordinates into a C-dimensional bird's-eye view feature in an H×W bird's-eye view space, thus obtaining the bird's-eye view feature. The bird's-eye view features are the first sample features of the first sample point cloud data sequence.
[0055] Example 4: Exemplarily, the process of determining the first sample latent space features based on the first sample point cloud data sequence and the first sample voxel information may include: determining the first sample features of the first sample point cloud data sequence based on the first sample point cloud data sequence and the first sample voxel information; and performing spatiotemporal compression encoding on the first sample features to obtain the first sample latent space features. In this way, a point cloud variational autoencoder with a spatiotemporal compression mechanism can perform spatiotemporal compression on long-time-series point cloud data, improving the efficiency of point cloud data generation compared to a single-frame point cloud autoencoder lacking a spatiotemporal compression mechanism, while also being compatible with spatiotemporal compression of single-frame point cloud data, thereby achieving efficient representation of point cloud data.
[0056] The first model to be trained may include an encoder, which performs spatiotemporal compression encoding on the features of the first sample to determine the latent space features of the first sample. Figure 4 This is a schematic diagram of the encoder structure of a point cloud variational autoencoder according to an embodiment of this application. Figure 4 As shown, compression is mainly achieved through three cascaded downsampling modules. Each downsampling module consists of a 3D causal convolutional residual module and a downsampling layer (spatial downsampling is achieved through 2D convolution with a stride of 2, and temporal downsampling is achieved through average pooling). After spatiotemporal compression by the encoder, the latent space feature distribution can be obtained. , This represents the mean. The variance is represented by the latent space feature distribution, from which the latent space feature characteristics of the first sample can be obtained. M is a value set according to actual needs (such as 16), thus achieving not only 8*8 compression in the spatial dimension but also nearly 4 times compression in the temporal dimension. At the same time, this scheme retains the ability to process single-frame point clouds, meaning it is applicable even when T=1.
[0057] Then, the latent space features of the first sample are decoded to determine the decoded features of the first sample.
[0058] For example, the first model to be trained may include a decoder, which decodes the latent space features of the first sample to determine the decoded features of the first sample. The decoded features of the first sample may include the reconstructed voxel information of the first sample. Reconstructing features from the first sample . Figure 5 This is a schematic diagram of a decoder structure for a point cloud variational autoencoder according to an embodiment of this application, which converts the latent space features of the first sample... After processing by the relevant modules in the decoder, the reconstructed voxel information of the first sample can be obtained. Reconstructing features from the first sample .
[0059] Then, voxel rendering is performed based on the decoding features of the first sample to determine the reconstructed point cloud data sequence of the first sample.
[0060] For example, the first model to be trained may include a voxel rendering module, which performs voxel rendering on the decoded features of the first sample to obtain a first sample reconstructed point cloud data sequence. For example, the first sample reconstructed point cloud data sequence may include at least one consecutive frame of first sample reconstructed point cloud data. Each frame of first sample reconstructed point cloud data may include at least one of the following pieces of information for each first sample reconstructed point in that frame: Cartesian coordinates, reflection intensity, depth, etc.
[0061] For example, the voxel rendering process is as follows: Given a ray r = o + hd emitted from a starting point o along direction d, where h represents the ray propagation distance, Nr points are uniformly sampled along the ray r. Reconstructing the feature space from the first sample Interpolation is used to obtain each sampling point Corresponding features Using two learnable MLPs respectively Opacity mapped to this sampling point and reflection intensity The weight of each sampling point can be calculated based on the opacity. This allows us to obtain the depth of the first sample reconstruction point. and reflection intensity , This represents the distance the light travels from the sampling point to the starting point. Based on the depth, the Cartesian coordinates of the first reconstructed sample point can be calculated as o+. d.
[0062] In this embodiment, the feature aggregation module can be omitted and the first sample point cloud data sequence and the first sample voxel information can be used directly as the input of the encoder. When rendering voxels, only the depth needs to be rendered.
[0063] For example, after obtaining the Cartesian coordinates of each reconstruction point, the voxel coordinates of each first sample reconstruction point are determined according to the above-described cylindrical coordinate system voxelization method. If the voxel information of the first sample reconstruction... If the value corresponding to the voxel coordinates of the first sample reconstruction point is 1, then the first sample reconstruction point is retained; if the voxel information in the first sample reconstruction is... If the value corresponding to the voxel coordinate of the first sample reconstruction point is 0, then the first sample reconstruction point is deleted.
[0064] Example 5: Exemplarily, the process of obtaining the result of the first loss function based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence includes: determining multiple target losses based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence; the multiple target losses include multiple of relative entropy loss, voxel reconstruction loss, depth loss, reflection intensity loss, masking loss, and regularization loss; and performing a weighted summation of the multiple target losses to obtain the result of the first loss function. In this method, the weighted summation of multiple loss functions can take into account the losses of the encoder, decoder, and voxel rendering module, thereby ensuring the accuracy of the encoder, decoder, and voxel rendering module.
[0065] For example, the first loss function can be expressed as in, yes The relative entropy loss between the standard normal distribution and the standard normal distribution (i.e., KL loss). It is the voxel reconstruction loss, and BCE represents the binary classification cross-entropy. It is a deep loss. This represents the depth of the first original point corresponding to the first sample reconstruction point. For the loss of reflection intensity, This represents the reflection intensity of the first original point corresponding to the first sample reconstruction point. The correspondence between the first sample reconstruction point and the first original point can be determined by confirming the correspondence between the nearest first sample reconstruction point and the first original point. For mask loss, it means that the residual transmittance of invalid rays must be 100%. For regularization loss, These are constant weighting coefficients.
[0066] Example 6: Reference Figure 6 The diagram shows a flowchart of a diffusion model training method according to an embodiment of this application.
[0067] like Figure 6 As shown, the diffusion model training method may include the following steps: Step 601: Using the point cloud variational autoencoder, determine the second sample voxel information of the second sample point cloud data sequence in cylindrical coordinates, and determine the second sample latent space features based on the second sample point cloud data sequence and the second sample voxel information.
[0068] In this embodiment of the application, multiple second sample point cloud data sequences and sample control conditions for training the diffusion model are predetermined, and the multiple second sample point cloud data sequences and sample control conditions are used as training samples for training the diffusion model.
[0069] For example, at least one consecutive frame of second raw point cloud data (e.g., LiDAR point cloud data) can be acquired by a data acquisition vehicle as a second sample point cloud data sequence. Each frame of second raw point cloud data may include at least one of the following information for each second raw point in the frame: Cartesian coordinates, reflection intensity, depth, etc. The system simultaneously acquires sample image information for each frame using a data acquisition vehicle and determines the sample road structure map information for each frame from a high-precision map. It then establishes the correspondence between the second sample point cloud data sequence, sample image information, and sample road structure map information based on timestamps (the closest timestamps correspond). A multimodal large model is used to determine the sample scene description information corresponding to the sample image information. A 3D detection model is used to detect the sample foreground object boundary information based on the second sample point cloud data sequence and the corresponding sample image information. The sample foreground object boundary information is then drawn in the corresponding sample road structure map information. Different object categories can be distinguished by border color. Finally, the system obtains the sample scene description information, sample foreground object boundary information, and sample road structure map information (including the drawn sample foreground object boundary information) corresponding to each frame of the second original point cloud data. At least one of these three information is used as the sample control condition for the second sample point cloud data sequence.
[0070] Using the trained point cloud variational autoencoder, the voxel information of the second sample point cloud data sequence in cylindrical coordinates is determined, and the latent space features of the second sample are determined based on the second sample point cloud data sequence and the voxel information. The specific process is similar to the relevant part of step 201 above, and can be referred to the relevant description above.
[0071] Step 602: Add noise to the latent space features of the second sample using the second training model to obtain the second sample noise sequence. Use the second sample noise sequence and sample control conditions to train the second training model, and use the trained second training model as the diffusion model.
[0072] The second model to be trained refers to the diffusion model to be trained. First, noise is added to the latent space features of the second sample using the second model to be trained to obtain the second sample noise sequence. Then, the second model to be trained is trained using the second sample noise sequence and sample control conditions. The trained second model to be trained is then used as the diffusion model.
[0073] Example 7: Exemplarily, the process of training the second training model using the second sample noise sequence and sample control conditions, and using the trained second training model as the diffusion model, may include: determining the predicted velocity features using the second sample noise sequence and sample control conditions, and determining the actual velocity features based on the second sample latent space features; obtaining the result of the second loss function based on the predicted velocity features and the actual velocity features; iteratively adjusting the second training model and continuing training when the result of the second loss function does not meet the second condition; determining that training is complete after the result of the second loss function meets the second condition, and using the trained second training model as the diffusion model. The second condition can be set according to actual needs, and this embodiment does not limit it. In this method, the diffusion model adopts a velocity prediction approach. Compared with the noise prediction of traditional diffusion models, velocity prediction can more efficiently model the dynamic correlation of long-term data, improve generation accuracy and sampling efficiency, and better adapt to the needs of multi-condition controllable generation.
[0074] Figure 7 This is a schematic diagram of a diffusion model according to an embodiment of this application.
[0075] like Figure 7 As shown, during the training of the diffusion model (excluding the encoder), the encoder of the trained point cloud variational autoencoder determines the latent space features of the second sample based on the second sample point cloud data sequence and the voxel information of the second sample. Latent space features of the second sample Forward diffusion yields a second sample noise sequence with added noise. Among them, time step , , It is a predefined constant. .
[0076] To achieve controllable generation of point cloud data, this embodiment injects sample scene description information, sample foreground object boundary information, and sample road structure map information as conditions into the second training model. Specifically, the sample road structure map information is obtained by performing spatiotemporal compression at the same rate using a pre-trained 3D image VAE (Variational Auto-Encoder), aligning it with the second sample noise sequence, and then concatenating the compressed image features with the second sample noise sequence along the feature dimension to achieve conditional injection. The sample scene description information is encoded using a pre-trained text encoder to obtain encoded text features, which are then projected onto the same feature space as the second sample noise sequence. The two projected features are then subjected to self-attention to achieve conditional injection. The sample foreground object boundary information is temporally compressed using a learnable bounding box coordinate encoder to maintain temporal alignment with the second sample noise sequence, and cross-attention is used to achieve conditional injection. In this way, the second training model can predict the predicted velocity feature at step t based on the second sample noise sequence and sample control conditions. This constitutes the second loss function. ,in This represents the actual speed characteristic. It should be noted that, for the sake of simplicity, Figure 7 Saves time steps This involves injecting the process into all features for adaptive normalization. Specific processing modules include... Figure 7 As shown.
[0077] After training and obtaining the point cloud variational autoencoder and diffusion model, the point cloud data generation method described in the above embodiment can be executed using the point cloud variational autoencoder and diffusion model. The target noise sequence and target control conditions are input into the diffusion model, with the target noise sequence replacing... Figure 7 The output of the forward diffusion is used to generate predicted velocity features through a velocity prediction sampling process based on the target noise sequence and target control conditions using a diffusion model. Target latent space features are then generated based on these predicted velocity features. Finally, a point cloud variational autoencoder decodes and renders the target latent space features to obtain the target point cloud data sequence under the stated target control conditions. The specific process for generating the target latent space features from the predicted velocity features can be handled based on practical experience with diffusion model processing, and will not be discussed in detail in this embodiment.
[0078] It should be noted that the point cloud data generation method involved in the above embodiments may include at least one of Embodiments 1 to 7. For example, Embodiment 1 can be implemented as an independent embodiment, Embodiment 1+2 can be implemented as an independent embodiment, Embodiment 1+3 can be implemented as an independent embodiment, Embodiment 1+2+3 can be implemented as an independent embodiment, Embodiment 1+3+4 can be implemented as an independent embodiment, Embodiment 1+2+3+4 can be implemented as an independent embodiment, Embodiment 1+3+5 can be implemented as an independent embodiment, Embodiment 1+2+3+5 can be implemented as an independent embodiment, Embodiment 1+3+4+5 can be implemented as an independent embodiment, Embodiment 1+2+3+4+5 can be implemented as an independent embodiment, Embodiment 1+6 can be implemented as an independent embodiment, Embodiment 1+2+6 can be implemented as an independent embodiment, Embodiment 1+3+6 can be implemented as an independent embodiment, Embodiment 1+2+3+6 can be implemented as an independent embodiment, Embodiment 1+3+4+6 can be implemented as an independent embodiment, Embodiment 1+2+3+4+6 can be implemented as an independent embodiment, Embodiment 1+3+5+6 can be implemented as an independent embodiment. Examples 1+2+3+5+6, 1+3+4+5+6, 1+2+3+4+5+6, 1+6+7, 1+2+6+7, 1+3+6+7, 1+2+3+6+7, 1+3+4+6+7, 1+3+6+7, 1+2+3+6+7, 1+3+4+6+7, 1+2+3+4+6+7, 1+3+5+6+7, 1+2+3+5+6+7, 1+3+4+5+6+7, 1+2+3+4+5+6+7, and so on, are not limited to these examples and will not be listed here.
[0079] Reference Figure 8 The diagram shows a structural block diagram of a point cloud data generation device according to an embodiment of this application.
[0080] like Figure 8 As shown, the point cloud data generation device may include the following modules: The generation module 801 is used to generate target latent space features based on the target noise sequence and target control conditions using a diffusion model; The decoding and rendering module 802 is used to decode and render voxels based on the target latent space features using a point cloud variational autoencoder to obtain a target point cloud data sequence under the target control conditions.
[0081] Optionally, the point cloud variational autoencoder is trained using the following modules: The first training module is used to determine the voxel information of the first sample point cloud data sequence in cylindrical coordinates using the first model to be trained, determine the latent space features of the first sample based on the first sample point cloud data sequence and the voxel information of the first sample, and perform decoding and voxel rendering based on the latent space features of the first sample to obtain the reconstructed point cloud data sequence of the first sample. The first determining module is used to obtain the result of the first loss function based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence; after the result of the first loss function satisfies the first condition, it determines that the training is completed, and uses the first trained model as the point cloud variational autoencoder.
[0082] Optionally, the first training module is specifically used to determine the first sample feature of the first sample point cloud data sequence based on the first sample point cloud data sequence and the first sample voxel information; and to perform spatiotemporal compression encoding on the first sample feature to obtain the first sample latent space feature.
[0083] Optionally, the first determining module is specifically used to determine multiple target losses based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence; the multiple target losses include multiple of relative entropy loss, voxel reconstruction loss, depth loss, reflection intensity loss, masking loss, and regularization loss; and the result of the first loss function is obtained by weighted summation of the multiple target losses.
[0084] Optionally, the diffusion model is trained using the following modules: The second determining module is used to determine the second sample voxel information in the cylindrical coordinate system of the second sample point cloud data sequence using the point cloud variational autoencoder, and to determine the second sample latent space features based on the second sample point cloud data sequence and the second sample voxel information. The second training module is used to add noise to the latent space features of the second sample using the second model to be trained to obtain a second sample noise sequence, and to train the second model to be trained using the second sample noise sequence and sample control conditions, and to use the trained second model to be trained as the diffusion model.
[0085] Optionally, the second training module is specifically used to determine the predicted velocity features using the second sample noise sequence and sample control conditions, and to determine the actual velocity features based on the second sample latent space features; to obtain the result of the second loss function based on the predicted velocity features and the actual velocity features; to determine that the training is complete after the result of the second loss function satisfies the second condition; and to use the trained second model as the diffusion model.
[0086] Optionally, the target control conditions include at least one of the following: target scene description information, target foreground object boundary information, and target road structure map information.
[0087] In this embodiment, a diffusion model and a point cloud variational autoencoder are used for processing. A target point cloud data sequence can be generated based on the target noise sequence and the target control conditions. On the one hand, the point cloud variational autoencoder enables the generation of long-term target point cloud data sequences, which improves the efficiency of point cloud data generation compared to the single-frame point cloud autoencoder processing method. On the other hand, a target control condition injection mechanism is added, which enables the generation of target point cloud data sequences under target control conditions. Compared to the method of processing point cloud data alone, this improves the controllability of point cloud data generation and achieves precise control of point cloud data generation.
[0088] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.
[0089] In embodiments of this application, an electronic device is also provided. This electronic device may include a processor and a computer-readable storage medium storing a computer program; when the computer program is executed by the processor, the processor performs the point cloud data generation method of any of the above embodiments.
[0090] Reference Figure 9 This diagram illustrates a structural block diagram of an electronic device according to an embodiment of this application. Figure 9 As shown, the electronic device 11 includes a processor 111 and a computer-readable storage medium 112, on which a computer program 1121 is stored.
[0091] The processor 111 is used to execute the computer program 1121 stored on the computer-readable storage medium 112. When the processor 111 executes the computer program 1121, it implements the point cloud data generation method of any of the above embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0092] The processor 111 mentioned above may include, but is not limited to: a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0093] The computer-readable storage medium 112 mentioned above may include, but is not limited to: read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), electronically erasable programmable read-only memory (EEPROM), hard disk, floppy disk, flash memory, etc.
[0094] In embodiments of this application, a computer-readable storage medium is also provided, on which a computer program is stored. The computer program can be executed by a processor of an electronic device. When the computer program is executed by the processor, the processor performs the point cloud data generation method as described in any of the above embodiments.
[0095] Reference Figure 10 This diagram illustrates a structural block diagram of a computer-readable storage medium according to an embodiment of this application. Figure 10 As shown, a computer program 211 is stored on a computer-readable storage medium 21. When the computer program 211 is executed by a processor, it causes the processor to perform the point cloud data generation method as described in any of the above embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0096] In the embodiments of this application, a computer program product is also provided, on which a computer program is stored, which can be executed by a processor of an electronic device. When the computer program is executed by the processor, the processor performs the point cloud data generation method as described in any of the above embodiments.
[0097] The various embodiments in this specification are related to each other and are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.
[0098] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and regulations of the locality and with authorization from the owner of the relevant device.
[0099] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.
[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.
[0101] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
[0102] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0103] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0104] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0106] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0107] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. In summary, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for generating point cloud data, characterized in that, The method includes: Using a diffusion model, target latent space features are generated based on the target noise sequence and target control conditions; Using a point cloud variational autoencoder, the target latent space features are decoded and voxel rendered to obtain a target point cloud data sequence under the target control conditions.
2. The method according to claim 1, characterized in that, The target control conditions include at least one of the following: target scene description information, target foreground object boundary information, and target road structure diagram information.
3. The method according to claim 1 or 2, characterized in that, The point cloud variational autoencoder is trained in the following manner: Using the first model to be trained, the voxel information of the first sample point cloud data sequence in cylindrical coordinate system is determined. The latent space features of the first sample are determined based on the first sample point cloud data sequence and the voxel information of the first sample. Decoding and voxel rendering are performed based on the latent space features of the first sample to obtain the reconstructed point cloud data sequence of the first sample. The result of the first loss function is obtained based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence. Training is considered complete after the result of the first loss function satisfies the first condition, and the first trained model is used as the point cloud variational autoencoder.
4. The method according to claim 3, characterized in that, The step of determining the latent space features of the first sample based on the first sample point cloud data sequence and the first sample voxel information includes: Based on the first sample point cloud data sequence and the first sample voxel information, determine the first sample feature of the first sample point cloud data sequence; The first sample features are spatiotemporally compressed and encoded to obtain the latent space features of the first sample.
5. The method according to claim 3 or 4, characterized in that, The step of obtaining the result of the first loss function based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence includes: Multiple target losses are determined based on the first sample point cloud data sequence, the first sample voxel information, the first sample latent space features, and the first sample reconstructed point cloud data sequence; the multiple target losses include multiple of the following: relative entropy loss, voxel reconstruction loss, depth loss, reflection intensity loss, masking loss, and regularization loss. The first loss function is obtained by weighted summation of the multiple target losses.
6. The method according to any one of claims 1 to 5, characterized in that, The diffusion model is trained in the following manner: Using the point cloud variational autoencoder, the voxel information of the second sample point cloud data sequence in cylindrical coordinates is determined, and the latent space features of the second sample are determined based on the second sample point cloud data sequence and the voxel information. The second sample noise sequence is obtained by adding noise to the latent space features of the second sample using the second training model. The second training model is then trained using the second sample noise sequence and sample control conditions. The trained second training model is then used as the diffusion model.
7. The method according to claim 6, characterized in that, The step of training the second training model using the second sample noise sequence and sample control conditions, and using the trained second training model as the diffusion model, includes: The predicted velocity characteristics are determined using the second sample noise sequence and sample control conditions, and the actual velocity characteristics are determined based on the second sample latent space characteristics. The result of the second loss function is obtained based on the predicted speed features and the actual speed features. After the result of the second loss function satisfies the second condition, the training is determined to be complete, and the second model to be trained after training is used as the diffusion model.
8. A point cloud data generation device, characterized in that, The device includes: The generation module is used to generate target latent space features based on the target noise sequence and target control conditions using a diffusion model. The decoding and rendering module is used to decode and render voxels based on the target latent space features using a point cloud variational autoencoder to obtain a target point cloud data sequence under the target control conditions.
9. An electronic device, characterized in that, The electronic device includes a processor and a computer-readable storage medium on which a computer program is stored; When the computer program is executed by the processor, the processor performs the point cloud data generation method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the point cloud data generation method as described in any one of claims 1 to 7.