Point cloud simulation model training method and related device

Abstract

The present application relates to the field of AI. The present application specifically relates to a point cloud simulation model training method and a related device. The method comprises: acquiring a plurality of pieces of first point cloud data at a plurality of different positions in a target scene; on the basis of the plurality of pieces of first point cloud data, constructing an initial point cloud simulation model, and determining a plurality of pieces of pose information of a target object, parameters of the initial point cloud simulation model comprising the reflection intensity and loss rate of a Gaussian sphere corresponding to a point in a point cloud, and the target object is an object in the plurality of pieces of first point cloud data; on the basis of camera intrinsic parameters, respectively converting the plurality of pieces of first point cloud data into a plurality of first depth maps; and training the initial point cloud simulation model on the basis of the plurality of pieces of pose information and the plurality of first depth maps, so as to obtain a target point cloud simulation model. Real-time rendering of point clouds can be achieved, and rendering precision of point clouds can be improved.

Description

Point cloud simulation model training methods and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202411220171.5, filed on August 30, 2024, entitled “Point Cloud Simulation Model Training Method and Related Equipment”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI), and in particular to a point cloud simulation model training method and related equipment. Background Technology

[0003] In the past two years, with the rapid rise of neural radiance fields (NeRF) in the field of computer vision, many image generation methods based on neural radiance fields have emerged. These methods have also demonstrated the unique advantages of neural radiance fields over other traditional methods in application scenarios such as autonomous driving, digital humans, and 3D scene modeling.

[0004] Neural Radiation Field (NeRF) is an emerging method for scene representation and image rendering. It implicitly records scene representations within deep neural networks, using these networks to implicitly learn a static 3D scene, indirectly enabling tasks such as 3D scene reconstruction and new perspective image generation. This method uses a fully connected (non-convolutional) deep network, taking a single continuous 5D coordinate as input (spatial position (x, y, z) and viewing direction (θ, φ)) and outputting the volume density at that spatial position and view-related color information. The view is synthesized by querying the 5D coordinates along camera rays, and the output color and density are projected onto the image using classic volumetric rendering techniques. Furthermore, in point cloud simulation, by converting the point cloud into a distance image and using the point cloud depth information to supervise the scene's geometric distribution, scene reconstruction and end-to-end point cloud simulation can be achieved. However, due to the long simulation time required, real-time point cloud simulation is not possible.

[0005] 3D Gaussian splatting (3DGS) is another emerging method for scene reconstruction and image rendering. Compared to neural radiation field rendering, 3DGS abandons the process of emitting a ray from each pixel and sampling a large number of rays. Instead, it represents the scene as a Gaussian sphere and obtains the rendering result by projecting the Gaussian sphere, achieving a real-time rendering effect. However, this method can only be used for image rendering and cannot be used for point cloud rendering. Summary of the Invention

[0006] This application provides a point cloud simulation model training method and related equipment. Using this application is beneficial for achieving real-time rendering of point clouds and improving the rendering accuracy of point clouds.

[0007] In a first aspect, embodiments of this application provide a method for training a point cloud simulation model. The method includes:

[0008] The training device acquires multiple first point cloud data points at different locations in the target scene; based on the multiple first point cloud data points, the training device constructs an initial point cloud simulation model and determines multiple pose information of the target object. The parameters of the initial point cloud simulation model include the reflection intensity and loss rate of the Gaussian sphere corresponding to the point in the point cloud; the target object is the object in the multiple first point cloud data points; the training device converts the multiple first point cloud data points into multiple first depth maps based on camera intrinsic parameters; the training device trains the initial point cloud simulation model based on the multiple pose information and the multiple first depth maps to obtain the target point cloud simulation model.

[0009] It can be seen that by virtualizing multiple first point cloud data sequences into multiple first depth maps, and then using these first depth maps to train the initial point cloud simulation model, the joint optimization of point cloud depth information, reflection intensity, and loss rate is achieved. When rendering point clouds using the trained point cloud simulation model, it is beneficial to achieve real-time point cloud simulation and improve the simulation accuracy. Furthermore, the conversion of the first point cloud data into the first depth map used to train the initial point cloud simulation model is based on camera intrinsic parameters, making the trained point cloud simulation model applicable to the simulation of point clouds from different types of LiDAR.

[0010] In conjunction with the first aspect, in one possible implementation, the parameters of the initial point cloud simulation model include initial parameters of the background point cloud, initial parameters of the target object point cloud, and initial parameters of the road point cloud. The training device constructs the initial point cloud simulation model based on multiple first point cloud datasets, including:

[0011] The training device identifies point cloud data of the target object from multiple first point cloud datasets, removes the target object's point cloud data from each first point cloud dataset to obtain multiple second point cloud datasets; the training device constructs a point cloud map of the target scene based on the multiple second point cloud datasets; the initial parameters of the background point cloud and the road point cloud are determined based on the point cloud map of the target scene; the training device obtains the target object's mesh data based on the target object's point cloud data from the multiple first point cloud datasets; the training device downsamples the target object's mesh data to obtain the initial parameters of the target object's point cloud.

[0012] It can be seen that by decoupling the point cloud simulation model into background point cloud, road point cloud, and target object point cloud components, it is easier to optimize the road point cloud and target object point cloud components separately during subsequent training, thereby improving the accuracy of the point cloud simulation model. Furthermore, by using the target object's mesh data obtained from multiple first point cloud datasets to initialize the target object's point cloud parameters, the problem of inaccurate geometric reconstruction and penetration issues caused by the sparse target object point cloud in the first point cloud dataset is resolved.

[0013] In conjunction with the first aspect, in one possible implementation, the initial parameters of the background point cloud include the parameters of the Gaussian sphere corresponding to all points in the background point cloud; the initial parameters of the target object point cloud include the parameters of the Gaussian sphere corresponding to all points in the target object point cloud; the initial parameters of the road point cloud include the parameters of the Gaussian sphere corresponding to all points in the road point cloud; the principal axis normal of the Gaussian sphere corresponding to all points in the road point cloud is consistent with the direction of the road in the target scene; the parameters of the Gaussian sphere include position parameters, rotation parameters, opacity, scale parameters, reflection intensity, loss rate, and material feature parameters.

[0014] It can be seen that, for the parameters of the Gaussian sphere, in addition to the original position parameters, rotation parameters, opacity parameters, and scale parameters, reflection intensity, loss rate, and material feature parameters are also introduced. By training the parameters of the Gaussian sphere, it is beneficial to improve the accuracy of the obtained point cloud when using the point cloud simulation model.

[0015] In conjunction with the first aspect, in one possible implementation, the training device trains an initial point cloud simulation model based on multiple pose information and multiple first depth maps to obtain a target point cloud simulation model, including:

[0016] The training device performs hybrid spatiotemporal coding based on multiple pose information, multiple time points, and initial parameters of the target object point cloud to obtain reflection intensity offsets and loss rate offsets at multiple time points. These multiple time points represent the acquisition times of multiple first point cloud data sets. Based on the reflection intensity offsets and loss rate offsets at multiple time points, and the initial parameters of the target object point cloud, the training device obtains the first parameters of the target object point cloud at multiple time points. The first parameters include a first reflection intensity and a first loss rate. The first reflection intensity is determined based on the reflection intensity and reflection intensity offset in the initial parameters of the target object point cloud, and the time corresponding to the first reflection intensity is the same as the time corresponding to the reflection intensity offset. The first loss rate is determined based on the loss rate and loss rate offset in the initial parameters of the target object point cloud, and the time corresponding to the first loss rate is the same as the time corresponding to the loss rate offset. The training device then processes multiple third point cloud data sets... Gaussian rasterization is used to obtain multiple second depth maps; multiple third point cloud data correspond to multiple time points, each third point cloud data includes initial parameters of the background point cloud, initial parameters of the road point cloud, second parameters, and first parameters of the time point corresponding to the third point cloud data; the second parameters include the parameters of the target object point cloud's initial parameters other than reflection intensity and loss rate; the training device calculates depth loss value, reflection intensity loss value, and loss rate loss value based on multiple first depth maps and multiple second depth maps; the training device adjusts the initial parameters of the background point cloud, the target object point cloud, and the road point cloud based on the depth loss value, reflection intensity loss value, and loss rate loss value to obtain the target parameters of the background point cloud, the target object point cloud, and the road point cloud; the parameters of the target point cloud simulation model include the target parameters of the background point cloud, the target object point cloud, and the road point cloud.

[0017] It can be seen that by introducing hybrid spatiotemporal coding to predict the offsets of reflection intensity and loss rate of the target object point cloud at different times, and adjusting the initial parameters of the target object point cloud based on the offsets of reflection intensity and loss rate, the problem of inconsistent reflection intensity and loss rate of the target object point cloud caused by the different positions / orientations of the target object at different times in consecutive frames (i.e., multiple first point cloud data) is overcome, thus ensuring the simulation accuracy of the target object point cloud in dynamic scenes. When training the point cloud simulation model, the training device adjusts the parameters of the initial point cloud simulation model based on the depth loss value, reflection intensity loss value, and loss rate loss value, which helps to improve the accuracy of the point cloud simulation model.

[0018] In conjunction with the first aspect, in one possible implementation, the training device performs hybrid spatiotemporal coding based on multiple pose information, multiple time steps, and initial parameters of the target object point cloud to obtain reflection intensity offsets at multiple time steps and loss rate offsets at multiple time steps, including:

[0019] The training device encodes multiple pose information to obtain pose features; it encodes multiple time points to obtain temporal features; the training device extracts three three-dimensional planar features from the initial parameters of the target object point cloud based on multiple time points and multiple pose information; the training device processes the pose features, temporal features, and three three-dimensional planar features to obtain the reflection intensity offset and the loss rate offset at multiple time points.

[0020] It can be seen that by introducing hybrid spatiotemporal coding, the offsets of reflection intensity and loss rate of the target object point cloud at different times can be predicted. This facilitates the subsequent adjustment of the reflection intensity and loss rate in the initial parameters of the target object point cloud based on the offsets of reflection intensity and loss rate. This overcomes the problem of inconsistency in the reflectivity and loss rate of the target object point cloud caused by the different positions / orientations of the target object point cloud at different times between consecutive frames (i.e., the first point cloud data of multiple frames), thus ensuring the accuracy of the target object point cloud simulation in dynamic scenes.

[0021] In conjunction with the first aspect, in one possible implementation, the training device converts multiple first point cloud data into multiple first depth maps based on camera intrinsic parameters, including:

[0022] The training device projects the first point cloud data into a first depth map based on the camera intrinsic parameters, where the number of points projected onto a single pixel in the first point cloud data is the largest.

[0023] During projection, the number of points projected onto a single pixel in the first point cloud data is maximized, thereby preserving as much point cloud data information as possible in the first depth map. Compared to directly projecting the point cloud data into a distance image, this reduces the loss of point cloud data information. Using this first point cloud data to train a point cloud simulation model is beneficial to improving the accuracy of the point cloud simulation model.

[0024] Secondly, embodiments of this application provide a training device. The training device includes an acquisition unit, a determination unit, and a training unit.

[0025] The acquisition unit is used to acquire cloud data of multiple first points at multiple different locations in the target scene;

[0026] The determination unit is used to construct an initial point cloud simulation model based on multiple first point cloud data and determine multiple pose information of the target object. The parameters of the point cloud simulation model include the reflection intensity and loss rate of the Gaussian sphere corresponding to the point in the point cloud; the target object is the object in the multiple first point cloud data; and the multiple first point cloud data are converted into multiple first depth maps based on the camera intrinsic parameters.

[0027] The training unit is used to train an initial point cloud simulation model based on multiple pose information and multiple first depth maps to obtain a target point cloud simulation model.

[0028] In one feasible implementation, the parameters of the initial point cloud simulation model include initial parameters of the background point cloud, initial parameters of the target object point cloud, and initial parameters of the road point cloud; in terms of constructing the initial point cloud simulation model based on multiple first point cloud data, the determining unit is used for:

[0029] The process involves identifying point cloud data of the target object from multiple first point cloud datasets, removing the target object's point cloud data from each first point cloud dataset to obtain multiple second point cloud datasets, constructing a point cloud map of the target scene based on the multiple second point cloud datasets, determining the initial parameters of the background point cloud and road point cloud based on the point cloud map of the target scene, obtaining the target object's mesh data based on the target object's point cloud data from the multiple first point cloud datasets, and downsampling the target object's mesh data to obtain the initial parameters of the target object's point cloud.

[0030] In one feasible implementation, the initial parameters of the background point cloud include the parameters of the Gaussian sphere corresponding to all points in the background point cloud; the initial parameters of the target object point cloud include the parameters of the Gaussian sphere corresponding to all points in the target object point cloud; the initial parameters of the road point cloud include the parameters of the Gaussian sphere corresponding to all points in the road point cloud; the principal axis normal of the Gaussian sphere corresponding to all points in the road point cloud is consistent with the direction of the road in the target scene; the parameters of the Gaussian sphere include position parameters, rotation parameters, opacity, scale parameters, reflection intensity, loss rate, and material feature parameters.

[0031] In one feasible implementation, the training unit is used for:

[0032] Hybrid spatiotemporal coding is performed based on multiple pose information, multiple time points, and initial parameters of the target object point cloud to obtain reflection intensity offsets and loss rate offsets at multiple time points. The multiple time points refer to the acquisition times of multiple first point cloud data. Based on the reflection intensity offsets and loss rate offsets at multiple time points, and the initial parameters of the target object point cloud, the first parameters of the target object point cloud at multiple time points are obtained. The first parameters include a first reflection intensity and a first loss rate. The first reflection intensity is determined based on the reflection intensity and reflection intensity offset in the initial parameters of the target object point cloud, and the time corresponding to the first reflection intensity is the same as the time corresponding to the reflection intensity offset. The first loss rate is determined based on the loss rate and loss rate offset in the initial parameters of the target object point cloud, and the time corresponding to the first loss rate is the same as the time corresponding to the loss rate offset. Gaussian rasterization is then performed on the multiple third point cloud data. The process involves processing to obtain multiple second depth maps; multiple third point cloud data corresponding to multiple time points, each third point cloud data including initial parameters of the background point cloud, initial parameters of the road point cloud, second parameters, and first parameters of the time point corresponding to the third point cloud data; the second parameters include the parameters of the initial parameters of the target object point cloud except for reflection intensity and loss rate; depth loss value, reflection intensity loss value, and loss rate loss value are calculated based on multiple first depth maps and multiple second depth maps; the initial parameters of the background point cloud, the initial parameters of the target object point cloud, and the initial parameters of the road point cloud are adjusted based on the depth loss value, reflection intensity loss value, and loss rate loss value to obtain target parameters of the background point cloud, the target object point cloud, and the road point cloud, and the parameters of the target point cloud simulation model include the target parameters of the background point cloud, the target object point cloud, and the road point cloud.

[0033] In a feasible implementation, the training unit is specifically used for spatiotemporal coding based on multiple pose information, multiple time steps, and initial parameters of the target object point cloud to obtain reflection intensity offsets and loss rate offsets at multiple time steps:

[0034] Multiple pose information is encoded to obtain pose features; multiple time points are encoded to obtain temporal features; three three-dimensional planar features are extracted from the initial parameters of the target object point cloud based on multiple time points and multiple pose information; the pose features, temporal features and three-dimensional planar features are processed to obtain the reflection intensity offset and the loss rate offset at multiple time points.

[0035] In one feasible implementation, in the aspect of converting the plurality of first point cloud data into a plurality of first depth maps based on camera intrinsic parameters, the determining unit is specifically used for:

[0036] The first point cloud data is projected into a first depth map based on the camera intrinsic parameters, where the number of points in the first point cloud data projected onto a unique pixel is the largest.

[0037] Thirdly, embodiments of this application provide a training apparatus, including a processor and a memory. The memory is used to store program code. The processor is used to invoke the program code stored in the memory to execute the method provided in the first aspect or any possible implementation thereof.

[0038] Fourthly, embodiments of this application provide a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform a method as provided in any possible implementation of the first aspect.

[0039] Fifthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the method provided in any possible implementation of the first aspect.

[0040] It is understood that the training equipment described in the second or third aspect above is used to execute any of the methods provided in the first aspect, and the computer storage medium described in the fourth aspect and the computer program product described in the fifth aspect are both used to implement any of the methods provided in the first aspect. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0041] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0042] Figure 2 is a schematic diagram of a point cloud simulation model training process provided in an embodiment of this application;

[0043] Figure 3 is a flowchart illustrating a point cloud simulation model training method provided in an embodiment of this application;

[0044] Figure 4a is a comparison diagram of the normal of a Gaussian sphere being inconsistent with and consistent with the road direction according to an embodiment of this application;

[0045] Figure 4b is a schematic diagram of a road Gaussian sphere provided in an embodiment of this application;

[0046] Figure 5 is a schematic diagram of a hybrid spatiotemporal coding provided in an embodiment of this application;

[0047] Figure 6 shows a comparison of rendering effects for static scenes;

[0048] Figure 7 illustrates the before and after removal of the target object in the point cloud;

[0049] Figure 8 shows point cloud maps from different perspectives of a road scene;

[0050] Figure 9 is a schematic diagram illustrating different types of laser point cloud simulations;

[0051] Figure 10 is a schematic diagram of the structure of a training device provided in an embodiment of this application;

[0052] Figure 11 is a schematic diagram of another training device provided in an embodiment of this application. Detailed Implementation

[0053] The terms “first,” “second,” “third,” and “fourth,” etc., used in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order.

[0054] "Multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating three possible relationships. For example, A and / or B means: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0055] The embodiments of this application will now be described with reference to the accompanying drawings.

[0056] Referring to Figure 1, Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application. As shown in Figure 1, the system architecture includes a point cloud simulation device 101, multiple point cloud data acquisition devices 102, and a training device 103. There is one or more point cloud simulation devices 101.

[0057] The point cloud simulation device 101 can be a server, such as a cloud server, distributed server, rack server, blade server, tower server, etc.

[0058] The point cloud data acquisition device 102 can be a vehicle, a terminal device, etc. The terminal device can be a smartphone, smartwatch, smart bracelet, desktop computer, laptop, tablet, etc. The vehicle can be a bicycle, tricycle, small car, medium-sized car, etc.

[0059] Training device 103 is a device capable of data processing, such as a server or computer.

[0060] The training device 103 and the point cloud simulation device 101 can be the same device or different devices.

[0061] A point cloud data acquisition device (such as a lidar) 102 acquires multiple first point cloud data at multiple locations in the target scene and transmits these multiple first point cloud data to a training device 103. The training device 103 trains a target point cloud simulation model based on the multiple first point cloud data.

[0062] In a specific example, as shown in Figure 2, the training device 103 converts multiple first point cloud data into multiple first depth maps based on the intrinsic parameters of the first camera. The training device 103 performs target detection and tracking on the multiple first point cloud data using a target detector and a target tracker to obtain the point cloud data corresponding to the target objects in the multiple first point cloud data and multiple pose information corresponding to the target objects in the multiple first point cloud data. There can be multiple target objects, such as three. The training device 103 removes the point cloud data corresponding to the target objects from the multiple first point cloud data to obtain multiple second point cloud data. The training device 103 constructs a point cloud map of the target scene based on the multiple second point cloud data, and determines the initial parameters of the background point cloud and the road point cloud based on the point cloud map of the target scene. The training device 103 obtains the mesh data of the target objects based on the point cloud data of the target objects. In one example, the training device processes the point cloud data of the target objects based on the MV-deepSDF model to obtain the mesh data of the target objects. The mesh data of the target object is downsampled to obtain the initial parameters of the target object point cloud. The initial parameters of the background point cloud include the parameters of the Gaussian sphere corresponding to all points in the background point cloud, the target object point cloud, and the road point cloud. The principal axis normal of the Gaussian sphere corresponding to all points in the road point cloud is consistent with the direction of the road in the target scene. The parameters of the Gaussian sphere include position parameters, rotation parameters, opacity, scale parameters, reflection intensity, loss rate, and material feature parameters, which are used as learning parameters. The training device 103 trains the initial parameters of the background point cloud, the target object point cloud, and the road point cloud based on multiple pose information, multiple first point cloud data, and multiple first depth maps to obtain the target parameters of the background point cloud, the target object point cloud, and the road point cloud. The parameters of the trained point cloud simulation model include the target parameters of the target background point cloud, the background point cloud, and the road point cloud. The loss values ​​used during training include depth loss, reflection intensity loss, and loss rate loss.

[0063] The point cloud simulation device 101 acquires the pose information of the target object and the camera intrinsic parameters; the point cloud simulation device 101 processes the point cloud simulation model and the pose information of the target object to obtain the third depth map corresponding to the second time moment; the point cloud simulation device 101 processes the third depth map corresponding to the second time moment based on the camera intrinsic parameters to obtain the point cloud of the target scene at the second time moment.

[0064] It should be noted that the target point cloud simulation model is the depth map corresponding to the static point cloud of the target scene. The point cloud simulation device adjusts the depth information of the target object point cloud in the depth map corresponding to the static point cloud of the target scene based on the input pose, so that the pose of the target object in the point cloud of the target scene obtained by inverse mapping the adjusted depth map based on the input camera intrinsic parameters is consistent with the input pose. In other words, the point cloud simulation device can dynamically adjust the pose of the target object in the point cloud of the target scene, thereby realizing the dynamic real-time simulation of the point cloud of the target scene.

[0065] As can be seen, in this embodiment, during training, multiple first point cloud data sequences are virtually converted into multiple first depth maps. These first depth maps are then used to train the initial point cloud simulation model, achieving joint optimization of point cloud depth information, reflection intensity, and loss rate. When rendering point clouds using the trained point cloud simulation model, real-time point cloud simulation is achieved, while also improving simulation accuracy. Furthermore, converting the first point cloud data into first depth maps for training the initial point cloud simulation model is based on camera intrinsic parameters, enabling the trained point cloud simulation model to be applicable to the simulation of point clouds from different LiDAR models. By decoupling the point cloud simulation model into background point cloud, road point cloud, and target object point cloud components, subsequent optimization of the road and target object point cloud components during training is facilitated, thereby improving the accuracy of the point cloud simulation model.

[0066] Referring to Figure 3, Figure 3 is a flowchart illustrating a point cloud simulation model training method provided in an embodiment of this application. As shown in Figure 3, the method includes:

[0067] S301, The training device acquires cloud data of multiple first points at multiple different locations in the target scene.

[0068] Optionally, the first point cloud data can be acquired through LiDAR or through other means, such as acquiring the point cloud data from other devices after they have acquired it through LiDAR.

[0069] S302, The training equipment constructs an initial point cloud simulation model based on multiple first point cloud data and determines multiple pose information of the target object.

[0070] The target objects are objects within multiple first point cloud datasets. More specifically, the target objects are moving objects within these first point cloud datasets, such as people, vehicles, and animals. The parameters of the initial point cloud simulation model include the reflection intensity and loss rate of the Gaussian sphere corresponding to each point in the point cloud.

[0071] In one feasible implementation, the training device determines point cloud data of the target object from multiple first point cloud datasets. The training device removes the target object's point cloud data from each first point cloud dataset to obtain multiple second point cloud datasets, and determines the pose information of the target object in each first point cloud dataset. That is, the training device determines multiple pose information of the target object, and these multiple pose information correspond to multiple first point cloud datasets. The training device constructs a point cloud map of the target scene based on the multiple second point cloud datasets. The training device determines initial parameters of the background point cloud and the road point cloud based on the point cloud map of the target scene. The training device obtains the mesh data of the target object based on the point cloud data of the target object from the multiple first point cloud datasets. The training device downsamples the mesh data of the target object to obtain the initial parameters of the target object's point cloud. The initial point cloud simulation model includes the initial parameters of the background point cloud, the initial parameters of the target object's point cloud, and the initial parameters of the road point cloud.

[0072] For example, the training device processes multiple first point cloud datasets using an object detection-based tracking algorithm to determine the point cloud data of the target object within the multiple first point cloud datasets. Optionally, the object detection-based tracking algorithm can be a joint detection and embedding (JDE) algorithm or a simple online and real-time tracking (SORT) algorithm, and of course, other algorithms are also possible, without limitation here.

[0073] For example, the training device detects target objects in a plurality of first point cloud data based on an object detection algorithm to determine the target objects in the first point cloud data, and tracks the target objects in the plurality of first point cloud data based on an object tracking algorithm to determine the pose information of the target objects in the plurality of first point cloud data. Optionally, the object detection algorithm can be the YOLO algorithm, the Single Shot Multibox Detector (SSD) algorithm, or the R-CNN algorithm, etc., and other algorithms are also possible, without limitation. Optionally, the object tracking algorithm can be the SORT algorithm based on deep learning or the Robust Associations Multi-Pedestrian Tracking (BoT-SORT) algorithm, and other algorithms are also possible, without limitation.

[0074] Optionally, the training device downsamples multiple second point cloud data to obtain multiple fourth point cloud data; and constructs a point cloud map of the target scene based on the multiple fourth point cloud data.

[0075] By downsampling the second point cloud data, the amount of point cloud data can be reduced, which is beneficial to improving the efficiency of building point cloud maps of the target scene and also helps to reduce storage overhead.

[0076] In one feasible implementation, the initial parameters of the background point cloud include the parameters of the Gaussian sphere corresponding to all points in the background point cloud, the initial parameters of the target object point cloud include the parameters of the Gaussian sphere corresponding to all points in the target object point cloud, and the principal axis normal of the Gaussian sphere corresponding to all points in the road point cloud is consistent with the direction of the road in the target scene.

[0077] Specifically, for the Gaussian spheres corresponding to all points in the background point cloud, the target object point cloud, and the road point cloud, the parameters of each Gaussian sphere include location, rotation, opacity, scale, intensity, raydrop, and materialvec parameters.

[0078] It should be noted that the reflection intensity parameter of the Gaussian sphere corresponding to the midpoint of the point cloud is related to the surface material of the object, as well as the incident direction and distance of the pixel ray. The training device can determine the reflection intensity of the Gaussian sphere corresponding to the midpoint of the point cloud based on the surface material of the object and the incident direction and distance of the pixel ray. Specifically, the training device uses a multilayer perceptron (MLP) to process the features obtained by cascading d, γ(x), and materialvec to obtain the reflection intensity of the Gaussian sphere. Where d is the incident direction of the ray, γ(x) is the distance of the ray, and materialvec is used to characterize the surface material features of the object.

[0079] It should be noted that the points in the point cloud correspond to the pixels in the depth map. In this embodiment, the depth map can be viewed as a three-channel image, with the three channels representing depth, reflection intensity, and loss rate, respectively.

[0080] In one example, the reflection intensity parameter of the Gaussian sphere corresponding to the i-th point in the point cloud (including background point cloud, target object point cloud, or road point cloud) can be expressed as:

[0081] Where, α i Let α be the opacity of the Gaussian sphere corresponding to the i-th point. j Let be the opacity of the Gaussian sphere corresponding to the j-th point. T is the reflection intensity value of the pixel corresponding to the i-th point obtained by integration. iThis represents the weight of the reflection intensity of the Gaussian sphere corresponding to the i-th point. The reflection intensity attribute of each point in the point cloud is obtained through Gaussian rasterization based on Gaussian splashing, that is, the reflection intensity of each point in the point cloud is obtained by accumulating the values ​​of the reflection intensity parameter of the Gaussian sphere through which the ray of the pixel corresponding to that point passes.

[0082] The loss rate of the Gaussian sphere corresponding to each point in the point cloud is related to the incident direction and reflection intensity of the ray from the pixel corresponding to that point. The training device determines the loss rate of the Gaussian sphere corresponding to each point in the point cloud based on the incident direction and reflection intensity of the ray. Specifically, the training device processes the features obtained by cascading the incident direction and reflection intensity of the ray using an MLP to obtain the loss rate of the Gaussian sphere corresponding to each point.

[0083] In one example, the loss rate of the Gaussian sphere corresponding to the i-th point in the point cloud can be expressed as:

[0084] The loss rate attribute of each point in the point cloud is obtained through Gaussian rasterization based on Gaussian splashing. Specifically, the loss rate of each point in the point cloud is obtained by accumulating the loss rate parameters of the Gaussian sphere through which the ray passing from the corresponding pixel passes. To make the distribution of the simulated point cloud more realistic, points with a loss rate lower than a preset loss rate are discarded.

[0085] It should be understood that the reflection intensity of the Gaussian sphere corresponding to the i-th point refers to the signal intensity of the ray of the pixel corresponding to the i-th point reflected back by the Gaussian sphere, and the loss rate of the Gaussian sphere corresponding to the i-th point refers to the probability that the ray of the pixel corresponding to the i-th point, after passing through the Gaussian sphere, does not pass through the Gaussian sphere.

[0086] For each Gaussian sphere corresponding to a point in the road point cloud, in addition to setting the reflection intensity and loss rate of the Gaussian sphere as described above, the principal axis normal of the Gaussian sphere corresponding to all points in the road point cloud is made consistent with the direction of the road in the target scene. The left image of Figure 4a shows the case where the normal of the Gaussian sphere is inconsistent with the road direction, and the right image of Figure 4a shows the case where the normal of the Gaussian sphere is consistent with the road direction. By constraining the size of the Gaussian sphere, the road is reconstructed as a flat thin shell, as shown in Figure 4b, ensuring the accuracy of the road view geometry.

[0087] S303, The training device converts multi-frame first point cloud data into corresponding multi-frame first depth images based on camera intrinsic parameters.

[0088] In a feasible implementation, the point cloud simulation device projects a first point cloud data header into a first depth map based on camera intrinsic parameters. In one example, the camera intrinsic parameters are (k... x k y, W, H), where, (k x k y Let W be the camera's focal length, and W and H be the dimensions of the first depth image. It's important to note that to retain as much point cloud data information as possible in the first depth image, appropriate camera intrinsics are needed. To obtain suitable camera intrinsics, these intrinsics are used as optimization parameters. The optimization objective is to maximize the number of points in the point cloud data projected onto a unique pixel. Based on this idea, a nonlinear optimization method can be used to determine the appropriate camera intrinsics.

[0089] In one example, the nonlinear optimization method can be expressed as: P 2D =F(k) x ,k y ,W,H,P 3D ), P 2D argmax(count(P 2D ))

[0090] Among them, P 3D P represents the 3D coordinates of a point in the point cloud data. 2D These are the two-dimensional coordinates of the pixels in the depth map.

[0091] It can be seen that by projecting point cloud data into a depth map through camera intrinsics, and by setting the camera intrinsics during the projection process, the number of points projected onto a single pixel in the point cloud data can be maximized, so as to retain as much point cloud data information as possible. Compared with directly projecting point cloud data into a distance image, this reduces the loss of point cloud data information and is also more compatible with the Gaussian rasterized point cloud simulation framework of this application.

[0092] S304. The training device trains an initial point cloud simulation model based on multiple pose information and multiple first depth maps to obtain a target point cloud simulation model.

[0093] The parameters of the target point cloud simulation model include the target parameters of the target background point cloud, the target parameters of the background point cloud, and the target parameters of the road point cloud.

[0094] In one feasible implementation, the training device performs hybrid spatiotemporal coding based on multiple pose information, multiple first moments, and initial parameters of the target object point cloud to obtain multiple reflection intensity offsets and multiple loss rate offsets at the first moments; the multiple first moments are the acquisition moments of multiple first point cloud data.

[0095] In one example, the training device encodes multiple pose information to obtain pose features; it encodes multiple first moments to obtain temporal features; the training device extracts three triplane features from the initial parameters of the target object point cloud based on multiple first moments and multiple pose information; the training device processes the pose features, temporal features, and three triplane features to obtain the reflection intensity offset and the loss rate offset of multiple first moments.

[0096] Specifically, as shown in Figure 5, the training device encodes multiple pose information (i.e., pose information) through a pose encoding layer to obtain pose features; it encodes the aforementioned multiple first moments (i.e., time information) through a temporal encoding layer to obtain temporal features; the training device uses a spatiotemporal embedding layer to encode the initial parameters of the target object's point cloud and multiple first moments to obtain three three-dimensional planar features of the target object; these three three-dimensional planar features can be represented as (x,y,t), (y,z,t), and (x,z,t), respectively; where (x,y,z) represents spatial coordinates, and t represents time. There can be N target objects, where N is an integer greater than 1. The training device concatenates the three three-dimensional planar features, temporal features, and pose features of the target object to obtain concatenated features; the training device uses a multilayer perceptron machine to process the concatenated features to obtain the reflection intensity offsets at multiple first moments and the loss rate offsets at multiple times.

[0097] The training device obtains the first parameters of the target object point cloud at multiple first moments based on the reflection intensity offset and the loss rate offset at multiple first moments, as well as the initial parameters of the target object point cloud. The first parameters include the first reflection intensity and the first loss rate. The first reflection intensity is determined based on the reflection intensity and the reflection intensity offset in the initial parameters of the target point cloud object. The first moment corresponding to the first reflection intensity is the same as the first moment corresponding to the reflection intensity offset. The first loss rate is determined based on the loss rate in the initial parameters of the target point cloud object and the loss rate offset. The first moment corresponding to the first loss rate is the same as the first moment corresponding to the loss rate offset.

[0098] The training device performs Gaussian rasterization on multiple third point cloud data to obtain multiple second depth maps; the multiple third point cloud data correspond to the multiple first time points, and each third point cloud data includes the initial parameters of the background point cloud, the initial parameters of the road point cloud, the second parameters, and the first parameters of the same time point as the first time point corresponding to the third point cloud data; the second parameters include the parameters of the initial parameters of the target point cloud object other than the reflection intensity and the loss rate.

[0099] For example, the initial parameters of the target object point cloud include initial reflection intensity, initial loss rate, and other initial parameters. The reflection intensity offsets at three time points (including the reflection intensity offsets at time t1, t2, and t3) and the loss rate offsets at three time points (including the loss rate offsets at time t1, t2, and t3) are obtained as described above. The training device determines the reflection intensity at time t1, t2, and t3 based on the initial reflection intensity in the target object point cloud's initial parameters and the reflection intensity offsets at time t1, t2, and t3, respectively; these are the first reflection intensities at the three time points. Similarly, the training device determines the loss rate at time t1, t2, and t3 based on the initial loss rate in the target object point cloud's initial parameters and the loss rate offsets at time t1, t2, and t3, respectively; these are the first loss rates at the three time points.

[0100] The training device obtains three third point cloud data based on the first reflection intensity, the first loss rate, the second parameter, the initial parameters of the background point cloud, and the initial parameters of the road point cloud at three time points. The three third point cloud data correspond to the three time points. The third point cloud data at each of the three time points includes the second parameter, the initial parameters of the background point cloud, and the initial parameters of the road point cloud. The difference is that the third point cloud data at each of the three time points also includes the first reflection intensity and the first loss rate at the corresponding time point.

[0101] The training device calculates depth loss values, reflection intensity loss values, and loss rate loss values ​​based on multiple first depth maps and multiple second depth maps. The training device then adjusts the initial parameters of the background point cloud, the target object point cloud, and the road point cloud based on the depth loss values, the reflection intensity loss values, and the loss rate loss values ​​to obtain the target parameters of the background point cloud, the target object point cloud, and the road point cloud.

[0102] Optionally, the training device calculates a comprehensive loss value based on the depth loss value, the reflection intensity loss value, and the loss rate loss value, such as by weighted summation. The training device adjusts the initial parameters of the background point cloud, the target object point cloud, and the road point cloud based on the comprehensive loss value to obtain the target parameters of the background point cloud, the target object point cloud, and the road point cloud.

[0103] It should be noted that the above training process can be executed iteratively multiple times until the number of training iterations reaches the threshold or the above loss value converges.

[0104] It can be seen that during training, by virtually converting multiple first point cloud data sequences into multiple first depth maps, and then using these first depth maps to train the initial point cloud simulation model, the joint optimization of point cloud depth information, reflection intensity, and loss rate is achieved. When rendering point clouds using the trained point cloud simulation model, real-time point cloud simulation is facilitated, and the simulation accuracy is improved. Furthermore, converting the first point cloud data into first depth maps for training the initial point cloud simulation model is based on camera intrinsic parameters, making the trained point cloud simulation model applicable to the simulation of point clouds from different LiDAR models. By decoupling the point cloud simulation model into background point cloud, road point cloud, and target object point cloud components, subsequent optimization of the road and target object point cloud components during training is facilitated, thereby improving the accuracy of the point cloud simulation model. Furthermore, based on the target object's point cloud data obtained from multiple first point cloud datasets, the target object's mesh data is used to initialize the target object's point cloud parameters. This solves the problems of inaccurate geometric reconstruction and penetration caused by the sparse target object point cloud in the first point cloud dataset. By introducing hybrid spatiotemporal coding to predict the offsets of reflection intensity and loss rate of the target object's point cloud at different times, and adjusting the reflection intensity and loss rate in the initial parameters of the target object's point cloud based on these offsets, the problem of inconsistent reflection intensity and loss rate caused by the different positions / orientations of the target object at different times in consecutive frames (i.e., multiple first point cloud datasets) is overcome, ensuring the simulation accuracy of the target object's point cloud in dynamic scenes. When training the point cloud simulation model, the training device adjusts the parameters of the initial point cloud simulation model based on depth loss, reflection intensity loss, and loss rate loss values, which helps improve the accuracy of the point cloud simulation model.

[0105] The reasoning process based on the point cloud simulation model is described below.

[0106] The point cloud simulation device acquires the pose information of the target object and the camera intrinsic parameters; and processes the point cloud simulation model and the pose information of the target object to obtain the third depth map corresponding to the second time step; the point cloud simulation device processes the third depth map corresponding to the second time step based on the camera intrinsic parameters to obtain the point cloud of the target scene at the second time step.

[0107] It should be noted that the target point cloud simulation model is a depth map corresponding to the static point cloud of the target scene. The point cloud simulation device adjusts the depth information of the target object point cloud in the depth map corresponding to the static point cloud of the target scene based on the input pose, so that the pose of the target object in the point cloud of the target scene obtained by inverse mapping the adjusted depth map based on the input camera intrinsic parameters is consistent with the input pose. In other words, the point cloud simulation device can dynamically adjust the pose of the target object in the point cloud of the target scene, or input a pose sequence, thereby realizing dynamic real-time simulation of the point cloud of the target scene. Furthermore, by changing the input pose or pose sequence, point cloud simulation from different perspectives can be achieved. During training and simulation, the camera intrinsic parameters used for point cloud data corresponding to different types of laser devices are different. Therefore, by changing the camera intrinsic parameters, simulation of point cloud data corresponding to different types of laser devices can be achieved.

[0108] It should be noted that the point cloud data in this application can be point cloud data corresponding to LiDAR or point cloud data corresponding to millimeter-wave radar. The solution in this application can be applied to fields that require point cloud generation, such as: autonomous driving, AR / VR, high-precision surveying and mapping, 3D scene reconstruction, etc.

[0109] The beneficial effects of this application are explained below with reference to the accompanying drawings.

[0110] Figure 6 illustrates a comparison of rendering effects for static scenes. Figure 6a shows the point cloud image corresponding to the first point cloud data; Figure 6b shows the point cloud image corresponding to the point cloud image obtained using the proposed solution; Figure 6c shows the point cloud image corresponding to the point cloud obtained using the NeRF method; and Figure 6d shows the point cloud image corresponding to the point cloud obtained using the point cloud generator (PCGen). Using Figure 6a as a reference, the performance of the proposed solution, the NeRF method, and PCGen is shown in Table 1 below.

[0111] Table 1

[0112] It should be noted that the chamfer distance is used to characterize the similarity between two point cloud maps; the smaller the chamfer distance, the higher the similarity between the two point cloud maps. As shown in Table 1, the point cloud map using the scheme of this application is most similar to the reference point cloud map, and the scheme of this application has the highest rendering frame rate. Therefore, it is evident that the scheme of this application is superior to the other two schemes in both point cloud rendering accuracy and rendering efficiency.

[0113] Figure 7 illustrates the point cloud before and after the removal of the target object. Figures a and c in Figure 7 are point cloud diagrams before the target object removal, while figures b and d are point cloud diagrams after the target object has been removed, as achieved using the solution of this application. Therefore, it can be seen that the solution of this application can remove the target object from the point cloud data.

[0114] Figure 8 illustrates point cloud images from different perspectives of a road scene. In Figure 8, image a is the point cloud image of the original view road scene rendered according to the solution of this application. Images b and c in Figure 8 are the point cloud images of the left lane change and right lane change corresponding to image a, rendered according to the solution of this application, respectively. It is evident that the solution of this application can render point clouds from different perspectives for the same scene.

[0115] Figure 9 shows a point cloud simulation diagram corresponding to different types of laser equipment. In Figure 9, a is the point cloud diagram of the original view road scene, b is the point cloud diagram corresponding to the simulated M2 laser, and c is the point cloud diagram corresponding to the simulated M2 laser right lane change.

[0116] Referring to Figure 10, which is a schematic diagram of the structure of a training device provided in an embodiment of this application, the training device 1000 includes:

[0117] Acquisition unit 1001 is used to acquire multiple first point cloud data at multiple different locations in the target scene;

[0118] The determination unit 1002 is used to construct an initial point cloud simulation model based on multiple first point cloud data and determine multiple pose information of the target object. The parameters of the point cloud simulation model include the reflection intensity and loss rate of the Gaussian sphere corresponding to the point in the point cloud; the target object is the object in the multiple first point cloud data; and the multiple first point cloud data are converted into multiple first depth maps based on the camera intrinsic parameters.

[0119] Training unit 1003 is used to train an initial point cloud simulation model based on multiple pose information and multiple first depth maps to obtain a target point cloud simulation model.

[0120] In one feasible implementation, the parameters of the initial point cloud simulation model include initial parameters of the background point cloud, initial parameters of the target object point cloud, and initial parameters of the road point cloud; in terms of constructing the initial point cloud simulation model based on multiple first point cloud data, the determining unit 1002 is used for:

[0121] The process involves identifying point cloud data of the target object from multiple first point cloud datasets, removing the target object's point cloud data from each first point cloud dataset to obtain multiple second point cloud datasets, constructing a point cloud map of the target scene based on the multiple second point cloud datasets, determining the initial parameters of the background point cloud and road point cloud based on the point cloud map of the target scene, obtaining the target object's mesh data based on the target object's point cloud data from the multiple first point cloud datasets, and downsampling the target object's mesh data to obtain the initial parameters of the target object's point cloud.

[0122] In one feasible implementation, the initial parameters of the background point cloud include the parameters of the Gaussian sphere corresponding to all points in the background point cloud; the initial parameters of the target object point cloud include the parameters of the Gaussian sphere corresponding to all points in the target object point cloud; the initial parameters of the road point cloud include the parameters of the Gaussian sphere corresponding to all points in the road point cloud; the principal axis normal of the Gaussian sphere corresponding to all points in the road point cloud is consistent with the direction of the road in the target scene; the parameters of the Gaussian sphere include position parameters, rotation parameters, opacity, scale parameters, reflection intensity, loss rate, and material feature parameters.

[0123] In one feasible implementation, training unit 1003 is used for:

[0124] Hybrid spatiotemporal coding is performed based on multiple pose information, multiple time points, and initial parameters of the target object point cloud to obtain reflection intensity offsets and loss rate offsets at multiple time points. The multiple time points refer to the acquisition times of multiple first point cloud data. Based on the reflection intensity offsets and loss rate offsets at multiple time points, and the initial parameters of the target object point cloud, the first parameters of the target object point cloud at multiple time points are obtained. The first parameters include a first reflection intensity and a first loss rate. The first reflection intensity is determined based on the reflection intensity and reflection intensity offset in the initial parameters of the target object point cloud, and the time corresponding to the first reflection intensity is the same as the time corresponding to the reflection intensity offset. The first loss rate is determined based on the loss rate and loss rate offset in the initial parameters of the target object point cloud, and the time corresponding to the first loss rate is the same as the time corresponding to the loss rate offset. Gaussian rasterization is then performed on the multiple third point cloud data. The process involves processing to obtain multiple second depth maps; multiple third point cloud data corresponding to multiple time points, each third point cloud data including initial parameters of the background point cloud, initial parameters of the road point cloud, second parameters, and first parameters of the time point corresponding to the third point cloud data; the second parameters include the parameters of the initial parameters of the target object point cloud except for reflection intensity and loss rate; depth loss value, reflection intensity loss value, and loss rate loss value are calculated based on multiple first depth maps and multiple second depth maps; the initial parameters of the background point cloud, the initial parameters of the target object point cloud, and the initial parameters of the road point cloud are adjusted based on the depth loss value, reflection intensity loss value, and loss rate loss value to obtain target parameters of the background point cloud, the target object point cloud, and the road point cloud, and the parameters of the target point cloud simulation model include the target parameters of the background point cloud, the target object point cloud, and the road point cloud.

[0125] In a feasible implementation, the training unit 1003 is specifically used for: Spatiotemporal coding based on multiple pose information, multiple time points, and initial parameters of the target object point cloud to obtain reflection intensity offsets and loss rate offsets at multiple time points.

[0126] Multiple pose information is encoded to obtain pose features; multiple time points are encoded to obtain temporal features; three three-dimensional planar features are extracted from the initial parameters of the target object point cloud based on multiple time points and multiple pose information; the pose features, temporal features and three-dimensional planar features are processed to obtain the reflection intensity offset and the loss rate offset at multiple time points.

[0127] In one feasible implementation, in the aspect of converting the plurality of first point cloud data into a plurality of first depth maps based on camera intrinsic parameters, the determining unit 1002 is specifically used for:

[0128] The first point cloud data is projected into a first depth map based on the camera intrinsic parameters, where the number of points in the first point cloud data projected onto a unique pixel is the largest.

[0129] It is worth noting that the specific functional implementation of the training device 1000 is described in the relevant description of the point cloud simulation method above. For example, the acquisition unit 1001 is used to execute the relevant content of S301, the determination unit 1002 is used to execute the relevant content of S302 and S303, and the training unit 1003 is used to execute the relevant content of S304. Each unit or module in the training device 1000 can be individually or entirely merged into one or more other units or modules, or some of the units or modules can be further divided into multiple functionally smaller units or modules. This achieves the same operation without affecting the technical effect of the embodiments of this application. The above-mentioned units or modules are based on logical function division. In practical applications, the function of one unit (or module) is implemented by multiple units (or modules), or the function of multiple units (or modules) is implemented by one unit (or module).

[0130] Based on the description of the above method embodiments and related device embodiments, please refer to FIG11, which provides a schematic diagram of the structure of a training device 1100. The training device 1100 shown in FIG11 includes a memory 1101, a processor 1102, a communication interface 1103, and a bus 1104. The memory 1101, the processor 1102, and the communication interface 1103 are interconnected through the bus 1104.

[0131] Optionally, the memory 1101 can be a ROM, a static storage device, a dynamic storage device, or RAM.

[0132] The memory 1101 can store programs. When the program stored in the memory 1101 is executed by the processor 1102, the processor 1102 and the communication interface 1103 are used to execute the various steps of the point cloud simulation model training method of the embodiment shown in FIG3.

[0133] The processor 1102 uses a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), GPU, or one or more integrated circuits to execute relevant programs to implement the training method of the point cloud simulation model in the embodiment shown in Figure 3.

[0134] The processor 1102 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the flow scheduling in this application can be completed through integrated logic circuits in the hardware of the processor 1102 or through software instructions. Optionally, the processor 1102 can be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor is a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. Optional software modules are located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 1101. The processor 1102 reads the information in the memory 1101 and, in conjunction with its hardware, performs the functions required by the units included in the training device 1000 of this application embodiment, or executes the training method of the point cloud simulation model of the method embodiment of this application.

[0135] The communication interface 1103 uses transceiver-related devices, such as, but not limited to, transceivers, to enable communication between the training device 1100 and other devices or communication networks.

[0136] Bus 1104 may include a pathway for transmitting information between various components of training device 1100 (e.g., memory 1101, processor 1102, communication interface 1103).

[0137] It should be noted that although the training device 1100 shown in Figure 11 only illustrates the memory, processor, and communication interface, those skilled in the art should understand that in specific implementations, the training device 1100 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the training device 1100 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the training device 1100 may only include the devices necessary for implementing the embodiments of this application, and not necessarily all the devices shown in Figure 7.

[0138] This application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface to implement the point cloud simulation model training method of this application.

[0139] Optionally, as one implementation, the chip further includes a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the point cloud simulation model training method.

[0140] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of any of the above methods.

[0141] This application also provides a computer program product containing instructions. When the computer program product is run on a computer or processor, it causes the computer or processor to perform one or more steps of any of the methods described above.

[0142] Those skilled in the art will appreciate that the functionality described in conjunction with the various illustrative logic blocks, modules, and algorithmic steps disclosed herein can be implemented by hardware, software, firmware, or any combination thereof. If implemented in software, the functionality described by the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium that includes any medium facilitating the transfer of a computer program from one place to another (e.g., based on a communication protocol). In this way, the computer-readable medium may substantially correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product may comprise a computer-readable medium.

[0143] By way of example and not limitation, such computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection is properly referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are specifically referring to non-temporary tangible storage media. As used herein, disks and optical discs include Compact Discs (CDs), Laser Discs, Optical Discs, Digital Versatile Discs (DVDs), and Blu-ray Discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0144] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the functions described in the various illustrative logic blocks, modules, and steps described herein are provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, the techniques can be fully implemented within one or more circuit or logic elements.

[0145] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Optionally, the coupling, direct coupling, or communication connection shown or discussed between them may be through some interfaces, indirect coupling or communication connection of devices or units, such as electrical, mechanical, or other forms.

[0146] Optionally, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of this application are generated.

[0148] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A point cloud simulation model training method, characterized in that, The method includes: Acquire cloud data of multiple first points at multiple different locations in the target scene; An initial point cloud simulation model is constructed based on the multiple first point cloud data, and multiple pose information of the target object is determined. The parameters of the point cloud simulation model include the reflection intensity and loss rate of the Gaussian sphere corresponding to the point in the point cloud; the target object is the object in the multiple first point cloud data. Based on camera intrinsic parameters, the multiple first point cloud data are converted into multiple first depth maps; The initial point cloud simulation model is trained based on the multiple pose information and the multiple first depth maps to obtain the target point cloud simulation model.

2. The method according to claim 1, characterized in that, The parameters of the initial point cloud simulation model include the initial parameters of the background point cloud, the initial parameters of the target object point cloud, and the initial parameters of the road point cloud. The construction of the initial point cloud simulation model based on the plurality of first point cloud data includes: Determine the point cloud data of the target object from the plurality of first point cloud data, and remove the point cloud data of the target object from each of the first point cloud data to obtain a plurality of second point cloud data; A point cloud map of the target scene is constructed based on the multiple second point cloud data; the initial parameters of the background point cloud and the initial parameters of the road point cloud are determined based on the point cloud map of the target scene. The mesh data of the target object is obtained based on the point cloud data of the target object in the plurality of first point cloud data; The mesh data of the target object is downsampled to obtain the initial parameters of the point cloud of the target object.

3. The method according to claim 2, characterized in that, The initial parameters of the background point cloud include the parameters of the Gaussian sphere corresponding to all points in the background point cloud; the initial parameters of the target object point cloud include the parameters of the Gaussian sphere corresponding to all points in the target object point cloud; the initial parameters of the road point cloud include the parameters of the Gaussian sphere corresponding to all points in the road point cloud; the principal axis normal of the Gaussian sphere corresponding to all points in the road point cloud is consistent with the direction of the road in the target scene; the parameters of the Gaussian sphere include position parameters, rotation parameters, opacity, scale parameters, reflection intensity, loss rate, and material feature parameters.

4. The method according to claim 3, characterized in that, The step of training the initial point cloud simulation model based on the multiple pose information and the multiple first depth maps to obtain the target point cloud simulation model includes: Hybrid spatiotemporal coding is performed based on the multiple pose information, multiple time points, and the initial parameters of the target object point cloud to obtain the reflection intensity offset and the loss rate offset at multiple time points; the multiple time points are the acquisition times of the multiple first point cloud data. Based on the reflection intensity offset and the loss rate offset at the multiple time points, and the initial parameters of the target object point cloud, a first parameter of the target object point cloud at the multiple time points is obtained. The first parameter includes a first reflection intensity and a first loss rate. The first reflection intensity is determined based on the reflection intensity in the initial parameters of the target object point cloud and the reflection intensity offset. The time corresponding to the first reflection intensity is the same as the time corresponding to the reflection intensity offset. The first loss rate is determined based on the loss rate in the initial parameters of the target object point cloud and the loss rate offset. The time corresponding to the first loss rate is the same as the time corresponding to the loss rate offset. Multiple third point cloud data are subjected to Gaussian rasterization to obtain multiple second depth maps; the multiple third point cloud data correspond to the multiple time points, and each third point cloud data includes the initial parameters of the background point cloud, the initial parameters of the road point cloud, a second parameter, and a first parameter of the time point corresponding to the third point cloud data; the second parameter includes the parameters of the initial parameters of the target object point cloud other than the reflection intensity and the loss rate; Based on the plurality of first depth maps and the plurality of second depth maps, depth loss value, reflection intensity loss value and loss rate loss value are calculated; The initial parameters of the background point cloud, the initial parameters of the target object point cloud, and the initial parameters of the road point cloud are adjusted based on the depth loss value, the reflection intensity loss value, and the loss rate loss value to obtain the target parameters of the background point cloud, the target object point cloud, and the road point cloud. The parameters of the target point cloud simulation model include the target parameters of the background point cloud, the target object point cloud, and the road point cloud.

5. The method according to claim 4, characterized in that, The hybrid spatiotemporal coding based on the multiple pose information, multiple time points, and the initial parameters of the target object point cloud to obtain the reflection intensity offset and the loss rate offset at multiple time points includes: The multiple pose information is encoded to obtain pose features; the multiple time points are encoded to obtain time features; Based on the multiple time points and the multiple pose information, three three-dimensional planar features are extracted from the initial parameters of the point cloud of the target object; The pose features, the time features, and the three three-dimensional plane features are processed to obtain the reflection intensity offset and the loss rate offset at the multiple time points.

6. The method according to any one of claims 1-5, characterized in that, The process of converting the multiple first point cloud data into multiple first depth maps based on camera intrinsic parameters includes: The first point cloud data is projected into the first depth map based on the camera intrinsic parameters, wherein the number of points in the first point cloud data projected onto a unique pixel is the largest.

7. A training device, characterized in that, include: The acquisition unit is used to acquire cloud data of multiple first points at multiple different locations in the target scene; The determining unit is used to construct an initial point cloud simulation model based on the plurality of first point cloud data and determine multiple pose information of the target object. The parameters of the point cloud simulation model include the reflection intensity and loss rate of the Gaussian sphere corresponding to the point in the point cloud. The target object is an object in the plurality of first point cloud data. The plurality of first point cloud data are converted into multiple first depth maps based on camera intrinsic parameters. The training unit is used to train the initial point cloud simulation model based on the multiple pose information and the multiple first depth maps to obtain the target point cloud simulation model.

8. The training device according to claim 7, characterized in that, The parameters of the initial point cloud simulation model include initial parameters of the background point cloud, initial parameters of the target object point cloud, and initial parameters of the road point cloud; in the aspect of constructing the initial point cloud simulation model based on the plurality of first point cloud data, the determining unit is used for: Determine the point cloud data of the target object from the plurality of first point cloud data, and remove the point cloud data of the target object from each of the first point cloud data to obtain a plurality of second point cloud data; A point cloud map of the target scene is constructed based on the multiple second point cloud data. The initial parameters of the background point cloud and the initial parameters of the road point cloud are determined based on the point cloud map of the target scene. The mesh data of the target object is obtained based on the point cloud data of the target object in the plurality of first point cloud data; The mesh data of the target object is downsampled to obtain the initial parameters of the point cloud of the target object.

9. The training device according to claim 7 or 8, characterized in that, The initial parameters of the background point cloud include the parameters of the Gaussian sphere corresponding to all points in the background point cloud; the initial parameters of the target object point cloud include the parameters of the Gaussian sphere corresponding to all points in the target object point cloud; the initial parameters of the road point cloud include the parameters of the Gaussian sphere corresponding to all points in the road point cloud; the principal axis normal of the Gaussian sphere corresponding to all points in the road point cloud is consistent with the direction of the road in the target scene; the parameters of the Gaussian sphere include position parameters, rotation parameters, opacity, scale parameters, reflection intensity, loss rate, and material feature parameters.

10. The training device according to claim 9, characterized in that, The training unit is used for: Hybrid spatiotemporal coding is performed based on the multiple pose information, multiple time points, and the initial parameters of the target object point cloud to obtain the reflection intensity offset and the loss rate offset at multiple time points; the multiple time points are the acquisition times of the multiple first point cloud data. Based on the reflection intensity offset and the loss rate offset at the multiple time points, and the initial parameters of the target object point cloud, a first parameter of the target object point cloud at the multiple time points is obtained. The first parameter includes a first reflection intensity and a first loss rate. The first reflection intensity is determined based on the reflection intensity in the initial parameters of the target object point cloud and the reflection intensity offset. The time corresponding to the first reflection intensity is the same as the time corresponding to the reflection intensity offset. The first loss rate is determined based on the loss rate in the initial parameters of the target object point cloud and the loss rate offset. The time corresponding to the first loss rate is the same as the time corresponding to the loss rate offset. Multiple third point cloud data are subjected to Gaussian rasterization to obtain multiple second depth maps; the multiple third point cloud data correspond to the multiple time points, and each third point cloud data includes the initial parameters of the background point cloud, the initial parameters of the road point cloud, a second parameter, and a first parameter of the time point corresponding to the third point cloud data; the second parameter includes the parameters of the initial parameters of the target object point cloud other than the reflection intensity and the loss rate; Based on the plurality of first depth maps and the plurality of second depth maps, depth loss value, reflection intensity loss value and loss rate loss value are calculated; The initial parameters of the background point cloud, the initial parameters of the target object point cloud, and the initial parameters of the road point cloud are adjusted based on the depth loss value, the reflection intensity loss value, and the loss rate loss value to obtain the target parameters of the background point cloud, the target object point cloud, and the road point cloud. The parameters of the target point cloud simulation model include the target parameters of the background point cloud, the target object point cloud, and the road point cloud.

11. The training device according to claim 10, characterized in that, In the aspect of hybrid spatiotemporal coding based on the multiple pose information, multiple time points, and the initial parameters of the target object point cloud to obtain reflection intensity offsets and loss rate offsets at multiple time points, the training unit is specifically used for: The multiple pose information is encoded to obtain pose features; The multiple time points are encoded to obtain time features; Based on the multiple time points and the multiple pose information, three three-dimensional planar features are extracted from the initial parameters of the point cloud of the target object; The pose features, the time features, and the three three-dimensional plane features are processed to obtain the reflection intensity offset and the loss rate offset at the multiple time points.

12. The training device according to any one of claims 7-11, characterized in that, In the aspect of converting the plurality of first point cloud data into a plurality of first depth maps based on camera intrinsic parameters, the determining unit is specifically used for: The first point cloud data is projected into the first depth map based on the camera intrinsic parameters, wherein the number of points in the first point cloud data projected onto a unique pixel is the largest.

13. A training device, characterized in that, The method includes a processor and a memory, wherein the memory is used to store program code, and the processor is used to execute the program code to implement the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

15. A computer program product that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-6.

Images