Dynamic object three-dimensional reconstruction method and device, computer equipment and storage medium

Abstract

The invention relates to a dynamic object three-dimensional reconstruction method and device, computer equipment and a storage medium. The method comprises the following steps: extracting a cross-frame point cloud sequence of a target object from historical automatic driving data; processing the cross-frame point cloud sequence through space-time alignment and motion trail compensation to obtain a high-density accumulated point cloud; generating rays according to the sensor pose data and the high-density accumulated point cloud, sampling along the rays, and generating training data based on a sampling result; a double-branch neural network comprising geometric branches and radiation field branches is constructed, the geometric branches are used for modeling an object surface by adopting a symbolic distance field, and the radiation field branches are used for learning point cloud density and surface attributes by adopting a neural radiation field; performing iterative training on the double-branch neural network based on the training data; and extracting network parameters of the trained double-branch neural network to obtain a dynamic implicit representation model of the target object. According to the invention, the three-dimensional reconstruction precision of the dynamic object can be improved, and the space-time information storage efficiency of the dynamic object is improved.

Description

Technical Field

[0001] This application relates to the field of dynamic object 3D reconstruction, and in particular to a dynamic object 3D reconstruction method, apparatus, computer equipment and storage medium. Background Technology

[0002] Dynamic object 3D reconstruction is a core pillar of autonomous driving technology. Its role is not only to improve perception accuracy and safety, but also to lay the foundation for the large-scale deployment of advanced autonomous driving by promoting algorithm iteration, reducing hardware costs, and promoting vehicle-road cooperation.

[0003] However, current technologies for 3D reconstruction of dynamic objects struggle to achieve high-precision reconstruction from multi-frame point cloud data of dynamic objects (such as vehicles and pedestrians) due to issues like motion blur, occlusion, and sensor noise. Furthermore, storing the spatiotemporal information of dynamic objects frame-by-frame as point clouds or explicit mesh models results in massive amounts of data, which is insufficient to support the large-scale scenario requirements of autonomous driving simulation testing. Summary of the Invention

[0004] To address the aforementioned deficiencies or disadvantages, this application provides a method, apparatus, computer device, and storage medium for three-dimensional reconstruction of dynamic objects. The embodiments of this application can improve the accuracy of three-dimensional reconstruction of dynamic objects and improve the storage efficiency of spatiotemporal information of dynamic objects.

[0005] This application provides a method for three-dimensional reconstruction of dynamic objects according to a first aspect. In some embodiments, the method includes:

[0006] Extract cross-frame point cloud sequences of target objects from historical autonomous driving data;

[0007] High-density accumulated point clouds are obtained by processing cross-frame point cloud sequences through spatiotemporal alignment and motion trajectory compensation.

[0008] Rays are generated based on sensor pose data and high-density accumulated point cloud, and samples are taken along the rays. Training data is generated based on the sampling results.

[0009] A two-branch neural network is constructed, consisting of a geometric branch and a radiation field branch. The geometric branch is used to model the object surface using a signed distance field, and the radiation field branch is used to learn point cloud density and surface properties using a neural radiation field.

[0010] Iterative training of the dual-branch neural network is performed based on the training data;

[0011] Extract the network parameters of the trained dual-branch neural network to obtain the dynamic implicit representation model of the target object.

[0012] In some embodiments, a high-density accumulated point cloud is obtained by processing a cross-frame point cloud sequence through spatiotemporal alignment and motion trajectory compensation, including:

[0013] Transform each frame of the point cloud sequence across frames into a local coordinate system with the center of the target object as the origin.

[0014] Based on the motion trajectory of the target object and the timestamp of each point in each frame of the point cloud, motion compensation is performed on each point to align each point to the same timestamp, resulting in a high-density cumulative point cloud.

[0015] In some embodiments, the ray includes a first ray and a second ray; the first ray refers to a ray from the point cloud of the target object, and the second ray refers to a ray not reflected by the target object; the training data includes multiple training samples; the multiple training samples include positive training samples and negative training samples; each positive training sample includes observation view information corresponding to a first ray, spatiotemporal information of each sampling point on the first ray, and ground truth information, the ground truth information in the positive training sample includes the true depth information, true intensity information, and true color information corresponding to the first ray; the negative training sample includes observation view information corresponding to a second ray, spatiotemporal information of each sampling point on the second ray, and ground truth information, the ground truth information in the negative training sample includes the ray termination probability value; the spatiotemporal information includes three-dimensional coordinates and timestamps.

[0016] In some embodiments, the dual-branch neural network further includes an encoding layer, which encodes the spatiotemporal information of each sampling point in the training samples input to the dual-branch neural network into an intermediate feature vector; a geometric branch is used to generate a symbolic distance field prediction value for each sampling point based on the intermediate feature vector, and to generate depth prediction information for each sampling point based on the symbolic distance field prediction value; and a radiation field branch is used to generate intensity prediction information, color prediction information, and probability values ​​of rays not being reflected by objects for each sampling point based on the observation viewpoint information and the intermediate feature vector in the training samples input to the dual-branch neural network.

[0017] When training a dual-branch neural network, the L1 loss function is used to constrain the difference between the predicted depth information and the true depth information, the L1 loss function is used to constrain the difference between the predicted intensity information and the true intensity information, the L2 loss function is used to constrain the difference between the predicted color information and the true color information, and the binary cross-entropy loss function is used to constrain the difference between the probability value of a ray not being reflected by an object and the probability value of the ray termination.

[0018] In some embodiments, the operation of generating depth prediction information for each sampling point based on the symbolic distance field prediction value of each sampling point includes:

[0019] Calculate the opacity of each sampling point based on the predicted value of the symbol distance field at each sampling point;

[0020] The weight value of each sampling point is calculated based on the opacity of each sampling point.

[0021] Calculate the depth prediction information for each sampling point based on its weight value;

[0022] In some embodiments, the opacity of each sampling point is calculated based on the symbolic distance field prediction value of each sampling point, including:

[0023] The opacity of each sampling point is calculated using the following formula:

[0024]

[0025] Where, α i s represents the opacity of the i-th sampling point. i β represents the predicted symbolic distance field value of the i-th sampling point, and β represents the preset weight;

[0026] The weight value of each sampling point is calculated based on its opacity, including:

[0027] The weight value of each sampling point is calculated using the following formula:

[0028]

[0029] Where, α i α represents the opacity of the i-th sampling point. j ω represents the opacity of the j-th sampling point. i This represents the weight value of the i-th sampling point;

[0030] The depth prediction information for each sampling point is calculated based on its weight value, including:

[0031] The depth prediction information for each sampling point is calculated using the following formula:

[0032]

[0033] Where, ω i τ represents the weight value of the i-th sampling point. i N represents the depth prediction information for the i-th sampling point; r E[Dl(r)] represents the total number of sampling points on ray r; E[Dl(r)] represents the expected predicted depth of ray r; ray r represents the ray corresponding to the training sample input to the dual-branch neural network.

[0034] In some embodiments, after obtaining the dynamic implicit representation model of the target object, the method further includes:

[0035] Rendering from any perspective, calculating collision volume, or inferring dynamic deformation based on a dynamic implicit representation model.

[0036] This application provides a dynamic object three-dimensional reconstruction apparatus according to a second aspect. In some embodiments, the apparatus includes:

[0037] The point cloud sequence extraction module is used to extract cross-frame point cloud sequences of target objects from historical autonomous driving data;

[0038] The point cloud sequence processing module is used to process cross-frame point cloud sequences through spatiotemporal alignment and motion trajectory compensation to obtain high-density cumulative point clouds.

[0039] The sampling module is used to generate rays based on sensor pose data and high-density accumulated point cloud, and to sample along the rays, and generate training data based on the sampling results;

[0040] The neural network building module is used to build a two-branch neural network containing a geometric branch and a radiation field branch. The geometric branch is used to model the object surface using a signed distance field, and the radiation field branch is used to learn point cloud density and surface properties using a neural radiation field.

[0041] The neural network training module is used for iterative training of a two-branch neural network based on training data.

[0042] The network parameter extraction module is used to extract the network parameters of the trained dual-branch neural network to obtain the dynamic implicit representation model of the target object.

[0043] According to a third aspect, this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the dynamic object three-dimensional reconstruction method provided in any of the above embodiments.

[0044] According to a fourth aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the dynamic object three-dimensional reconstruction method provided in any of the above embodiments.

[0045] The embodiments described above in this application achieve efficient and high-precision reconstruction of dynamic objects by driving an implicit neural field through multi-frame point cloud fusion and combining decoupled modeling of geometric and radiation fields. Simultaneously, through end-to-end training of high-density accumulated point clouds and dual-branch neural networks, multi-frame dynamic data is compressed into a compact implicit neural field (symbolic distance field - neural radiation field), significantly reducing storage overhead. This also improves the storage efficiency of spatiotemporal information for dynamic objects. The dynamic implicit representation model established for the target object through the above embodiments supports arbitrary viewpoint rendering, collision volume calculation, and dynamic deduction capabilities, providing a high-fidelity, lightweight data foundation for autonomous driving simulation and accelerating the iterative verification of perception and planning algorithms. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating a dynamic object 3D reconstruction method provided in this application according to one or more embodiments;

[0047] Figure 2 This is a schematic diagram of a dual-branch neural network architecture provided in one or more embodiments of this application;

[0048] Figure 3 This is a structural block diagram of a dynamic object three-dimensional reconstruction device provided in one or more embodiments of this application;

[0049] Figure 4 This is an internal structural diagram of a computer device provided in one or more embodiments of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0051] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0052] In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0053] In view of the shortcomings or defects of related technologies, this application provides a method for three-dimensional reconstruction of dynamic objects, which can improve the accuracy of three-dimensional reconstruction of dynamic objects and improve the storage efficiency of spatiotemporal information of dynamic objects.

[0054] In some embodiments, the dynamic object 3D reconstruction method includes, for example, Figure 1 The steps shown are illustrated below, with an example of applying this method to computer devices (such as servers, in-vehicle computing devices, etc.).

[0055] S110: Extract cross-frame point cloud sequences of target objects from historical autonomous driving data.

[0056] The target object refers to the dynamic object to be modeled in 3D. Historical autonomous driving data consists of raw data collected by sensors on the vehicle (such as LiDAR). This raw data includes point cloud data. LiDAR is a device that uses lasers to measure distance. It calculates the distance to an object by emitting laser pulses and measuring the time it takes for these pulses to reflect back. LiDAR can emit millions of laser pulses per second, each pulse returning the position information of a point. These points combine to form a point cloud, which acts like a 3D map composed of countless tiny points, describing the shape and position of the vehicle's surroundings. The position information of a point is specifically a 3D coordinate (x, y, z), with the origin being the location of the LiDAR.

[0057] Computer devices can use offline target detection models and motion tracking models to perform object detection and tracking tasks on the raw data, obtain highly accurate 3D bounding boxes of target objects, and finally extract cross-frame point cloud sequences of target objects from the raw data based on the 3D bounding boxes.

[0058] A cross-frame point cloud sequence of a target object refers to a series of point cloud data obtained by performing multiple LiDAR scans on the same target object within consecutive time frames. Each frame of the point cloud records the three-dimensional shape and position of the target object at a specific moment. The cross-frame point cloud sequence can demonstrate the changes of the target object at different points in time.

[0059] S120: High-density cumulative point cloud is obtained by processing cross-frame point cloud sequences through spatiotemporal alignment and motion trajectory compensation.

[0060] High-density cumulative point clouds refer to point cloud data that are merged together through multiple scans or measurements to form a higher-density point cloud. Computer devices can process cross-frame point cloud sequences through spatiotemporal alignment and motion trajectory compensation to obtain high-density cumulative point clouds. In some embodiments, each frame of the cross-frame point cloud sequence can be transformed to a local coordinate system with the center of the target object as the origin to eliminate the spatial offset caused by the overall motion of the target object. Furthermore, motion compensation is performed on each point based on the motion trajectory of the target object and the timestamp of each point in each frame of the point cloud to align each point to the same timestamp, thereby eliminating motion blur and obtaining a high-density cumulative point cloud.

[0061] S130: Generate a ray based on sensor pose data and high-density accumulated point cloud, and sample along the ray to generate training data based on the sampling results.

[0062] Sensor pose data describes the state of a sensor (such as a lidar) in space and time, including the sensor's position (i.e., spatial coordinates or three-dimensional coordinates), orientation, and field of view.

[0063] Computer equipment can generate rays based on sensor pose data, using existing ray generation techniques. All rays originate from the sensor position. Each ray has corresponding observation viewpoint information, including observation coordinates (i.e., sensor position, such as the spatial coordinates of a lidar sensor) and the ray's pitch angle (vertical pitch angle, denoted by Pitch) and yaw angle (horizontal deflection angle, denoted by Yaw). Two types of rays can be identified from these rays using a high-density accumulated point cloud: first rays and second rays. First rays originate from the point cloud of the target object (high-density accumulated point cloud), i.e., rays emitted by the sensor that strike the target object's surface and reflect back a valid signal. Each first ray has corresponding true depth, true intensity, and true color information. The true intensity and true color information are obtained from the high-density accumulated point cloud, while the true depth information is calculated. Second rays are rays not reflected by the target object, including rays that strike the target object but do not reflect back a valid signal and / or rays that do not strike the target object.

[0064] The training data is used to train the two-branch neural network and includes multiple training samples. These training samples include positive training samples and negative training samples.

[0065] Each positive training sample includes observation viewpoint information corresponding to a first ray, spatiotemporal information (including 3D coordinates and timestamps) of each sampling point on that first ray, and ground truth information. The ground truth information in each positive training sample includes the true depth information, true intensity information, and true color information corresponding to the relevant first ray. The generation process of positive training samples can be as follows: sampling is performed on a first ray, the spatiotemporal information of each sampling point is recorded, and the observation viewpoint information, true depth information, true intensity information, and true color information corresponding to that first ray are obtained. The true depth information, true intensity information, and true color information corresponding to that first ray are used as ground truth information to generate positive training samples. Performing the above sampling operation on multiple first rays can yield multiple positive training samples.

[0066] Each negative training sample includes observation viewpoint information corresponding to a second ray, spatiotemporal information (including 3D coordinates and timestamps) of each sampling point on that second ray, and ground truth information. The ground truth information of the second ray includes the ray termination probability value. The generation process of negative training samples can be as follows: randomly select a second ray, sample on that second ray, record the spatiotemporal information of each sampling point, obtain the observation viewpoint information of that ray, and use the ray termination probability value of that second ray (a value of 1 indicates that the probability of the ray not being reflected by the object is 100%) as the ground truth information, thereby generating negative training samples. Performing the above sampling operation on multiple second rays can yield multiple negative training samples.

[0067] During training, positive training samples drive the dual-branch neural network to learn the surface of dynamic objects, while negative training samples optimize the ray interaction probability. Together, they ensure the robustness and accuracy of the dual-branch neural network in complex autonomous driving scenarios.

[0068] Computer equipment can avoid overfitting of the model to the object surface and improve robustness to background noise by using mixed sampling of the first and second rays.

[0069] S140: Construct a two-branch neural network that includes a geometric branch and a radiation field branch.

[0070] Please see Figure 2 , Figure 2 The diagram illustrates an exemplary network architecture of a two-branch neural network. This network comprises two branches, each corresponding to a different modeling objective. The geometry branch models the object's surface using a signed distance field (SDF), supporting accurate geometric reconstruction and collision volume calculation. The neural radiance field (NeRF) branch learns point cloud density and surface properties (such as color, intensity, and the probability that rays are not reflected by the object) using the neural radiance field, enabling realistic rendering. By separating the geometry and radiance field models, training instability caused by feature coupling is avoided, thus improving reconstruction accuracy.

[0071] S150: Iteratively train the dual-branch neural network based on the training data.

[0072] The input data for a two-branch neural network consists of the spatiotemporal information of each sampling point in the training samples. By introducing temporal information (timestamps) on top of spatial information (three-dimensional coordinates), it is possible to model the motion and deformation of an object over time.

[0073] In some embodiments, the dual-branch neural network further includes an encoding layer, which encodes the spatiotemporal information of each sampling point in the training samples input to the dual-branch neural network into an intermediate feature vector; a geometric branch is used to generate a symbolic distance field prediction value for each sampling point based on the intermediate feature vector, and to generate depth prediction information for each sampling point based on the symbolic distance field prediction value; and a radiation field branch is used to generate intensity prediction information, color prediction information, and probability values ​​of rays not being reflected by objects for each sampling point based on the observation viewpoint information and the intermediate feature vector in the training samples input to the dual-branch neural network.

[0074] The above-described operation of generating depth prediction information for each sampling point based on the symbolic distance field prediction value of each sampling point includes: calculating the opacity of each sampling point based on the symbolic distance field prediction value of each sampling point; calculating the weight value of each sampling point based on the opacity of each sampling point; and calculating the depth prediction information of each sampling point based on the weight value of each sampling point.

[0075] When calculating the opacity of each sampling point based on the symbolic distance field prediction value, the opacity of each sampling point can be calculated according to the following formula. By converting the symbolic distance field prediction value into a physically reasonable opacity, the surface of the target object can be implicitly defined:

[0076]

[0077] Where, α i s represents the opacity of the i-th sampling point. i β represents the predicted symbolic distance field value of the i-th sampling point, and β represents the preset weight.

[0078] After converting the predicted symbolic distance field values ​​of each sampling point into opacity, ray integral rendering is used to determine the value of each sampling point.

[0079] When calculating the weight value of each sampling point based on its opacity, the weight value of each sampling point can be calculated using the following formula:

[0080]

[0081] Where, α i α represents the opacity of the i-th sampling point. j ω represents the opacity of the j-th sampling point. i This represents the weight value of the i-th sampling point.

[0082] When calculating the depth prediction information of each sampling point based on its weight value, the following formula can be used:

[0083]

[0084] Where, ω i τ represents the weight value of the i-th sampling point. i N represents the depth prediction information for the i-th sampling point; r E[Dl(r)] represents the total number of sampling points on ray r; E[Dl(r)] represents the expected predicted depth of ray r; ray r represents the ray corresponding to the training sample input to the dual-branch neural network.

[0085] When training a dual-branch neural network, for the geometry branch, the difference between the predicted depth information and the true depth information can be constrained using the L1 loss function to optimize the surface localization accuracy of the symbolic distance field. For the radiation field branch, the difference between the predicted intensity information and the true intensity information can be constrained using the L1 loss function, the difference between the predicted color information and the true color information can be constrained using the L2 loss function, and the difference between the probability value of a ray not being reflected by an object and the probability value of ray termination can be constrained using the binary cross-entropy loss function. This effectively distinguishes dynamic objects from background noise and improves rendering robustness.

[0086] S160: Extract the network parameters of the trained dual-branch neural network to obtain the dynamic implicit representation model of the target object.

[0087] The above embodiments use a two-branch neural network as a specific implementation framework. By training the two-branch neural network, an SDF-NeRF model (i.e., dynamic object representation model) can be obtained. The weight parameters of the two-branch neural network constitute the SDF-NeRF model, which can implicitly represent the geometric and surface properties of dynamic objects.

[0088] The trained SDF-NeRF model can infer the future motion state of the target object (such as a vehicle turning or a pedestrian walking) by adjusting the time parameter t, and generate rendering results of consecutive frames.

[0089] The above embodiments fuse multi-frame point cloud data of the target object through spatiotemporal alignment and motion compensation, which can solve the problems of motion blur and temporal inconsistency in dynamic scenes. Furthermore, implicit modeling is achieved through SDF-NeRF, enabling the construction of high-precision models. Simultaneously, by compressing the spatiotemporal information of dynamic objects into compact neural field parameters, storage overhead can be significantly reduced. Moreover, this method achieves end-to-end differentiability from point cloud preprocessing to rendering output, supporting data-driven model self-optimization.

[0090] In some embodiments, after obtaining the dynamic implicit representation model of the target object, the method further includes: performing arbitrary viewpoint rendering, collision volume calculation, or dynamic deformation deduction based on the dynamic implicit representation model.

[0091] The trained dynamic implicit representation model possesses implicit 3D information about the target object, thus allowing it to obtain corresponding outputs by inputting rays not present in the training data, such as point cloud data from a new perspective. When point cloud data from a new perspective is needed, the sensor pose data corresponding to the new perspective is first determined. Rays are generated based on the sensor pose data corresponding to the new perspective, and samples are taken along the rays. The observation perspective information of each ray and the spatiotemporal information of each sampling point on that ray are used as input data to the dynamic implicit representation model, obtaining the model prediction results for each ray (including ray termination probability value, depth prediction value, color prediction value, and intensity prediction value). Based on the depth prediction values, color prediction values, and intensity prediction values ​​of all rays with termination probability values ​​not equal to 1, new perspective point cloud data is generated. Subsequently, the rendering result of the target object under the new perspective can be generated based on the new perspective point cloud data.

[0092] It should be noted that, regarding the various steps included in the dynamic object 3D reconstruction method provided in any of the above embodiments, unless explicitly stated herein, there is no strict order restriction on the execution of these steps; they can be executed in other orders. Furthermore, at least some of these steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is also not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0093] Based on the same inventive concept, this application also provides a dynamic object three-dimensional reconstruction device. In some embodiments, such as Figure 3 As shown, the dynamic object 3D reconstruction device includes the following modules:

[0094] The point cloud sequence extraction module 110 is used to extract cross-frame point cloud sequences of target objects from historical autonomous driving data.

[0095] The point cloud sequence processing module 120 is used to process cross-frame point cloud sequences through spatiotemporal alignment and motion trajectory compensation to obtain high-density cumulative point clouds.

[0096] The sampling module 130 is used to generate rays based on sensor pose data and high-density accumulated point cloud and to sample along the rays, and to generate training data based on the sampling results.

[0097] The neural network building module 140 is used to build a two-branch neural network containing a geometric branch and a radiation field branch. The geometric branch is used to model the object surface using a signed distance field, and the radiation field branch is used to learn point cloud density and surface properties using a neural radiation field.

[0098] The neural network training module 150 is used for iterative training of a dual-branch neural network based on training data;

[0099] The network parameter extraction module 160 is used to extract the network parameters of the trained dual-branch neural network to obtain the dynamic implicit representation model of the target object.

[0100] In some embodiments, the point cloud sequence processing module 120 includes:

[0101] The spatiotemporal transformation submodule is used to transform each frame of the point cloud in a cross-frame point cloud sequence to a local coordinate system with the center of the target object as the origin.

[0102] The motion compensation submodule is used to perform motion compensation on each point based on the motion trajectory of the target object and the timestamp of each point in each frame of the point cloud, so as to align each point to the same timestamp and obtain a high-density cumulative point cloud.

[0103] In some embodiments, the ray includes a first ray and a second ray; the first ray refers to a ray from the point cloud of the target object, and the second ray refers to a ray not reflected by the target object; the training data includes multiple training samples; the multiple training samples include positive training samples and negative training samples; each positive training sample includes observation view information corresponding to a first ray, spatiotemporal information of each sampling point on the first ray, and ground truth information, the ground truth information in the positive training sample includes the true depth information, true intensity information, and true color information corresponding to the first ray; the negative training sample includes observation view information corresponding to a second ray, spatiotemporal information of each sampling point on the second ray, and ground truth information, the ground truth information in the negative training sample includes the ray termination probability value; the spatiotemporal information includes three-dimensional coordinates and timestamps.

[0104] In some embodiments, the dual-branch neural network further includes an encoding layer, which encodes the spatiotemporal information of each sampling point in the training samples input to the dual-branch neural network into an intermediate feature vector; a geometric branch is used to generate a symbolic distance field prediction value for each sampling point based on the intermediate feature vector, and to generate depth prediction information for each sampling point based on the symbolic distance field prediction value; and a radiation field branch is used to generate intensity prediction information, color prediction information, and probability values ​​of rays not being reflected by objects for each sampling point based on the observation viewpoint information and the intermediate feature vector in the training samples input to the dual-branch neural network.

[0105] When training a dual-branch neural network, the neural network training module 150 constrains the difference between the depth prediction information and the true depth information through the L1 loss function, constrains the difference between the intensity prediction information and the true intensity information through the L1 loss function, constrains the difference between the color prediction information and the true color information through the L2 loss function, and constrains the difference between the probability value of a ray not being reflected by an object and the probability value of the ray termination through the binary cross-entropy loss function.

[0106] In some embodiments, the operation of generating depth prediction information for each sampling point based on the symbolic distance field prediction value of each sampling point includes: calculating the opacity of each sampling point based on the symbolic distance field prediction value of each sampling point; calculating the weight value of each sampling point based on the opacity of each sampling point; and calculating the depth prediction information of each sampling point based on the weight value of each sampling point.

[0107] In some embodiments, the operation of calculating the opacity of each sampling point based on the symbolic distance field prediction value of each sampling point includes:

[0108] The opacity of each sampling point is calculated using the following formula:

[0109]

[0110] Where, α i s represents the opacity of the i-th sampling point. i β represents the predicted symbolic distance field value of the i-th sampling point, and β represents the preset weight;

[0111] The operation of calculating the weight value of each sampling point based on the opacity of each sampling point includes:

[0112] The weight value of each sampling point is calculated using the following formula:

[0113]

[0114] Where, α i α represents the opacity of the i-th sampling point. j ω represents the opacity of the j-th sampling point. i This represents the weight value of the i-th sampling point;

[0115] The operation of calculating the depth prediction information for each sampling point based on its weight value includes:

[0116] The depth prediction information for each sampling point is calculated using the following formula:

[0117]

[0118] Where, ω i τ represents the weight value of the i-th sampling point. iN represents the depth prediction information for the i-th sampling point; r E[Dl(r)] represents the total number of sampling points on ray r; E[Dl(r)] represents the expected predicted depth of ray r; ray r represents the ray corresponding to the training sample input to the dual-branch neural network.

[0119] In some embodiments, the apparatus further includes:

[0120] The rendering module is used for rendering from any perspective based on a dynamic implicit representation model.

[0121] The collision volume calculation module is used to calculate the collision volume based on the dynamic implicit representation model.

[0122] The dynamic deformation inference module is used to perform dynamic deformation inference based on the dynamic implicit representation model.

[0123] Specific limitations regarding the dynamic object 3D reconstruction device can be found in the limitations of the dynamic object 3D reconstruction method described above, and will not be repeated here. Each module in the aforementioned dynamic object 3D reconstruction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0124] This application also provides a computer device. In some embodiments, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the steps of the dynamic object three-dimensional reconstruction method provided in any of the above embodiments.

[0125] In some embodiments, the internal structure diagram of a computer device may be as follows: Figure 4 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data related to 3D reconstruction, such as historical autonomous driving data and sensor pose data; the specific data stored can also be found in the limitations described in the above method embodiments. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a dynamic object 3D reconstruction method.

[0126] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0127] This application also provides a computer-readable storage medium, in some embodiments of which a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the steps of the dynamic object three-dimensional reconstruction method provided in any of the above embodiments.

[0128] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0129] Those skilled in the art will understand that implementing all or part of the processes in the above method embodiments can be accomplished by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink, DRAM (SLDRAM), RAMbus, direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0130] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for three-dimensional reconstruction of dynamic objects, characterized in that, The method includes: Extract cross-frame point cloud sequences of target objects from historical autonomous driving data; The cross-frame point cloud sequence is processed by spatiotemporal alignment and motion trajectory compensation to obtain a high-density cumulative point cloud; A ray is generated based on the sensor pose data and the high-density accumulated point cloud, and samples are taken along the ray. Training data is generated based on the sampling results. A two-branch neural network is constructed, comprising a geometric branch and a radiation field branch. The geometric branch is used to model the object surface using a signed distance field, and the radiation field branch is used to learn point cloud density and surface properties using a neural radiation field. The dual-branch neural network is iteratively trained based on the training data; Extract the network parameters of the trained dual-branch neural network to obtain the dynamic implicit representation model of the target object.

2. The method as described in claim 1, characterized in that, The cross-frame point cloud sequence is processed by spatiotemporal alignment and motion trajectory compensation to obtain a high-density accumulated point cloud, including: Transform each frame point cloud in the cross-frame point cloud sequence to a local coordinate system with the center of the target object as the origin; Based on the motion trajectory of the target object and the timestamp of each point in each frame of the point cloud, motion compensation is performed on each point to align each point to the same timestamp, resulting in a high-density cumulative point cloud.

3. The method as described in claim 1, characterized in that, The ray includes a first ray and a second ray; the first ray refers to a ray from the point cloud of the target object, and the second ray refers to a ray not reflected by the target object; the training data includes multiple training samples; the multiple training samples include positive training samples and negative training samples; each positive training sample includes observation view information corresponding to a first ray, spatiotemporal information of each sampling point on the first ray, and ground truth information, and the ground truth information in the positive training sample includes the true depth information, true intensity information, and true color information corresponding to the first ray; The negative training samples include observation viewpoint information corresponding to a second ray, spatiotemporal information of each sampling point on the second ray, and ground truth information. The ground truth information in the negative training samples includes the ray termination probability value; the spatiotemporal information includes three-dimensional coordinates and timestamps.

4. The method as described in claim 3, characterized in that, The dual-branch neural network further includes an encoding layer, which is used to encode the spatiotemporal information of each sampling point in the training samples input to the dual-branch neural network into an intermediate feature vector; the geometric branch is used to generate the symbolic distance field prediction value of each sampling point based on the intermediate feature vector, and to generate the depth prediction information of each sampling point based on the symbolic distance field prediction value of each sampling point. The radiation field branch is used to generate intensity prediction information, color prediction information, and probability values ​​of rays not being reflected by objects for each sampling point based on the observation perspective information in the training samples input to the dual-branch neural network and the intermediate feature vector. When training the dual-branch neural network, the difference between the depth prediction information and the true depth information is constrained by the L1 loss function, the difference between the intensity prediction information and the true intensity information is constrained by the L1 loss function, the difference between the color prediction information and the true color information is constrained by the L2 loss function, and the difference between the probability value of a ray not being reflected by an object and the probability value of ray termination is constrained by the binary cross-entropy loss function.

5. The method as described in claim 4, characterized in that, The operation of generating depth prediction information for each sampling point based on the symbolic distance field prediction value of each sampling point includes: Calculate the opacity of each sampling point based on the predicted value of the symbol distance field at each sampling point; The weight value of each sampling point is calculated based on the opacity of each sampling point. The depth prediction information for each sampling point is calculated based on the weight value of each sampling point.

6. The method as described in claim 5, characterized in that, The opacity of each sampling point is calculated based on the predicted symbol distance field value of each sampling point, including: The opacity of each sampling point is calculated using the following formula: Where, α i s represents the opacity of the i-th sampling point. i β represents the predicted symbolic distance field value of the i-th sampling point, and β represents the preset weight; The weight value of each sampling point is calculated based on its opacity, including: The weight value of each sampling point is calculated using the following formula: Where, α i α represents the opacity of the i-th sampling point. j ω represents the opacity of the j-th sampling point. i This represents the weight value of the i-th sampling point; The depth prediction information for each sampling point is calculated based on its weight value, including: The depth prediction information for each sampling point is calculated using the following formula: Where, ω i τ represents the weight value of the i-th sampling point. i N represents the depth prediction information for the i-th sampling point; r E[Dl(r)] represents the total number of sampling points on ray r; E[Dl(r)] represents the expected predicted depth of ray r; ray r represents the ray corresponding to the training sample input into the dual-branch neural network.

7. The method as described in claim 1, characterized in that, After obtaining the dynamic implicit representation model of the target object, the method further includes: Based on the aforementioned dynamic implicit representation model, arbitrary viewpoint rendering, collision volume calculation, or dynamic deformation deduction can be performed.

8. A method for three-dimensional reconstruction of dynamic objects, characterized in that, The device includes: The point cloud sequence extraction module is used to extract cross-frame point cloud sequences of target objects from historical autonomous driving data; The point cloud sequence processing module is used to process the cross-frame point cloud sequence through spatiotemporal alignment and motion trajectory compensation to obtain a high-density cumulative point cloud; The sampling module is used to generate a ray based on the sensor pose data and the high-density accumulated point cloud, and to sample along the ray, and to generate training data based on the sampling results; A neural network construction module is used to construct a two-branch neural network containing a geometric branch and a radiation field branch. The geometric branch is used to model the object surface using a signed distance field, and the radiation field branch is used to learn point cloud density and surface properties using a neural radiation field. A neural network training module is used to iteratively train the dual-branch neural network based on the training data; The network parameter extraction module is used to extract the network parameters of the trained dual-branch neural network to obtain the dynamic implicit representation model of the target object.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images