Dynamic scene reconstruction method and system based on separation of dynamic object and static scene

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a dynamic scene reconstruction method and system based on separation of a dynamic object and a static scene, and the method comprises the steps: firstly, decoupling the dynamic scene into the dynamic object and the static scene through multi-modal perception data; in a dynamic object reconstruction stage, through constructing local object representation and global object representation, object motion parameters are explicitly modeled as differentiable optimization variables, and high-precision modeling of a dynamic object on a continuous space-time dimension is realized; aiming at the information limitation of single-frame observation data, through cross-frame time sequence correlation modeling, the observation data of a dynamic object in multiple frames are fused, and the three-dimensional reconstruction precision is effectively enhanced; and finally, realizing fusion of the dynamic object and the static background by using the interpolation pose to obtain a complete dynamic scene with time-space consistency.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a dynamic scene reconstruction method and system based on the separation of dynamic objects and static scenes. Background Art

[0002] The planning module of the current autonomous driving system heavily relies on manual rule design and is difficult to cover complex and changeable road scenes. Although the data-driven end-to-end planning model shows potential, its training depends on large-scale and highly generalized multi-modal data. Therefore, by constructing a digitally processable three-dimensional model, it is used to support new perspective generation, scene understanding, and interaction in the field of autonomous driving.

[0003] However, for temporal scene modeling, existing methods have many limitations. Traditional 3D reconstruction techniques focus on static scene modeling. However, they assume that the scene is static and cannot handle dynamic objects (such as moving vehicles), resulting in ghosting or missing of dynamic objects in the reconstruction results. In existing dynamic modeling solutions, most methods decouple the continuous motion process into discrete frame-by-frame independent processing. This paradigm breaks the temporal correlation and is difficult to ensure the motion consistency of dynamic objects in the time dimension. Eventually, the reconstruction results may show non-physically realistic broken motion trajectories or sudden surface deformation mutations. Summary of the Invention

[0004] The present invention provides a dynamic scene reconstruction method and system based on the separation of dynamic objects and static scenes.

[0005] The technical solution of the present invention is as follows:

[0006] The present invention provides a dynamic scene reconstruction method based on the separation of dynamic objects and static scenes, including:

[0007] S1: Collect continuous frame images and multi-view images of corresponding frames, and select one frame from the continuous frame images as a reference frame. Use the reference frame to perform instance segmentation on the continuous frame multi-view images to obtain a multi-view image sequence of the target object and the corresponding target object mask. The reference frame target object mask is processed by the visual hull algorithm to obtain the reference frame object point cloud data;

[0008] S2: Based on the multi-view image sequence of the target object, the reference frame object point cloud data, and the camera parameters, optimize the appearance parameters of the current frame by fixing the relative pose parameters;

[0009] Fix the optimized appearance parameters of the current frame, and optimize the relative pose parameters by calculating the loss between the multi-view images of the adjacent frames of the current frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the action of the relative pose parameters on the optimized appearance parameters of the current frame;

[0010] The optimized relative pose parameters are dot - multiplied with the absolute pose parameters to obtain updated absolute pose parameters, and the adjacent frames of the current frame are used as optimized frames; return and execute S2 until all frames are processed to obtain an optimized frame set;

[0011] Based on the optimized frame set, by calculating the loss between the multi - view images of each optimized frame and the corresponding rendered images obtained by three - dimensional Gaussian splashing under the global three - dimensional appearance parameters and absolute pose parameters of the object, a three - dimensional dynamic object representation is obtained;

[0012] S3: Based on the three - dimensional dynamic object representation, using the rotation matrix and translation vector of adjacent frames, an interpolation pose is calculated. After the interpolation pose is dot - multiplied with the global three - dimensional appearance parameters of the object, it is superimposed with the static scene to generate a reconstructed dynamic scene.

[0013] The operation of S2 is as follows:

[0014] S21: Use the appearance representation reconstructed from the current frame data as the current frame appearance parameters, and the relative change from the current frame to the adjacent frame as the relative pose parameters;

[0015] Use the appearance representation reconstructed by fusing multi - frame data as the global three - dimensional appearance parameters of the object, and the change from the reference frame to the current frame as the absolute pose parameters;

[0016] S22: Initialize the current frame appearance parameters according to the object point cloud data of the reference frame and the absolute pose parameters;

[0017] S23: Fix the relative pose parameters, and optimize the current frame appearance parameters by calculating the loss between the multi - view images of the current frame and the corresponding rendered images obtained by three - dimensional Gaussian splashing under the initialized current frame appearance parameters;

[0018] S24: Fix the optimized current frame appearance parameters, and optimize the relative pose parameters by calculating the loss between the multi - view images of the adjacent frames of the current frame and the corresponding rendered images obtained by three - dimensional Gaussian splashing under the relative pose parameters acting on the optimized current frame appearance parameters;

[0019] S25: The optimized relative pose parameters are dot - multiplied with the absolute pose parameters to obtain updated absolute pose parameters, and the adjacent frames of the current frame are used as optimized frames; execute S21 until all frames are processed to obtain an optimized frame set;

[0020] S26: Based on the optimized frame set, by calculating the loss between the multi - view images of each optimized frame and the corresponding rendered images obtained by three - dimensional Gaussian splashing under the global three - dimensional appearance parameters and absolute pose parameters of the object, a three - dimensional dynamic object representation is obtained.

[0021] S2 also includes representing the three - dimensional appearance and geometric structure of the dynamic object using three - dimensional Gaussian splashing, by the formula: Implementation; where T is the pose parameter, T = {R, t}, R is the rotation matrix, and t is the translation vector; is the appearance parameter; P is the total number of Gaussian ellipsoids; p is the Gaussian ellipsoid serial number; represents the mean of the Gaussian distribution; ||q p || = 1 represents the rotation parameter in quaternion form; where matrix_to_quaternion(R) represents the conversion from the rotation matrix to quaternion; q T represents the quaternion corresponding to the rotation matrix R; represents the multiplication of quaternions; represents the anisotropic scaling factor; represents the transparency of each Gaussian ellipsoid; sh p represents the spherical harmonic coefficient.

[0022] For the said S2, by calculating the loss between the multi-view images of the current frame and adjacent frames and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the action of the relative pose parameter on the optimized current frame appearance parameter, the relative pose parameter is optimized, which is achieved through the formula: Implementation;

[0023] In the formula, is the relative pose parameter during the optimization process; is the relative pose parameter; is the current frame appearance parameter;

[0024] is the corresponding rendered image obtained by three-dimensional Gaussian splashing under the action of the relative pose parameter on the optimized current frame appearance parameter; is the image of the adjacent frame of the current frame at view angle k, is the mask of the object of the adjacent frame of the current frame at view angle k.

[0025] For the said S2, based on the optimized frame set, by calculating the loss between the multi-view images of each optimized frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the object's global three-dimensional appearance parameter and absolute pose parameter, a three-dimensional dynamic object representation is obtained, which is achieved through the formula: Implementation;

[0026] In the formula, L global is the loss value; s is the sampled frame in the optimized frame set S; T s is the absolute pose parameter; is the object's global three-dimensional appearance parameter; is the corresponding rendered image obtained by three-dimensional Gaussian splashing under the object's global three-dimensional appearance parameter and absolute pose parameter; is the image of the sampled frame s at view angle k, The mask of the object in sampling frame s at viewing angle k.

[0027] In step S3, based on the three-dimensional dynamic object representation, using the rotation matrices and translation vectors of adjacent frames, calculate the interpolated pose, specifically:

[0028] Based on the three-dimensional dynamic object representation, using the rotation matrix of adjacent frames, calculate the rotational interpolation through spherical linear interpolation of quaternions;

[0029] Based on the translation vectors of adjacent frames, perform linear weighting to calculate the translational interpolation;

[0030] According to the formula: T(λ) = {R(λ), t(λ)}, obtain the interpolated pose;

[0031] In the formula, T(λ), R(λ), and t(λ) are the interpolated pose, rotational interpolation, and translational interpolation respectively, and λ is the interpolation parameter.

[0032] In step S3, after multiplying the interpolated pose by the global three-dimensional appearance parameters of the object, superimpose it with the static scene to generate the reconstructed dynamic scene, specifically:

[0033] According to the formula: Generate the reconstructed dynamic scene;

[0034] In the formula, are the appearance representations of the reconstructed dynamic scene, static scene, and the product of the interpolated pose and the global three-dimensional appearance parameters of the object respectively;

[0035] The static scene is obtained by reconstructing the image collected without dynamic objects.

[0036] In step S22, initialize the appearance parameters of the current frame according to the point cloud data and absolute pose parameters of the reference frame object, specifically:

[0037] According to the formula: P i = T i ·P init to generate the point cloud of the current frame, and use the point cloud of the current frame to initialize the appearance parameters of the current frame;

[0038] In the formula, P i is the point cloud of the current frame, T i is the absolute pose parameter, and P init is the point cloud data of the reference frame object.

[0039] In step S1, use the reference frame to perform instance segmentation on the multi-view images of consecutive frames to obtain the multi-view image sequence of the target object and the corresponding target object masks. The reference frame target object mask is processed by the visual hull algorithm to obtain the point cloud data of the reference frame object, specifically:

[0040] After the multi-view images of the reference frame are subjected to object detection, instance segmentation is performed on the multi-view images of consecutive frames to generate a multi-view image sequence of the target object and the corresponding target object mask;

[0041] The target object mask of the multi-view image of the reference frame is back-projected into the three-dimensional space to obtain a contour cone. The intersection of the contour cones of each view is taken as the visible hull of the target object, and the visible hull is voxelized and sampled to obtain the reference frame object point cloud data.

[0042] The present invention also provides a dynamic scene reconstruction system based on the separation of dynamic objects and static scenes, including:

[0043] Separation module: used to collect consecutive frame images and the multi-view images of the corresponding frames, select one frame from the consecutive frame images as the reference frame, perform instance segmentation on the multi-view images of consecutive frames using the reference frame to obtain a multi-view image sequence of the target object and the corresponding target object mask, and the reference frame target object mask is processed by the visible hull algorithm to obtain the reference frame object point cloud data;

[0044] Three-dimensional dynamic object generation module: based on the multi-view image sequence of the target object, the reference frame object point cloud data and the camera parameters, optimize the appearance parameters of the current frame by fixing the relative pose parameters;

[0045] Fix the optimized appearance parameters of the current frame, and optimize the relative pose parameters by calculating the loss between the multi-view images of the adjacent frames of the current frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the action of the relative pose parameters on the optimized appearance parameters of the current frame;

[0046] Multiply the optimized relative pose parameters by the absolute pose parameters to obtain the updated absolute pose parameters, and use the adjacent frames of the current frame as the optimized frames; return to enter the three-dimensional dynamic object generation module until all frames are processed to obtain a set of optimized frames;

[0047] Based on the set of optimized frames, calculate the loss between the multi-view images of each optimized frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the object global three-dimensional appearance parameters and the absolute pose parameters to obtain the three-dimensional dynamic object representation;

[0048] Dynamic scene reconstruction module: based on the three-dimensional dynamic object representation, use the rotation matrix and translation vector of adjacent frames to calculate the interpolation pose. After multiplying the interpolation pose by the object global three-dimensional appearance parameters, superimpose it with the static scene to generate the reconstructed dynamic scene.

[0049] Beneficial effects

[0050] The present invention realizes dynamic scene reconstruction by separating dynamic objects from static scenes based on three-dimensional Gaussian splashing. First, multi-modal perception data is used to decouple the dynamic scene into dynamic objects and static scenes. In the dynamic object reconstruction stage, by constructing local object representations and global object representations, the object motion parameters are explicitly modeled as differentiable optimization variables to achieve high-precision modeling of dynamic objects in continuous spatio-temporal dimensions. Aiming at the information limitation of single-frame observation data, through cross-frame temporal correlation modeling, the observation data of dynamic objects in multiple frames is fused to effectively enhance the three-dimensional reconstruction accuracy. Finally, the dynamic objects and the static background are fused using interpolated poses to finally obtain a complete dynamic scene with spatio-temporal consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic flowchart of step S1 of the present invention.

[0052] Figure 2 It is a schematic flowchart of step S2 of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0053] The following embodiments are intended to illustrate the present invention rather than further limit the present invention.

[0054] The present invention provides a dynamic scene reconstruction method based on the separation of dynamic objects and static scenes, including the following steps:

[0055] S1: Collect continuous frame images and multi-view images of corresponding frames, select one frame from the continuous frame images as a reference frame, use the reference frame to perform instance segmentation on the continuous frame multi-view images to obtain a multi-view image sequence of the target object and the corresponding target object mask, and process the reference frame target object mask through a visual hull algorithm to obtain reference frame object point cloud data.

[0056] This stage preprocesses the raw data collected by the sensor (multiple cameras) and decouples the dynamic objects and static background in the raw data. As Figure 1 shown, the specific processing steps are as follows:

[0057] Step 1: Collect continuous frame images and multi-view images of corresponding frames, and select a reference frame.

[0058] In order to be able to associate and determine the target object in different frames and different perspectives, and then continue with the following operations: Based on an object detection method (such as YOLO), perform real-time object detection on the multi-view images of the reference frame to obtain a set of two-dimensional bounding boxes of the target object in each perspective.

[0059] Step 2: Based on the camera calibration parameters, perform three-dimensional back-projection calculations on the center points of each two-dimensional bounding box to generate back-projection rays pointing from the optical centers of each camera to the target space center, and all the back-projection rays form a three-dimensional ray cluster.

[0060] Step 3: Calculate the spatial distance of the back-projected rays from different perspectives, determine the same target object observed from multiple perspectives according to a preset threshold, and obtain the target bounding boxes of the target object in each perspective, thereby realizing the dynamic target instance-level matching across cameras.

[0061] Step 4: According to the target bounding boxes of the target object in each perspective, perform instance segmentation on the multi-perspective images of consecutive frames (such as processing through the SegmentAnything Model 2, abbreviated as SAM2), and generate a multi-perspective image sequence of the target object and the corresponding target object masks.

[0062] Step 5: Back-project the target object masks of the reference frame multi-perspective images into the three-dimensional space to obtain contour cones, take the intersection of the contour cones of each perspective as the visual hull of the target object, and perform voxelization sampling on the visual hull to obtain the dynamic object point cloud, that is, the reference frame object point cloud data.

[0063] S2: Based on the multi-perspective image sequence of the target object, the reference frame object point cloud data, and the camera parameters, perform three-dimensional Gaussian splashing (3DGS) to reconstruct the three-dimensional dynamic object.

[0064] In the process of reconstructing the dynamic object in this stage, the multi-perspective image sequence obtained by S1 The corresponding object masks And the reference frame object point cloud data P init , combined with the internal parameter matrices K of multiple cameras j , the external parameter matrix where j represents the camera serial number and i represents the frame serial number. Use this data for dynamic object reconstruction to achieve geometric modeling, appearance modeling, and motion trajectory capture of the object.

[0065] The operations are as follows:

[0066] S21: Use the appearance representation reconstructed from the current frame data as the current frame appearance parameters The relative change from the current frame to the adjacent frame is the relative pose parameter Thus, construct a local object representation To accurately estimate the relative motion of the dynamic object between adjacent frames.

[0067] Use the appearance representation reconstructed by fusing multi-frame data as the object global three-dimensional appearance parameters The change from the reference frame to the current frame is the absolute pose parameter T i ; Thus, construct a global object representation To achieve cross-frame three-dimensional information fusion.

[0068] The S21 also includes representing the three-dimensional appearance and geometric structure of dynamic objects using three-dimensional Gaussian splashes, which is achieved through the formula: Implementation;

[0069] In the formula, T is the pose parameter, T = {R, t}, R is the rotation matrix, and t is the translation vector; Is the appearance parameter; P is the total number of Gaussian ellipsoids; p is the Gaussian ellipsoid serial number; Represents the mean of the Gaussian distribution; ||q p || = 1 represents the rotation parameter in quaternion form; where matrix_to_quaternion represents the conversion from the rotation matrix to quaternion; q T Represents the quaternion corresponding to the rotation matrix R; Represents the multiplication of quaternions; Represents the anisotropic scaling factor; Represents the transparency of each Gaussian ellipsoid; sh p Represents the spherical harmonic coefficient.

[0070] After completing S21, a progressive optimization strategy starting from the reference frame as the initial frame is adopted to achieve motion estimation and global model optimization through frame-by-frame iteration. For the i-th frame, first optimize the local 3DGS, obtain the relative motion between the current local 3DGS and the local 3DGS of the adjacent frame, and update the absolute motion of the global 3DGS. Finally, optimize the global 3DGS to achieve cross-frame information fusion of the global 3DGS. See Figure 2 Specifically as follows:

[0071] S22: Initialize the appearance parameters of the current frame according to the object point cloud data and absolute pose parameters of the reference frame. Specifically:

[0072] According to the formula: P i = T i ·P init , generate the point cloud of the current frame, and initialize the appearance parameters of the current frame using the point cloud of the current frame

[0073] In the formula, P i Is the point cloud of the current frame, T i Is the absolute pose parameter, and P init Is the object point cloud data of the reference frame.

[0074] S23: Fix the relative pose parameters, and optimize the appearance parameters of the current frame by calculating the loss between the multi-view images of the current frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the initialized appearance parameters of the current frame.

[0075] It can be achieved through the formula: Implementation;

[0076] In the formula, is the current frame appearance parameter during the optimization process; is the current frame appearance parameter; is the corresponding rendered image obtained by three-dimensional Gaussian splashing under the current frame appearance parameter in the initialization; is the image of the current frame at view angle k, is the mask of the object in the current frame at view angle k.

[0077] S24: Fix the optimized current frame appearance parameter, and optimize the relative pose parameter by calculating the loss between the multi-view images of the adjacent frames of the current frame and the corresponding rendered image obtained by three-dimensional Gaussian splashing under the action of the relative pose parameter on the optimized current frame appearance parameter.

[0078] This optimization process can obtain accurate inter-frame relative motion parameters, which is convenient for inter-frame relative motion estimation, and is realized by the formula: implemented;

[0079] In the formula, is the relative pose parameter during the optimization process; is the relative pose parameter; is the current frame appearance parameter; is the corresponding rendered image obtained by three-dimensional Gaussian splashing under the action of the relative pose parameter on the optimized current frame appearance parameter; is the image of the adjacent frame of the current frame at view angle k, is the mask of the object of the adjacent frame of the current frame at view angle k.

[0080] S25: Multiply the optimized relative pose parameter by the absolute pose parameter to obtain the updated absolute pose parameter, and take the adjacent frame of the current frame as the optimized frame; return to execute S21 until all frames are processed to obtain the optimized frame set.

[0081] S26: Based on the optimized frame set, obtain the three-dimensional dynamic object representation by calculating the loss between the multi-view images of each optimized frame and the corresponding rendered image obtained by three-dimensional Gaussian splashing under the object global three-dimensional appearance parameter and the absolute pose parameter.

[0082] This optimization process improves the accuracy of the global 3DGS, and is realized by the formula:

[0083] implemented;

[0084] In the formula, L global is the loss value; s is the sampled frame in the optimized frame set S; T s is the absolute pose parameter; is the object global three-dimensional appearance parameter; is the corresponding rendered image obtained by three-dimensional Gaussian splashing under the global three-dimensional appearance parameters and absolute pose parameters of the object; is the image of the sampling frame s at the viewing angle k, is the mask of the object in the sampling frame s at the viewing angle k.

[0085] In the dynamic object reconstruction stage of the present invention, by constructing local object representation and global object representation, the object motion parameters are explicitly modeled as differentiable optimization variables, realizing high-precision modeling of dynamic objects in continuous spatio-temporal dimensions; aiming at the information limitation of single-frame observation data, through cross-frame temporal correlation modeling, the observation data of dynamic objects in multiple frames are fused to effectively enhance the 3D reconstruction accuracy.

[0086] S3: Based on the rotation matrix and translation vector of adjacent frames, calculate the interpolated pose. After multiplying the interpolated pose by the global three-dimensional appearance parameters of the object, it is superimposed with the static scene to generate the reconstructed dynamic scene.

[0087] When fusing the dynamic object and the static scene in this stage, the three-dimensional dynamic object representation generated by S2 is utilized which includes the global three-dimensional appearance representation of the object and the frame-by-frame motion representation To obtain the continuous motion of the object, the object motion is interpolated between adjacent frames to realize the rendering of the dynamic scene. Specifically as follows:

[0088] First, in the process of fusing the dynamic object, at a given time t, let the adjacent frames be T a ={R a , t a} and T b ={R b , t b}, where R and t are the rotation matrix and translation vector respectively, and the interpolation parameter λ=(t - t a ) / (t b - t a ), t ∈ [t a , t b . The interpolated pose T(λ) is generated through the following steps:

[0089] Based on the rotation matrix of adjacent frames, use spherical linear interpolation (Slerp) of quaternions to smoothly transition the rotation state and calculate the rotation interpolation R(λ); the formula is as follows:

[0090]

[0091] where, q a = matrix_to_quaternion(R a ), representing the rotation matrix Ra The corresponding quaternion; q b = matrix_to_quaternion(R b ), representing the rotation matrix R b The corresponding quaternion; quaternion_to_matrix represents the conversion from quaternion to rotation matrix; θ = arccos(q a ·q b ), representing the angle between two quaternions.

[0092] Based on the translation vectors of adjacent frames, perform linear weighting to calculate the translation interpolation t(λ); the formula is as follows:

[0093] T(λ) = (1 - λ)t a + λt b ;

[0094] According to the formula: T(λ) = {R(λ), t(λ)}, obtain the interpolated pose;

[0095] In the formula, T(λ), R(λ), and t(λ) are the interpolated pose, rotation interpolation, and translation interpolation respectively, and λ is the interpolation parameter.

[0096] Finally, after the interpolated pose is dot-multiplied with the global three-dimensional appearance parameters of the object and superimposed with the static scene, a reconstructed dynamic scene is generated. Specifically:

[0097] According to the formula: Generate the reconstructed dynamic scene;

[0098] In the formula, are the appearance representations of the reconstructed dynamic scene, static scene, and the result of dot-multiplying the interpolated pose with the global three-dimensional appearance parameters of the object respectively;

[0099] The static scene is obtained by reconstructing (such as 3DGS) the image collected without dynamic objects.

[0100] The present invention realizes the reconstruction of a dynamic scene with the separation of dynamic objects and static scenes based on three-dimensional Gaussian splashing. First, the dynamic scene is decoupled into dynamic objects and static scenes by using multi-modal perception data; in the dynamic object reconstruction stage, by constructing local object representations and global object representations, the object motion parameters are explicitly modeled as differentiable optimization variables to achieve high-precision modeling of dynamic objects in continuous spatio-temporal dimensions; aiming at the information limitation of single-frame observation data, through cross-frame temporal correlation modeling, the observation data of dynamic objects in multiple frames are fused to effectively enhance the three-dimensional reconstruction accuracy; finally, the interpolated pose is used to realize the fusion of dynamic objects and static backgrounds to obtain a complete dynamic scene with spatio-temporal consistency.

[0101] The present invention also provides a dynamic scene reconstruction system based on the separation of dynamic objects and static scenes, including:

[0102] Separation module: used to collect continuous frame images and multi-view images of corresponding frames, select one frame from the continuous frame images as the reference frame, perform instance segmentation on the continuous frame multi-view images using the reference frame to obtain a multi-view image sequence of the target object and the corresponding target object mask, and process the reference frame target object mask through the visual hull algorithm to obtain the reference frame object point cloud data;

[0103] Three-dimensional dynamic object generation module: Based on the multi-view image sequence of the target object, the reference frame object point cloud data, and the camera parameters, optimize the appearance parameters of the current frame by fixing the relative pose parameters;

[0104] Fix the optimized appearance parameters of the current frame, and optimize the relative pose parameters by calculating the loss between the multi-view images of the adjacent frames of the current frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the action of the relative pose parameters on the optimized appearance parameters of the current frame;

[0105] Multiply the optimized relative pose parameters by the absolute pose parameters to obtain the updated absolute pose parameters, and use the adjacent frames of the current frame as the optimized frames; return to enter the three-dimensional dynamic object generation module until all frames are processed to obtain an optimized frame set;

[0106] Based on the optimized frame set, obtain the three-dimensional dynamic object representation by calculating the loss between the multi-view images of each optimized frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the object global three-dimensional appearance parameters and absolute pose parameters;

[0107] Dynamic scene reconstruction module: Based on the three-dimensional dynamic object representation, use the rotation matrix and translation vector of adjacent frames to calculate the interpolation pose. After multiplying the interpolation pose by the object global three-dimensional appearance parameters, superimpose it with the static scene to generate the reconstructed dynamic scene.

Claims

1. A dynamic scene reconstruction method based on the separation of dynamic objects and static scenes, characterized in that Including: S1: Collect consecutive frame images and multi-view images of corresponding frames, select one frame from the consecutive frame images as the reference frame, perform instance segmentation on the consecutive frame multi-view images using the reference frame to obtain a multi-view image sequence of the target object and the corresponding target object mask, and process the reference frame target object mask through the visual hull algorithm to obtain the reference frame object point cloud data; S2: Based on the multi-view image sequence of the target object, the reference frame object point cloud data, and the camera parameters, optimize the appearance parameters of the current frame by fixing the relative pose parameters; Fix the optimized appearance parameters of the current frame, and optimize the relative pose parameters by calculating the loss between the multi-view images of the adjacent frames of the current frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the action of the relative pose parameters on the optimized appearance parameters of the current frame; Multiply the optimized relative pose parameters by the absolute pose parameters to obtain the updated absolute pose parameters, and use the adjacent frames of the current frame as the optimized frames; Return to execute S2 until all frames are processed to obtain an optimized frame set; Based on the optimized frame set, obtain the three-dimensional dynamic object representation by calculating the loss between the multi-view images of each optimized frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the object global three-dimensional appearance parameters and the absolute pose parameters; S3: Based on the three-dimensional dynamic object representation, use the rotation matrix and translation vector of adjacent frames to calculate the interpolation pose. After multiplying the interpolation pose by the object global three-dimensional appearance parameters, superimpose it with the static scene to generate the reconstructed dynamic scene.

2. The dynamic scene reconstruction method based on separation of dynamic objects and static scenes according to claim 1, wherein, The operation of S2 is as follows: S21: Use the appearance representation reconstructed from the current frame data as the appearance parameters of the current frame, and the relative change from the current frame to the adjacent frame as the relative pose parameters; Use the appearance representation reconstructed by fusing multi-frame data as the object global three-dimensional appearance parameters, and the change from the reference frame to the current frame as the absolute pose parameters; S22: Initialize the appearance parameters of the current frame according to the reference frame object point cloud data and the absolute pose parameters; S23: Fix the relative pose parameters, and optimize the appearance parameters of the current frame by calculating the loss between the multi-view images of the current frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the initialized appearance parameters of the current frame; S24: Fix the optimized appearance parameters of the current frame, and optimize the relative pose parameters by calculating the loss between the multi-view images of the adjacent frames of the current frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the action of the relative pose parameters on the optimized appearance parameters of the current frame; S25: Multiply the optimized relative pose parameters by the absolute pose parameters to obtain the updated absolute pose parameters, and use the adjacent frames of the current frame as the optimized frames; Execute S21 until all frames are processed to obtain an optimized frame set; S26: Based on the optimized frame set, obtain the three-dimensional dynamic object representation by calculating the loss between the multi-view images of each optimized frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the object global three-dimensional appearance parameters and the absolute pose parameters.

3. The dynamic scene reconstruction method based on the separation of dynamic objects and static scenes according to claim 1, characterized in that, S2 also includes representing the three-dimensional appearance and geometric structure of dynamic objects using three-dimensional Gaussian splashes, which is achieved through the formula: where T is the pose parameter, T = {R, t}, R is the rotation matrix, and t is the translation vector; is the appearance parameter; P is the total number of Gaussian ellipsoids; p is the Gaussian ellipsoid serial number; represents the mean of the Gaussian distribution; ||q p || = 1 represents the rotation parameter in quaternion form; where matrix_to_quaternion(R) represents the conversion from the rotation matrix to quaternion; q T represents the quaternion corresponding to the rotation matrix R; represents the multiplication of quaternions; represents the anisotropic scaling factor; represents the transparency of each Gaussian ellipsoid; sh p represents the spherical harmonic coefficient.

4. The dynamic scene reconstruction method based on the separation of dynamic objects and static scenes according to claim 3, characterized in that S2 optimizes the relative pose parameters by calculating the loss between the multi-view images of the current frame and its adjacent frames and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the optimized appearance parameters of the current frame under the action of the relative pose parameters, which is achieved by the formula: Implementation; In the formula, is the relative pose parameter during the optimization process; is the relative pose parameter; is the appearance parameter of the current frame; is the corresponding rendered image obtained by three-dimensional Gaussian splashing under the action of the relative pose parameter on the optimized appearance parameter of the current frame; is the image of the adjacent frame of the current frame at view angle k, is the mask of the object of the adjacent frame of the current frame at view angle k.

5. The dynamic scene reconstruction method based on the separation of dynamic objects and static scenes according to claim 3, characterized in that, S2, based on the optimized frame set, obtains a three-dimensional dynamic object representation by calculating the loss between each multi-view image of the optimized frame and the corresponding rendered image obtained by three-dimensional Gaussian splashing under the global three-dimensional appearance parameters and absolute pose parameters of the object, which is implemented by the formula: Implementation; where L global is the loss value; s is the sampled frame in the optimized frame set S; T s is the absolute pose parameter; is the global three-dimensional appearance parameter of the object; is the corresponding rendered image obtained by three-dimensional Gaussian splashing under the global three-dimensional appearance parameter and the absolute pose parameter of the object; is the image of the sampling frame s at the viewing angle k, is the mask of the object of the sampling frame s at the viewing angle k.

6. The dynamic scene reconstruction method based on the separation of dynamic objects and static scenes according to claim 1, wherein For S3, based on the three-dimensional dynamic object representation, use the rotation matrix and translation vector of adjacent frames to calculate the interpolation pose. Specifically: Based on the three-dimensional dynamic object representation, using the rotation matrices of adjacent frames, through spherical linear interpolation of quaternions, calculate the rotation interpolation; Based on the translation vectors of adjacent frames, perform linear weighting to calculate the translation interpolation; According to the formula: T(λ) = {R(λ), t(λ)}, obtain the interpolated pose; In the formula, T(λ), R(λ), and t(λ) are the interpolated pose, rotation interpolation, and translation interpolation respectively, and λ is the interpolation parameter.

7. The dynamic scene reconstruction method based on separation of dynamic objects and static scenes according to claim 1, wherein In step S3, after multiplying the interpolated pose by the global three-dimensional appearance parameters of the object and then superimposing it with the static scene, generate the reconstructed dynamic scene. Specifically: According to the formula: Generate a reconstructed dynamic scene; wherein, are respectively the appearance representations after dot multiplication of the reconstructed dynamic scene, static scene, interpolated pose, and the global three-dimensional appearance parameter points of the object; The static scene is obtained by reconstructing the image collected without dynamic objects.

8. The dynamic scene reconstruction method based on the separation of dynamic objects and static scenes according to claim 2, characterized in that, In step S22, initialize the appearance parameters of the current frame according to the point cloud data and absolute pose parameters of the object in the reference frame. Specifically: According to the formula: P i = T i ·P init , generate the current frame of point cloud, and initialize the current frame appearance parameters with the current frame of point cloud; Where, P i is the current frame point cloud, T i is the absolute pose parameter, and P init is the object point cloud data of the reference frame.

9. The dynamic scene reconstruction method based on the separation of dynamic objects and static scenes according to claim 1, characterized in that In step S1, use the reference frame to perform instance segmentation on the multi-view images of consecutive frames to obtain a multi-view image sequence of the target object and the corresponding target object masks. The target object mask of the reference frame is processed by the visual hull algorithm to obtain the point cloud data of the object in the reference frame. Specifically: After performing object detection on the multi-view images of the reference frame, perform instance segmentation on the multi-view images of consecutive frames to generate a multi-view image sequence of the target object and the corresponding target object masks; Back-project the target object mask of the multi-view image of the reference frame into three-dimensional space to obtain a contour cone, take the intersection of the contour cones of each view as the visual hull of the target object, and perform voxelization sampling on the visual hull to obtain the point cloud data of the object in the reference frame.

10. A dynamic scene reconstruction system based on the separation of dynamic objects and static scenes, characterized in that, It includes: Separation module: used to collect consecutive frame images and the multi-view images of the corresponding frames, select one frame from the consecutive frame images as the reference frame, use the reference frame to perform instance segmentation on the multi-view images of consecutive frames to obtain a multi-view image sequence of the target object and the corresponding target object masks, and process the target object mask of the reference frame by the visual hull algorithm to obtain the point cloud data of the object in the reference frame; Three-dimensional dynamic object generation module: Based on the multi-view image sequence of the target object and the camera parameters, optimize the appearance parameters of the current frame by fixing the relative pose parameters; Fix the optimized appearance parameters of the current frame, and optimize the relative pose parameters by calculating the loss between the multi-view images of the adjacent frames of the current frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the action of the relative pose parameters on the optimized appearance parameters of the current frame; Multiply the optimized relative pose parameters by the absolute pose parameters to obtain the updated absolute pose parameters, and use the adjacent frames of the current frame as the optimized frames; Return to enter the three-dimensional dynamic object generation module until all frames are processed to obtain a set of optimized frames; Based on the set of optimized frames, obtain the three-dimensional dynamic object representation by calculating the loss between the multi-view images of each optimized frame and the corresponding rendered images obtained by three-dimensional Gaussian splashing under the object's global three-dimensional appearance parameters and absolute pose parameters; Dynamic scene reconstruction module: Based on the three-dimensional dynamic object representation, use the rotation matrices and translation vectors of adjacent frames to calculate the interpolated pose. After multiplying the interpolated pose by the global three-dimensional appearance parameters of the object, superimpose it with the static scene to generate the reconstructed dynamic scene.

Images