Three-dimensional dynamic scene reconstruction method and apparatus, and storage medium

Abstract

The present disclosure relates to the field of computer vision and discloses a three-dimensional dynamic scene reconstruction method and apparatus, and a storage medium. The three-dimensional dynamic scene reconstruction method comprises: acquiring synchronized videos of a plurality of viewpoints of a dynamic scene; computing matching points between video images of different viewpoints, and estimating intrinsic and extrinsic parameters of each camera; obtaining a Gaussian splatting point set {p0} on the basis of a sparse point cloud constructed according to the depth of each matching point; for the first image frame of each video, using {p0} to perform static training thereon, to obtain a Gaussian splatting point set {p}; for the remaining image frames, dividing {p} into a static point set {S} and a dynamic point set {D}, performing dynamic training on {D}, and constructing a dynamic Gaussian splatting point set {P} from {p}, {S}, and the final {D}; and, in view of the intrinsic and extrinsic parameters of each camera, rendering {P} using a Gaussian splatting rendering pipeline, to obtain rendered images at different moments from new viewpoints.

Description

Method, device and storage medium for reconstructing three-dimensional dynamic scene

[0001] Cross-reference to related applications

[0002] The present disclosure is based on and claims priority to Chinese Patent Application No. 202411290735.2, filed on September 14, 2024, the entire contents of which are hereby incorporated by reference into the present disclosure. TECHNICAL FIELD

[0003] The present disclosure belongs to the field of computer vision, and particularly relates to a method, device and storage medium for reconstructing a three-dimensional dynamic scene based on multi-view video. BACKGROUND

[0004] With the vigorous development of virtual reality technology and robot technology in recent years, more and more application scenarios not only need to understand two-dimensional scenes and pictures, but also need algorithms or agents to understand, reconstruct and analyze geometric and texture information in three-dimensional world. Among them, reconstructing a dynamic three-dimensional scene from multi-view video is a very important and practical task. For example, in virtual reality, it is necessary to reproduce a dynamic scene in a virtual world to achieve a realistic experience of the real world in a virtual environment; or in dynamic environment navigation of intelligent robots, it may be necessary to directly model the visual changes of the dynamic environment.

[0005] Currently, researchers have proposed three-dimensional scene reconstruction methods based on multi-plane images, layered rendering, neural radiance fields and dynamic point clouds. However, these methods do not perform well in dynamic scene reconstruction, and are inefficient, do not support streaming and incremental updates.

[0006] In fact, for a three-dimensional dynamic scene, which often contains a large number of immobile backgrounds or static objects or environments, this part of data is actually irrelevant to time, so if the dynamic and static primitives can be separated, the amount of calculation required for reconstruction can be greatly reduced, and the reconstruction process can be greatly accelerated. However, such explicit separation is not possible for the implicit three-dimensional representation based on the previous method, because these methods do not have explicit physical quantities of three-dimensional points in the primitive representation space, but use continuous functions to approximate the shape and appearance of the entire scene. Therefore, in order to solve these problems, researchers have begun to focus on dynamic three-dimensional scene reconstruction techniques based on Gaussian splatting. Gaussian splatting is an effective method for converting three-dimensional point cloud data into voxel or surface representation, and its basic idea is to take each three-dimensional point as the center of the Gaussian kernel, and "project" or "diffuse" to the surrounding voxel space according to the density and color information of the point, forming a continuous voxel model. In dynamic three-dimensional scene reconstruction, Gaussian splatting can well capture the details and boundaries of moving objects, and due to its discrete nature, it is easier to achieve explicit separation of dynamic and static parts compared to continuous function approximation methods. By matching and tracking the point cloud data between different frames, dynamic parts can be efficiently identified and updated, while maintaining a stable representation of the static background, thereby significantly improving reconstruction speed and efficiency. In addition, Gaussian splatting also supports streaming processing and incremental updating, which means it can adapt to real-time scene changes, receive and process newly acquired data in real time, and then update the reconstructed three-dimensional scene model in a timely manner. This is undoubtedly an important advantage for virtual reality applications and robot dynamic environment navigation. Existing dynamic scene reconstruction techniques based on Gaussian splatting do not make good use of the properties of Gaussian splatting as a display three-dimensional representation, but simply use methods in neural radiance fields in Gaussian splatting, so it is difficult to achieve efficient streaming reconstruction and real-time rendering, and usually requires relatively large memory to store Gaussian splatting points. SUMMARY

[0007] The present disclosure aims to at least partially solve one of the technical problems existing in the prior art.

[0008] To this end, the embodiments of the present disclosure propose a three-dimensional dynamic scene reconstruction method, device and storage medium based on multi-view video, which utilizes the display characteristics of three-dimensional Gaussian splatting to construct Gaussian splatting points from multi-view video and realizes dynamic and static separation of Gaussian points according to pixel-level changes, achieving fast, streamable and incrementally updated dynamic scene reconstruction.

[0009] In order to achieve the above purpose, the present disclosure adopts the following technical solutions:

[0010] The first aspect of the present disclosure proposes a three-dimensional dynamic scene reconstruction method, comprising the following steps:

[0011] S1, acquiring multi-view synchronous videos of a dynamic scene as training data using multiple cameras;

[0012] S2, calculating matching points between different view video images to estimate the internal and external parameters of each camera;

[0013] S3, constructing a sparse point cloud according to the depth of each matching point, and generating an initial set of Gaussian splatting points {p0} according to the sparse point cloud;

[0014] S4, for the first frame images of all videos in the training data, calculating the color radiance value of a three-dimensional space point using each Gaussian splatting point in the Gaussian splatting point set {p0} and through Gaussian distribution, obtaining a rendered image according to the color radiance value, and performing the following iterative optimization on the Gaussian splatting point set {p0}: calculating the gradient value of a two-dimensional Gaussian point projected onto a plane according to the loss function between the rendered image and the first frame image, judging whether to perform Gaussian densification processing on the Gaussian splatting point set {p0} according to the size relationship between the gradient value and a split threshold, deleting the Gaussian points in the Gaussian splatting point set {p0} whose volume is greater than a volume set threshold, the Gaussian points that are invisible from all views, and the Gaussian points whose transparency is less than a transparency set threshold, and obtaining a Gaussian splatting point set {p} for the first frame images of all videos;

[0015] For the remaining frame images of all videos, calculating the pixel point change variance of each view video, optimizing the change voxel field according to the calculated pixel point change variance, dividing the Gaussian splatting point set {p} into a static point set {S} and a dynamic point set {D} according to the value of the change voxel field; the static point set {S} is no longer updated at the time steps corresponding to the remaining frames; for the dynamic point set {D}, updating the attributes of each dynamic point at the time steps corresponding to the remaining frame images according to the rigid body constraint between the dynamic point set {D} and the Gaussian splatting point set {p} and the rendering loss of the dynamic point set {D} with the remaining frame images;

[0016] The dynamic Gaussian splatting point set is composed of the Gaussian splatting point set {p}, the static point set {S}, and the finally obtained dynamic point set {D}

[0017] S5, combining the internal and external parameters of the cameras to obtain the dynamic Gaussian splatting point set Rendering according to the rendering pipeline of Gaussian splatting to obtain rendered images of new views at different times, and realizing dynamic three-dimensional scene reconstruction.

[0018] In some embodiments, in step S2, the pixel matching points between the different view video images are calculated by a motion recovery structure method.

[0019] In some embodiments, in step S4, the i-th Gaussian splatting point in the Gaussian splatting point set {p0} is represented as {x i ,Ri s i c i o i x i R i s i c i o i respectively, the position, rotation angle, scale, color and transparency of the i-th Gaussian splatting point, and the calculation formula of the color radiance value of the three-dimensional space point is:

[0020] wherein c(x) represents the color radiance value of the three-dimensional space point x, and N represents the total number of Gaussian splatting points in the Gaussian splatting point set {p0};

[0021] For the first frame image of each view video, each Gaussian splatting point in the Gaussian splatting point set {p0} is projected onto the imaging plane of the corresponding view, and the projected two-dimensional Gaussian splatting points are sequentially rendered according to the depth from small to large, to obtain a rendering image I corresponding to the Gaussian splatting point set {p0}, and the formula of the differentiable rendering is as follows:

[0022] wherein c(p) is the color of the point p in the rendering image I, and i (p) represents the transparency of the point p in the rendering image I obtained by the i-th Gaussian splatting point in the Gaussian splatting point set {p0}, μ i is the projection position of the i-th Gaussian splatting point in the Gaussian splatting point set {p0} in the rendering image I, Cov i is the covariance matrix of the i-th Gaussian splatting point after projection in the rendering image I.

[0023] In some embodiments, the iterative optimization of the Gaussian splatting point set {p0} in step S4 specifically includes:

[0024] At each iteration in the optimization process, the gradient value of the two-dimensional Gaussian point projected onto the plane is calculated according to the loss function between the rendering image and the first frame image;

[0025] Every time K1 iterations are passed, the following operations are performed: calculating the gradient average value of each Gaussian splash point in the Gaussian splash point set {p0} recorded in the K1 iterations, and splitting or cloning each Gaussian splash point according to the projection size of the Gaussian splash point for each Gaussian splash point with a gradient average value greater than a splitting threshold, wherein, for a Gaussian splash point with a body volume greater than a volume threshold, the Gaussian splash point is split according to Gaussian distribution probability density, and for a Gaussian splash point with a body volume less than or equal to the volume threshold, a Gaussian splash point identical to the original Gaussian splash point is generated by using a cloning operation, so as to realize Gaussian densification processing of the Gaussian splash point; at the same time, a Gaussian point pruning method is used to remove the Gaussian splash point with a body volume greater than the volume threshold or a projection area greater than an area threshold, and remove all Gaussian splash points invisible from all perspectives;

[0026] Every time K2 iterations are passed, K2>K1, all Gaussian splash points with a transparency less than a transparency threshold are removed, and then the transparency of all Gaussian splash points is set to a fixed value less than the transparency threshold, so as to delete the Gaussian splash points with occlusion.

[0027] In the case of reaching the upper limit of the number of iterations, the Gaussian splash point set {p} for the first frame image of all videos is obtained.

[0028] In some embodiments, in step S4, the loss function between the rendered image and the first frame image is Loss(I, I g ), and the formula is as follows: Loss(I, I g ) = L2(I, I g ) + λ1·L D-SSIM (I, I g ) + λ2·L LPIPS (I, I g )

[0029] Wherein, I, I g represent the rendered image and the real image obtained by the Gaussian splash point set {p0} respectively; L2, L D-SSIM represent the 2-norm loss function and the structural dissimilarity loss function respectively, both of which are used to constrain the pixel-level similarity between images; L LPIPS represents a learnable perceptual image block similarity loss function, which is used to reduce the perceptual distance between the rendered image and the real image; λ1 and λ2 are the weights of the loss functions L D-SSIM and L LPIPS .

[0030] In some embodiments, in step S4, for the remaining frame images of all videos, the specific steps of dividing the Gaussian splash point set {p} into a static point set {S} and a dynamic point set {D} include:

[0031] For each consecutive video from multiple perspectives, calculate the variance of pixel changes over time, and perform Gaussian blur on the pixel variance to ensure the continuity of the boundary. Based on whether the pixel variance is greater than the dynamic threshold, each pixel is divided into dynamic pixels and static pixels.

[0032] A randomly initialized variable voxel field V(x,y,z) is used to represent whether a voxel (x,y,z) in space is dynamic. The variable voxel field V(x,y,z) is then optimized under supervised conditions by rendering the differentiable variable voxel field V(x,y,z) and using the variance of pixel changes over time in each video as the basis for supervision.

[0033] Among them, L V Represents the optimized variable voxel field; Here is the expectation operator; M(r) represents the dynamic / static category of the pixel corresponding to the ray r, M(r) = 1 indicates that the pixel is a dynamic pixel, and M(r) = 0 indicates that the pixel is a static pixel; This represents the change in voxel field V(r(y) at point r using light rays r). i Estimated pixel variation field The first ray on ray r One sampling point, This indicates the position of the sampling point on the ray r; s represents the sigmoid activation function; N r This represents the set of Gaussian sputtering points through which the ray r passes;

[0034] The voxels corresponding to each Gaussian sputtering point in the Gaussian sputtering point set {p} are found according to the nearest neighbor principle. Then, each Gaussian sputtering point is determined to be a dynamic Gaussian sputtering point according to whether the voxel is a dynamic voxel. That is, for any Gaussian sputtering point in the Gaussian sputtering point set {p}, if the voxel corresponding to the Gaussian sputtering point is a dynamic voxel, then the Gaussian sputtering point is a dynamic Gaussian sputtering point; otherwise, the Gaussian sputtering point is a static Gaussian sputtering point. Thus, the Gaussian sputtering point set {p} is divided into a static point set {S} and a dynamic point set {D}.

[0035] In some embodiments, in step S4, the rigid body constraint between the k-th Gaussian sputtering point in the dynamic point set {D} and the l-th Gaussian sputtering point in the Gaussian sputtering point set {p} is set as follows: Its expression is:

[0036] in, and Let these represent the position and rotation angle of the k-th Gaussian sputtering point in the dynamic point set {D} at time t. and represents the position and rotation angle of the lth Gaussian sputtering point in the Gaussian sputtering point set {p} at the tth frame; w k,l represents the loss function weight, which is negatively related to the distance between the two Gaussian sputtering points, and is specifically represented as and represent the positions of the kth and lth Gaussian sputtering points in the first frame image.

[0037] In some embodiments, in step S4, the rigid body constraint between the dynamic point set {D} and the Gaussian sputtering point set {p} is L rigid , which is represented as follows:

[0038] wherein, represents the total number of Gaussian sputtering points in the dynamic point set {D}, represents a set composed of the kth Gaussian sputtering point selected from the Gaussian sputtering point set {p} and its w nearest points.

[0039] The second aspect of the present disclosure provides a reconstruction device based on the reconstruction method of any one of the embodiments of the first aspect of the present disclosure, comprising:

[0040] A first module has stored therein training data, the training data being multi-view synchronous videos of a dynamic scene acquired using a plurality of cameras;

[0041] A second module is configured to calculate matching points between different view video images, so as to estimate the internal and external parameters of each camera;

[0042] A third module is configured to construct a sparse point cloud according to the depth of each matching point, and generate an initial Gaussian sputtering point set {p0} according to the sparse point cloud;

[0043] A fourth module is configured to, for the first frame image of all videos in the training data, calculate the color radiance value of a three-dimensional space point using each Gaussian sputtering point in the Gaussian sputtering point set {p0} and through Gaussian distribution, obtain a rendered image according to the color radiance value, and perform the following iterative optimization on the Gaussian sputtering point set {p0}: calculate the gradient value of a two-dimensional Gaussian point projected onto a plane according to the loss function between the rendered image and the first frame image, determine whether to perform Gaussian densification processing on the Gaussian sputtering point set {p0} according to the size relationship between the gradient value and a split threshold, delete the Gaussian points in the Gaussian sputtering point set {p0} whose volume is greater than a volume set threshold, the Gaussian points that are invisible from all views, and the Gaussian points whose transparency is less than a transparency set threshold, and obtain the Gaussian sputtering point set {p} for the first frame image of all videos.

[0044] For the rest of the frame images of all videos, the pixel point change variance of each view video is calculated, the change voxel field is optimized according to the calculated pixel point change variance, and the Gaussian sputtering point set {p} is divided into a static point set {S} and a dynamic point set {D} according to the value of the change voxel field; the static point set {S} is no longer updated at the time steps corresponding to the rest of the frames; for the dynamic point set {D}, the attributes of each dynamic point at the time steps corresponding to the rest of the frame images are updated according to the rigid body constraint of the dynamic point set {D} and the Gaussian sputtering point set {p} and the rendering loss of the dynamic point set {D} and the rest of the frame images;

[0045] The dynamic Gaussian sputtering point set is composed of the Gaussian sputtering point set {p}, the static point set {S} and the finally obtained dynamic point set {D}

[0046] The fifth module is configured to combine the internal and external parameters of the camera to update the dynamic Gaussian sputtering point set According to the rendering pipeline of Gaussian sputtering, the rendering images of the new view at different times are obtained, and the dynamic three-dimensional scene reconstruction is realized.

[0047] The third aspect of the present disclosure provides a computer readable storage medium, which stores computer instructions for causing the computer to execute the reconstruction method of any one of the embodiments of the first aspect of the present disclosure.

[0048] Compared with the prior art, the present disclosure has the following characteristics:

[0049] 1. In the three-dimensional dynamic reconstruction, the present disclosure introduces a voxel change field, which can well utilize the prior information in the multi-view video to capture the dynamic situation of the three-dimensional space, and greatly reduces the number of Gaussian sputtering points required for expressing the three-dimensional dynamic scene.

[0050] 2. The present disclosure uses Gaussian sputtering points and volume rendering to represent the three-dimensional scene and render new view images, which can complete the reconstruction of the dynamic scene within minutes, is less likely to have "floating objects" and "ghosts" and other phenomena compared with the method based on neural radiation field, and is more adaptable to the case of fewer input views.

[0051] 3. The present disclosure can be applied to various fields requiring three-dimensional dynamic scene reconstruction, can quickly and incrementally reconstruct the three-dimensional dynamic scene, and can realize streaming and real-time rendering on any end-side device supporting CUDA or OpenGL. BRIEF DESCRIPTION OF DRAWINGS

[0052] FIG. 1 is a whole flowchart of a three-dimensional dynamic scene reconstruction method according to the first aspect of the present disclosure.

[0053] FIG. 2 is a structural schematic diagram of an electronic device according to the third aspect of the present disclosure. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure is described in further detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and do not limit the present disclosure.

[0055] On the contrary, the present disclosure covers any substitution, modification, equivalent method and solution defined by the claims on the essence and scope of the present disclosure. Further, in order to make the public have a better understanding of the present disclosure, some specific details are described in detail in the following detailed description of the present disclosure. The present disclosure can also be completely understood without the description of these details by those skilled in the art.

[0056] Referring to FIG. 1, the method for reconstructing a three-dimensional dynamic scene provided by the first aspect embodiment of the present disclosure comprises the following steps:

[0057] S1, using a plurality of cameras to acquire multi-view synchronous videos of the dynamic scene as training data;

[0058] S2, calculating matching points between different view video images to estimate the internal and external parameters of each camera;

[0059] S3, constructing a sparse point cloud according to the depth of each matching point, and generating an initial set of Gaussian splash points {p0} according to the sparse point cloud;

[0060] S4, for the first frame images of all videos in the training data, calculating the color radiance value of any point in the three-dimensional space by using each Gaussian splash point in the Gaussian splash point set {p0} and through Gaussian distribution, obtaining a rendered image according to the color radiance value, and performing the following iterative optimization on the Gaussian splash point set {p0}: calculating the gradient value of the two-dimensional Gaussian point projected onto the plane according to the loss function between the rendered image and the first frame image, judging whether to perform Gaussian densification processing on the Gaussian splash point set {p0} according to the size relationship between the gradient value and the split threshold, deleting the Gaussian points in the Gaussian splash point set {p0} whose volume is greater than a volume set threshold, the Gaussian points that are invisible from all perspectives, and the Gaussian points whose transparency is less than a transparency set threshold, and obtaining the Gaussian splash point set {p} for the first frame images of all videos;

[0061] For the rest of the frame images of all videos, the pixel point change variance of each view video is calculated, the change voxel field is optimized according to the calculated pixel point change variance, and the Gaussian sputtering point set {p} is divided into a static point set {S} and a dynamic point set {D} according to the value of the change voxel field; the static point set {S} is no longer updated at the time step corresponding to the rest of the frames; for the dynamic point set {D}, the attributes of each dynamic point at the time step corresponding to the rest of the frame images are updated according to the rigid body constraint of the dynamic point set {D} and the Gaussian sputtering point set {p} and the rendering loss of the dynamic point set {D} and the rest of the frame images;

[0062] The dynamic Gaussian sputtering point set is composed of the Gaussian sputtering point set {p}, the static point set {S} and the finally obtained dynamic point set {D}

[0063] S5, combining the internal and external parameters of the camera to obtain the dynamic Gaussian sputtering point set According to the rendering pipeline of Gaussian sputtering, the rendering images of the new view at different times are obtained, and the dynamic three-dimensional scene reconstruction is realized.

[0064] In some embodiments, when the training data is acquired by step S1, multiple cameras are used to capture multi-view synchronous videos of the dynamic scene at different positions in the scene with fixed poses, and the synchronization of the multi-view videos is ensured through the hardware shutter connection between the multiple cameras.

[0065] In some embodiments, the motion recovery structure (Structure from Motion, SfM) method is used to calculate the pixel matching points between the video images of different views in step S2, and the internal and external parameters of each camera are estimated by using the existing three-dimensional vision method.

[0066] In some embodiments, in step S3, the depth corresponding to each pixel matching point is calculated from the pixel matching points generated in the motion recovery structure process, so as to construct a set of sparse point clouds, and the positions of the Gaussian sputtering points are initialized by using the sparse point clouds, thereby obtaining an initial set of Gaussian sputtering point set {p0}.

[0067] In some embodiments, step S4 first performs static training on the initial Gaussian sputtering point set {p0} by using the first frame images of each video in the training data, to obtain the Gaussian sputtering point set {p} of the first frame images, and then performs dynamic expansion on the obtained Gaussian sputtering point set {p} by using the rest of the frame images of each video in the training data, to obtain the expanded Gaussian sputtering point set Finally, the dynamic Gaussian sputtering point set is composed of the Gaussian sputtering point set {p} and the expanded Gaussian sputtering point set Step S4 specifically includes the following steps:

[0068] S41, fitting the first frame Gaussian sputtering point ​

[0069] After initialization, first train static Gaussian splatting points on the first frame of multi-view synchronous video; Gaussian splatting points express the radiance field of three-dimensional space using a set of spatial anisotropic Gaussian functions, specifically, each Gaussian splatting point contains parameters such as position x i , rotation angle R i , scale s i , color c i and transparency o i , denoted as {x i , R i , s i , c i , o i}, and the subscript i represents the serial number of the Gaussian splatting point; the specific process of static training is as follows:

[0070] S411, use each Gaussian splatting point in the Gaussian splatting point set {p0} obtained in step S3 and calculate the color radiance value of the spatial point x through Gaussian distribution, the calculation formula is as follows:

[0071] Wherein, c(x) represents the color radiance value of the three-dimensional space point x, {x i , R i , s i , c i , o i} is the Gaussian splatting point allowed to be optimized, N represents the total number of Gaussian splatting points in the Gaussian splatting point set {p0}, and the superscript T represents transposition;

[0072] S412, for the first frame image of each view video, project each Gaussian splatting point in the Gaussian splatting point set {p0} onto the imaging plane of the view, and sequentially perform differentiable rendering on each two-dimensional Gaussian splatting point according to the depth from small to large, to obtain the rendering image I corresponding to the Gaussian splatting point set {p0}, the formula of differentiable rendering is as follows:

[0073] Wherein, c(p) is the color of the point p in the rendering image I, a i (p) represents the transparency of the point p in the rendering image I obtained through the i-th Gaussian splatting point, μ i is the projection position of the i-th Gaussian splatting point in the rendering image I, Cov i is the covariance matrix of the i-th Gaussian splatting point after projection in the rendering image I. Such a rendering process can be rendered according to image blocking, and the rendering of each block image is completed by a CUDA (Compute Unified Device Architecture) process block, so as to fully utilize the powerful parallel computing capability of the image processing unit;

[0074] S413, set the loss function of the rendered image and the first frame image as Loss(I, I g ), and the formula is as follows: Loss(I, I g ) = L2(I, I g ) + λ1·L D-SSIM (I, I g ) + λ2·L LPIPS (I, I g )

[0075] Wherein, I, I g respectively represent the rendered image and the real image obtained by the Gaussian splash point set {p0}; L2, L D-SSIM respectively represent the 2-norm loss function and the structural dissimilarity loss function, which are used to constrain the pixel-level similarity between images; L LPIPS represents a learnable perceptual image block similarity loss function, which is used to reduce the perceptual distance between the rendered image and the real image; λ1 and λ2 are the weights of the loss functions L D-SSIM and L LPIPS , and in the embodiments of the present disclosure, λ1 = 1.0 and λ2 = 2.0.

[0076] S414, processing of the Gaussian splash point set {p0} in the iteration process

[0077] S4141, in each iteration in the optimization process, the gradient value of the two-dimensional Gaussian point projected onto the plane is calculated according to the loss function Loss(I, I g ). In order to add Gaussian splash points to the positions originally without Gaussian splash points in the space, a Gaussian densification process is used to add Gaussian splash points at the vacancies in the space: every time K1 = 100 iterations are updated, the average value of the gradient of the two-dimensional projection of each Gaussian splash point recorded in the 100 iterations is calculated; for each Gaussian splash point, if the average value of the two-dimensional point gradient of the past 100 iterations is greater than a split threshold thresh 2D (in the embodiments of the present disclosure, thresh 2D is empirically set to 0.001), then the Gaussian splash point is split or cloned according to the projection size of the Gaussian splash point, for the Gaussian splash point with a relatively large volume, the split is performed according to the probability density of the Gaussian distribution, and for the Gaussian splash point with a relatively small volume, a cloning operation is used to generate a Gaussian splash point identical to the original Gaussian splash point. In the embodiments of the present disclosure, the volume threshold is set to 0.0004, and for the Gaussian splash point with a volume greater than 0.0004, the split is performed, and for the Gaussian splash point with a volume less than or equal to 0.0004, the cloning is performed. For each Gaussian splash point with a gradient average value less than or equal to the split threshold thresh 2D , no Gaussian densification processing is performed.

[0078] S4142. During the optimization process, Gaussian point pruning method is used to remove unnecessary Gaussian sputtering points: after every 100 iterations, Gaussian points with excessive volume (the volume threshold is set to 0.0004 in the embodiment of this disclosure) or excessive area after projection (the area threshold is set to 0.2 in the embodiment of this disclosure) are deleted, and Gaussian sputtering points that are not visible from any training viewpoint are also removed.

[0079] S4143. While performing steps S4141 and S4142, after every K2 = 1000 optimization cycles, all Gaussian sputtering points with transparency less than the transparency threshold (in the embodiments of this disclosure, the transparency threshold is set to 0.02) are removed. Then, the transparency of all Gaussian sputtering points is set to 0.01, and subsequent optimization is performed. This step is repeated continuously to remove occluded Gaussian sputtering points. When the upper limit of the number of iterations is reached, the Gaussian sputtering point set {p} for the first frame image of all videos is obtained.

[0080] S42. Dynamically expand the Gaussian sputtering point set {p} obtained in step S41 to obtain the extended Gaussian sputtering point set. The specific process is as follows:

[0081] S421. For each consecutive video of the multi-view video, calculate the variance of the pixel points of each video over time, and perform Gaussian blur operation on the variance of the pixel point changes to ensure the continuity of the boundary. According to whether the variance of the pixel point changes is greater than the dynamic threshold, each pixel point is divided into dynamic pixel points and static pixel points. In the embodiments of this disclosure, the dynamic threshold is set to 0.03 empirically.

[0082] S422. Randomly initialize a variable voxel field V(x,y,z) to represent whether a voxel (x,y,z) in space is dynamic. The variable voxel field V(x,y,z) is then optimized using differentiable rendering and the pixel variance of the multi-view video obtained in step S421 as supervised optimization.

[0083] Among them, L V Represents the optimized variable voxel field; Here, M(r) represents the expectation operator; M(r) denotes the dynamic / static category of the pixel corresponding to ray r, where M(r) = 1 indicates that the pixel is a dynamic pixel, and M(r) = 0 indicates that the pixel is a static pixel. This indicates the use of the changing voxel field at point r. Estimated pixel variation field The first ray on ray r One sampling point, represents the position of the sampling point on the light ray r; s represents a sigmoid activation function; N r represents the set of Gaussian sputtering points through which the light ray r passes;

[0084] S423, find the voxel corresponding to each Gaussian sputtering point in the set of Gaussian sputtering points {p} according to the nearest neighbor principle, and determine whether each Gaussian sputtering point is a dynamic Gaussian sputtering point according to whether the voxel is a dynamic voxel, that is, for any Gaussian sputtering point, if the voxel corresponding to the Gaussian sputtering point is a dynamic voxel, the Gaussian sputtering point is a dynamic Gaussian sputtering point, otherwise the Gaussian sputtering point is a static Gaussian sputtering point, thereby dividing the set of Gaussian sputtering points {p} into a static point set {S} and a dynamic point set {D}, for each static Gaussian sputtering point in it, it does not change in the subsequent training process, for each dynamic Gaussian sputtering point, then use the multi-view video to update using step S424.

[0085] S424, update all attributes of each dynamic Gaussian point in the dynamic point set {D} using the multi-view video data from the second frame to the last frame according to the loss function in step S413 combined with the following rigid regularization constraint;

[0086] In the training process of the subsequent frame, the training process is constrained using the rigid regularization term, and the rigid constraint between the kth Gaussian sputtering point in the dynamic point set {D} and the lth Gaussian sputtering point in the set of Gaussian sputtering points {p} is represents the following:

[0087] wherein, and represents the position and rotation angle of the kth Gaussian sputtering point at the tth frame, and represents the position and rotation angle of the lth Gaussian sputtering point at the tth frame; w k,l represents the corresponding loss function weight, and is negatively related to the distance between the two Gaussian sputtering points, and can be specifically represented as and represents the position of the kth and lth Gaussian sputtering point in the first frame image.

[0088] In actual calculation, for each dynamic Gaussian sputtering point in the dynamic point set {D}, only 20 nearest neighbor points of the dynamic Gaussian sputtering point are selected from the set of Gaussian sputtering points {p} to constrain the dynamic Gaussian sputtering point with the rigid body, thereby the rigid body constraint L rigid represents the following:

[0089] wherein, denotes the total number of Gaussian sputtering points in the dynamic point set {D}, denotes a set consisting of 20 nearest points of the kth Gaussian sputtering point selected from the Gaussian sputtering point set {p}.

[0090] S43, a dynamic Gaussian sputtering point set is formed by the Gaussian sputtering point set {p}, the static point set {S}, and the finally obtained dynamic point set {D}

[0091] In some embodiments, step S5 specifically comprises the following steps:

[0092] S51, a dynamic Gaussian sputtering point set is obtained according to step S4 and the intrinsic and extrinsic parameters of the target view camera to be rendered, the transformation matrix and the projection matrix of the camera are constructed using the intrinsic and extrinsic parameters of the camera.

[0093] S52, the positions of the dynamic Gaussian sputtering point set are mapped to the imaging plane of the target view camera using the transformation matrix, and the two-dimensional covariance matrix corresponding to the three-dimensional Gaussian sputtering point is obtained according to the linear assumption combined with the Jacobian variation matrix.

[0094] S53, for all two-dimensional Gaussian sputtering points obtained by projection in step S52, Gaussian sputtering point rendering is performed in the order of depth from small to large, and the specific calculation method is the same as described in step S412, and the obtained image is transmitted to the front-end display to complete the image synthesis under the new view angle.

[0095] S54, for different time steps, switch the Gaussian sputtering points to be rendered according to the required time steps, and then perform rendering according to steps S51 to S53.

[0096] Further, for a computing device with NVIDIA image processing unit, use the hierarchical design in the CUDA program design model to make a thread block containing 256 threads responsible for the rendering of a 16x16 image block in the image, where each thread is responsible for the rendering of a pixel. The Gaussian sputtering points needed in the thread calculation are pre-processed, and the related data is transferred to the shared memory of the graphics processing unit to realize real-time rendering.

[0097] Further, for a computing device that can apply the open graphics standard library (OpenGL), the projection of the three-dimensional Gaussian sputtering point and the transformation of the covariance matrix are realized in the primitive shader, the rendering process described in step S412 is realized in the fragment shader, and the depth test in OpenGL is closed, the blending mode is opened, and the blending mode is selected as ONE_MINUS_DST_ALPHA to realize real-time rendering in the OpenGL framework.

[0098] Further, for the computing device supporting web page end, using WebGL framework, using the same primitive shader and fragment shader as the computing device which can apply OpenGL above to achieve efficient rendering.

[0099] The second aspect embodiment of the present disclosure provides a reconstruction device of a three-dimensional dynamic scene, comprising:

[0100] The first module has stored therein training data, the training data being multi-view synchronous videos of a dynamic scene acquired using a plurality of cameras;

[0101] The second module is configured to calculate matching points between different-view video images, so as to estimate the internal and external parameters of each camera;

[0102] The third module is configured to construct a sparse point cloud according to the depth of each matching point, and generate an initial set of Gaussian spattering points {p0} according to the sparse point cloud;

[0103] The fourth module is configured to, for the first frame images of all videos in the training data, calculate the color radiance value of a three-dimensional space point using each Gaussian spattering point in the set of Gaussian spattering points {p0} and through Gaussian distribution, obtain a rendered image according to the color radiance value, and perform the following iterative optimization on the set of Gaussian spattering points {p0}: calculate the gradient value of a two-dimensional Gaussian point projected onto a plane according to a loss function between the rendered image and the first frame image, determine whether to perform Gaussian densification processing on the set of Gaussian spattering points {p0} according to the size relationship between the gradient value and a split threshold, delete Gaussian points with a volume greater than a volume set threshold, Gaussian points invisible from all views, and Gaussian points with a transparency less than a transparency set threshold in the set of Gaussian spattering points {p0}, and obtain a set of Gaussian spattering points {p} for the first frame images of all videos;

[0104] For the remaining frame images of all videos, calculate the pixel point change variance of each view video, optimize the change voxel field according to the calculated pixel point change variance, divide the set of Gaussian spattering points {p} into a static point set {S} and a dynamic point set {D} according to the value of the change voxel field; the static point set {S} is no longer updated at the time steps corresponding to the remaining frames; for the dynamic point set {D}, update the attributes of each dynamic point at the time steps corresponding to the remaining frame images according to the rigid body constraint of the dynamic point set {D} and the set of Gaussian spattering points {p} and the rendering loss of the dynamic point set {D} with respect to the remaining frame images;

[0105] The dynamic Gaussian spattering point set is composed of the set of Gaussian spattering points {p}, the static point set {S}, and the finally obtained dynamic point set {D}

[0106] The fifth module is configured to combine the internal and external parameters of the cameras to obtain the dynamic Gaussian spattering point set According to the rendering pipeline of Gaussian sputtering, rendering images of different moments of new perspectives are obtained, and dynamic three-dimensional scene reconstruction is realized.

[0107] It should be noted that the foregoing embodiment of the method for reconstructing a three-dimensional dynamic scene is also applicable to the embodiment of the device for reconstructing a three-dimensional dynamic scene of the present disclosure, and will not be repeated here.

[0108] In order to realize the above-mentioned embodiment, the present embodiment also proposes a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to perform the method for reconstructing a three-dimensional dynamic scene of the above-mentioned embodiment.

[0109] Reference will now be made to FIG. 2, which shows a structural schematic diagram of an electronic device suitable for implementing the embodiments of the present disclosure. It should be noted that the electronic device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), and the like, as well as fixed terminals such as digital TVs, desktop computers, servers, and the like. The electronic device shown in FIG. 2 is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0110] As shown in FIG. 2, the electronic device can include a processing device (such as a central processor, a graphics processor, etc.) 101, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 102 or loaded from a storage device 108 into a random access memory (RAM) 103. Various programs and data required for the operation of the electronic device are also stored in the RAM 103. The processing device 101, the ROM 102, and the RAM 103 are connected to each other through a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0111] Generally, the following devices can be connected to the I / O interface 105: input devices 106 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, and the like; output devices 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 108 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 109. The communication devices 109 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 2 shows an electronic device having various devices, it should be understood that it is not required to implement or have all the shown devices. More or fewer devices can be alternatively implemented or provided.

[0112] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication device 109, or installed from the storage device 108, or installed from the ROM 102. When the computer program is executed by the processing device 101, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0113] It should be noted that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium may, for example, be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained in the computer readable medium can be transmitted by any suitable medium, including, but not limited to, a wire, an optical fiber, an RF (radio frequency) or the like, or any suitable combination of the above.

[0114] The computer readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and not be assembled into the electronic device.

[0115] The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the above-mentioned three-dimensional dynamic scene reconstruction method.

[0116] Computer program code for carrying out operations of the present disclosure can be written in any one or more programming languages or combinations of languages including object or visual programming languages such as Java, Smalltalk, C++, Python, conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0117] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0118] In addition, the terms "first", "second", etc. are used only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0119] Any process or method descriptions or descriptions of the flow diagrams described herein or otherwise described in the specification can be understood as representing the modules, segments, or portions of code that include executable instructions for performing specific logical functions or steps in the processes, and the scope of the preferred embodiments of the present disclosure includes additional implementations that can not be expressly shown or described in the description. It will be appreciated by those skilled in the art that the scope of the embodiments of the present disclosure includes additional implementations that can not be expressly shown or described in the description, including implementations that can perform functions in a different order, in substantially simultaneous fashion, or in reverse order, depending on the functionality involved, as will be understood by those skilled in the art.

[0120] The logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered as a sequence of instructions to implement logic functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a machine-readable storage device (e.g., magnetic, optical or other) a machine-readable storage diskette (e.g., floppy, flexible or other), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), optical fibers, and a portable optical disc (e.g., CD-ROM). Additionally, the computer-readable medium can be paper or other suitable medium upon which the program is printed, as the program can be electronically captured, for example via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in order to be executed.

[0121] It should be understood that portions of the present disclosure can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0122] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the developed programs can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0123] In addition, each of the functional units in the various embodiments of the present disclosure can be integrated in one processing module, or each unit can exist physically separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0124] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A method for reconstructing a three-dimensional dynamic scene, comprising the following steps: S1. acquiring multi-view synchronous videos of the dynamic scene as training data using a plurality of cameras; S2. calculating matching points between video images of different views to estimate the internal and external parameters of each camera; S3. constructing a sparse point cloud according to the depth of each matching point and generating an initial set of Gaussian splatting points {p0} according to the sparse point cloud; S4. for the first frame images of all videos in the training data, calculating the color radiance value of a three-dimensional space point using each Gaussian splatting point in the set of Gaussian splatting points {p0} and through Gaussian distribution, obtaining a rendered image according to the color radiance value, and performing the following iterative optimization on the set of Gaussian splatting points {p0}: calculating the gradient value of a two-dimensional Gaussian point projected onto a plane according to a loss function between the rendered image and the first frame image, determining whether to perform Gaussian densification processing on the set of Gaussian splatting points {p0} according to the size relationship between the gradient value and a split threshold, deleting Gaussian points with a volume greater than a volume set threshold, Gaussian points invisible from all views, and Gaussian points with a transparency less than a transparency set threshold in the set of Gaussian splatting points {p0}, and obtaining a set of Gaussian splatting points {p} for the first frame images of all videos; for the remaining frame images of all videos, calculating the pixel point change variance of each view video, optimizing the change voxel field according to the calculated pixel point change variance, dividing the set of Gaussian splatting points {p} into a static point set {S} and a dynamic point set {D} according to the value of the change voxel field; the static point set {S} is no longer updated at the time steps corresponding to the remaining frames; for the dynamic point set {D}, updating the attributes of each dynamic point at the time steps corresponding to the remaining frame images according to the rigid body constraint of the dynamic point set {D} and the set of Gaussian splatting points {p} and the rendering loss of the dynamic point set {D} with the remaining frame images; The dynamic Gaussian sputtering point set {D} is composed of the Gaussian sputtering point set {p}, the static point set {S}, and the finally obtained dynamic point set {D} S5, combine the camera's intrinsic and extrinsic parameters with the dynamic Gaussian sputtering point set rendering according to the rendering pipeline of Gaussian splatting to obtain rendered images of new views at different times, and realizing dynamic three-dimensional scene reconstruction.

2. The reconstruction method of claim 1, wherein, In step S2, the pixel matching points between the video images of different views are calculated by a motion recovery structure method.

3. The reconstruction method of claim 1, wherein, In step S4, the i-th Gaussian sputtering point in the Gaussian sputtering point set {p0} is represented as {x i ,R i ,s i ,c i ,o i}, x i ,R i ,s i ,c i ,o i being the position, rotation angle, scale, color and transparency of the i-th Gaussian sputtering point respectively, and the calculation formula of the color radiation value of the three-dimensional space point being: Wherein, c(x) represents the color radiance value of a three-dimensional space point x, and N represents the total number of Gaussian splatting points in the set of Gaussian splatting points {p0}; For the first frame image of each view video, each Gaussian splatting point in the Gaussian splatting point set {p0} is projected onto the imaging plane of the corresponding view, and each two-dimensional Gaussian splatting point after projection is sequentially rendered according to the depth from small to large, to obtain a rendered image I corresponding to the Gaussian splatting point set {p0}. The formula of the differentiable rendering is as follows: where c(p) is the color of the point p in the rendered image I, a i (p) represents the transparency of the point p in the rendered image I obtained by the i-th Gaussian sputtering point in the Gaussian sputtering point set {p0}, μ i is the projection position of the i-th Gaussian sputtering point in the rendered image I in the Gaussian sputtering point set {p0}, Cov i is the covariance matrix of the i-th Gaussian sputtering point after projection in the rendered image I.

4. The reconstruction method of claim 1, wherein, In step S4, the iterative optimization performed on the set of Gaussian splatting points {p0} specifically includes: In each iteration in the optimization process, the gradient value of a two-dimensional Gaussian point projected onto a plane is calculated according to a loss function between the rendered image and the first frame image. Every time after K1 iterations, the following operations are performed: calculating the gradient average value of each Gaussian splash point in the Gaussian splash point set {p0} recorded in the K1 iterations, and splitting or cloning each Gaussian splash point according to the projection size of the Gaussian splash point, wherein, for a Gaussian splash point with a body volume greater than a volume threshold, the Gaussian splash point is split according to the probability density of the Gaussian distribution, and for a Gaussian splash point with a body volume less than or equal to the volume threshold, a Gaussian splash point identical to the original Gaussian splash point is generated by using a cloning operation, so as to realize Gaussian densification processing of the Gaussian splash point; meanwhile, a Gaussian point pruning method is used to remove the Gaussian splash point with a body volume greater than the volume threshold or a projection area greater than an area threshold, and all Gaussian splash points invisible from all perspectives are removed. Every time after K2 iterations, K2>K1, all Gaussian splash points with a transparency less than a transparency threshold are removed, and then the transparency of all Gaussian splash points is set to a fixed value less than the transparency threshold, so as to delete the Gaussian splash points with occlusion. In the case of reaching an upper limit of the number of iterations, the Gaussian splash point set {p} for the first frame image of all videos is obtained.

5. The reconstruction method of claim 1, wherein, In step S4, let the loss function between the rendered image and the first frame image be Loss(I, I g ), and the formula is as follows: Loss(I, I g ) = L2(I, I g ) + λ1·L D-SSIM (I, I g ) + λ2·L LPIPS (I, I g ) where I, I g denote the rendered image and the real image obtained from the Gaussian sputtering point set {p0}, respectively; L2, L D-SSIM denote the 2-norm loss function and the structural dissimilarity loss function, respectively, both of which are used to constrain the pixel-level similarity between images; L LPIPS denotes the learnable perceptual image block similarity loss function, which is used to reduce the perceptual distance between the rendered image and the real image; λ1and λ2are the weights of the loss functions L D-SSIM and L LPIPS , respectively.

6. The reconstruction method of claim 1, wherein, In step S4, for the remaining frame images of all videos, the specific steps of dividing the Gaussian splash point set {p} into the static point set {S} and the dynamic point set {D} include: For each continuous video of the multi-view video, the variance of the pixel points of each video with time is calculated, and a Gaussian blur operation is performed on the pixel point variance to ensure the continuity of the boundary, and each pixel point is divided into a dynamic pixel point and a static pixel point according to whether the pixel point variance is greater than a dynamic threshold; A change voxel field V(x, y, z) is randomly initialized to represent whether a voxel (x, y, z) in the space is dynamic, and the change voxel field V(x, y, z) is optimized by differentiable rendering of the change voxel field V(x, y, z) and variance of the pixel points of the videos over time as supervision, that is, V(x, y, z) = arg min V(x, y, z) E[||V(x, y, z) - V(x, y, z)||2] + λ||V(x, y, z)||2, wherein L V represents the optimized change voxel field; M(r) is the dynamic or static class of the pixel corresponding to the light ray r, M(r) = 1 means that the pixel belongs to dynamic pixel, M(r) = 0 means that the pixel is static pixel; represents the pixel variation field estimated at the light ray r by the variation voxel field V(r(y i )) at the light ray r, represents the first derivative of the light ray r on the surface S a sampling point, denotes the position of the sampling point on the ray r; s denotes a sigmoid activation function; N r denotes the set of Gaussian sputter points that the ray r passes through; According to the nearest neighbor principle, the voxels corresponding to each Gaussian splash point in the Gaussian splash point set {p} are found, and whether each Gaussian splash point is a dynamic Gaussian splash point is determined according to whether the voxel is a dynamic voxel, that is, for any Gaussian splash point in the Gaussian splash point set {p}, if the voxel corresponding to the Gaussian splash point is a dynamic voxel, the Gaussian splash point is a dynamic Gaussian splash point, otherwise the Gaussian splash point is a static Gaussian splash point, so as to divide the Gaussian splash point set {p} into the static point set {S} and the dynamic point set {D}.

7. The reconstruction method of claim 1, wherein, In step S4, suppose that the rigid body constraint between the kth Gaussian sputter point in the dynamic point set {D} and the lth Gaussian sputter point in the Gaussian sputter point set {p} is The expression is: wherein and respectively represent the position and rotation angle of the kth Gaussian sputter point in the dynamic point set {D} at the tth frame, and respectively represent the position and rotation angle of the lth Gaussian sputtering point in the set of Gaussian sputtering points {p} at the tth frame; w l,l represents the loss function weight, which is negatively related to the distance between the two Gaussian sputtering points, and is specifically represented as and respectively represent the positions of the kth and the lth Gaussian splash points in the first frame image.

8. The reconstruction method of claim 7, wherein, In step S4, let the rigid body constraint between the dynamic point set {D} and the Gaussian spray point set {p} be L rigid , is represented as follows: wherein denotes the total number of Gaussian sputter points in the dynamic point set {D}, represents the kth Gaussian sputter point selected from the set of Gaussian sputter points {p} a set composed of the kth and the lth Gaussian splash points.

9. A reconstruction device based on the reconstruction method of any one of claims 1-8, comprising: a first module having stored therein training data, the training data being a multi-view synchronous video of a dynamic scene acquired using a plurality of cameras; a second module configured to calculate matching points between images of different view videos, so as to estimate the intrinsic and extrinsic parameters of each camera; a third module configured to construct a sparse point cloud according to the depth of each matching point, and generate an initial set of Gaussian splash points {p0} according to the sparse point cloud; a fourth module configured to divide the set of Gaussian splash points {p0} into a static point set {S} and a dynamic point set {D} according to whether each Gaussian splash point is a dynamic Gaussian splash point, that is, for any Gaussian splash point in the set of Gaussian splash points {p0}, if the voxel corresponding to the Gaussian splash point is a dynamic voxel, the Gaussian splash point is a dynamic Gaussian splash point, otherwise the Gaussian splash point is a static Gaussian splash point. A fourth module is configured to, for the first frame image of all videos in the training data, calculate the color radiance value of a three-dimensional space point by using each Gaussian splatting point in a Gaussian splatting point set {p0} and through Gaussian distribution, obtain a rendered image according to the color radiance value, and perform the following iterative optimization on the Gaussian splatting point set {p0}: calculate the gradient value of a two-dimensional Gaussian point projected onto a plane according to a loss function between the rendered image and the first frame image, determine whether to perform Gaussian densification processing on the Gaussian splatting point set {p0} according to the size relationship between the gradient value and a split threshold, delete Gaussian points in the Gaussian splatting point set {p0} whose volumes are greater than a volume set threshold, Gaussian points that are invisible from all perspectives, and Gaussian points whose transparencies are less than a transparency set threshold, and obtain a Gaussian splatting point set {p} for the first frame image of all videos; For the remaining frame images of all videos, calculate the pixel point change variance of each perspective video, optimize the change voxel field according to the calculated pixel point change variance, divide the Gaussian splatting point set {p} into a static point set {S} and a dynamic point set {D} according to the value of the change voxel field, the static point set {S} is no longer updated at the time steps corresponding to the remaining frames, and for the dynamic point set {D}, update the attributes of each dynamic point at the time steps corresponding to the remaining frame images according to the rigid body constraint between the dynamic point set {D} and the Gaussian splatting point set {p} and the rendering loss of the dynamic point set {D} and the remaining frame images; The dynamic Gaussian sputtering point set {D} is composed of the Gaussian sputtering point set {p}, the static point set {S}, and the finally obtained dynamic point set {D} A fifth module is configured to combine the dynamic Gaussian sputtering point set with the intrinsic and extrinsic parameters of the camera. Render according to the rendering pipeline of Gaussian splatting to obtain rendered images of different time instants of new perspectives, and realize dynamic three-dimensional scene reconstruction.

10. A computer readable storage medium storing computer instructions for causing a computer to perform the reconstruction method of any one of claims 1-8.

Images