Dynamic Scene Reconstruction Method and Device Based on Multi-Scale Gaussian Sphere

Abstract

The present invention discloses a dynamic scene reconstruction method and device based on multi-scale Gaussian spheres, which relates to the field of computer vision and includes: processing a video frame sequence to be reconstructed by using a structure from motion algorithm to generate a sparse point cloud, initializing the sparse point cloud to generate a 3D Gaussian sphere set; processing the 3D Gaussian sphere set by using a dual-domain deformation model and an adaptive timestamp to obtain a deformed 3D Gaussian sphere set; performing multi-scale Gaussian processing on the deformed 3D Gaussian sphere set to generate a multi-scale Gaussian sphere set; performing Gaussian screening based on pixel coverage rate on the multi-scale Gaussian sphere set to obtain an optimized multi-scale Gaussian sphere set; and performing Alpha blending processing based on the optimized multi-scale Gaussian sphere set to reconstruct an anti-aliased dynamic rendering scene image. The present invention solves the problems of large computational overhead and aliasing effect in current dynamic scene reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a dynamic scene reconstruction method and device based on a multi-scale Gaussian sphere. Background Art

[0002] The 3D reconstruction and real-time rendering of dynamic scenes are core challenges in the field of computer vision and graphics, especially in application scenarios such as virtual reality, autonomous driving, and film and television special effects. Traditional dynamic scene reconstruction technology mainly relies on representation methods based on voxels, point clouds, or grids, combined with motion estimation methods such as optical flow and non-rigid registration to achieve dynamic modeling. However, such methods generally have problems such as high computational complexity, difficulty in handling rapid non-rigid deformations, and aliasing effects caused by loss of high-frequency details.

[0003] The recently proposed 3D Gaussian Splatting technology can achieve real-time rendering speed by parameterizing scenes with explicit Gaussian primitives. However, its native framework has a major problem in dynamic scenes. When directly expanded to a time series model, the exponential growth of the number of Gaussian primitives leads to a sharp increase in video memory usage and computational overhead. On the other hand, existing anti-aliasing solutions mostly use pre-filtering or post-processing super-resolution technology, which cannot achieve frequency domain adaptive detail preservation in the reconstruction stage and will introduce additional computational load.

[0004] Therefore, how to reduce the computational overhead of 3D Gaussian splashing in dynamic scene reconstruction and how to suppress the aliasing effect have become urgent issues to be solved in this field. Summary of the invention

[0005] The purpose of this application is to propose a dynamic scene reconstruction method and device based on a multi-scale Gaussian sphere to address the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a method for reconstructing a dynamic scene based on a multi-scale Gaussian sphere, comprising the following steps:

[0007] Obtain a video frame sequence to be reconstructed, use a structure-from-motion algorithm to process the video frame sequence to be reconstructed, generate a sparse point cloud, initialize the sparse point cloud, and generate a 3D Gaussian sphere set;

[0008] The 3D Gaussian sphere set is processed by using a dual-domain deformation model and an adaptive time stamp to obtain a deformed 3D Gaussian sphere set;

[0009] Perform multi-scale Gaussian processing on the deformed 3D Gaussian sphere set to generate a multi-scale Gaussian sphere set; perform Gaussian screening based on pixel coverage on the multi-scale Gaussian sphere set to obtain an optimized multi-scale Gaussian sphere set;

[0010] Project the optimized multi-scale Gaussian sphere set onto a two-dimensional plane to generate the corresponding 2D Gaussian distribution on the two-dimensional plane, and perform Alpha blending on the optimized multi-scale Gaussian sphere set contained in a single pixel point of the 2D Gaussian on the two-dimensional plane determined by the 2D Gaussian distribution on the two-dimensional plane, and reconstruct the anti-aliased dynamic rendering scene image.

[0011] Preferably, use the structure from motion algorithm to process the video frame sequence to be reconstructed, generate a sparse point cloud, and initialize the sparse point cloud to generate a 3D Gaussian sphere set, specifically including:

[0012] Input the video frame sequence to be reconstructed and the known camera parameters into the structure from motion algorithm, and output a set of sparse point clouds P cloud with their corresponding camera poses;

[0013] For each point in the sparse point cloud P cloud initialize it as a 3D Gaussian sphere G, and each 3D Gaussian sphere is defined as:

[0014] ;

[0015] where G(x) represents the function corresponding to the 3D Gaussian sphere, x represents a three-dimensional space point in the 3D Gaussian sphere, T represents the transpose, is the center position of the 3D Gaussian sphere, is the covariance matrix of the 3D Gaussian sphere, and the covariance matrix of the 3D Gaussian sphere can be parameterized by the scaling vector s and the rotation quaternion q, expressed as: ; is the scaling matrix transformed by the scaling vector s, and R is the rotation matrix transformed by the rotation quaternion q;

[0016] Finally, construct the 3D Gaussian sphere set , representing the i-th 3D Gaussian sphere.

[0017] Preferably, use the dual-domain deformation model and the adaptive timestamp to process the 3D Gaussian sphere set to obtain a deformed 3D Gaussian sphere set, specifically including:

[0018] Introduce an adaptive timestamp to scale the normalized frame index time of each 3D Gaussian sphere in the 3D Gaussian sphere set to obtain the scaled time, as shown in the following formula:

[0019] ;

[0020] where, represents the scaled time, represents the normalized frame index time, and respectively represent the time scaling factor and the basic factor of the 3D Gaussian sphere;

[0021] Based on the time-varying residual after the input is scaled in time and the dual-domain deformation model, the basic properties of each 3D Gaussian sphere at the reference time are dynamically modeled to obtain the dynamic properties deformed with the scaled time, as shown in the following formula:

[0022] ;

[0023] where, S(t s ) is the dynamic property of the 3D Gaussian sphere deformed with the scaled time , S 0 is the basic property of the 3D Gaussian sphere at the reference time t 0 , and the properties include the center position, rotation information, and color information. The time-varying residual D(t s ) after the input is scaled in time is jointly fitted by a time-domain polynomial and a frequency-domain Fourier series, as shown in the following formula:

[0024] ;

[0025] where, is an Nth-order polynomial, n represents the nth order in the polynomial, is the polynomial coefficient of the nth order, is an Lth-order Fourier series, l represents the lth order in the Fourier series, , are the sine Fourier coefficient and cosine Fourier coefficient of the lth order;

[0026] The deformed 3D Gaussian sphere is obtained from the dynamic property of the 3D Gaussian sphere deformed with the scaled time , and a set of deformed 3D Gaussian spheres is constructed, representing the ith deformed 3D Gaussian sphere.

[0027] Preferably, multi-scale Gaussian processing is performed on the set of deformed 3D Gaussian spheres to generate a set of multi-scale Gaussian spheres, specifically including:

[0028] Each deformed 3D Gaussian sphere in the set of deformed 3D Gaussian spheres is projected into a 2D Gaussian corresponding to the deformed 3D Gaussian sphere , and the pixel coverage rate of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere in the screen space is calculated, as shown in the following formula:

[0029] ;

[0030] ;

[0031] ;

[0032] Among them, μ k represents the center position of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere, V k represents the covariance matrix of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere, u represents the horizontal axis length of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere in the plane, v represents the vertical axis length of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere in the plane, represents the opacity of the deformed 3D Gaussian sphere, and respectively represent the abscissa and ordinate components of the projection of the deformed 3D Gaussian sphere onto the two-dimensional plane, represents the preset opacity threshold of the Gaussian, represents taking the minimum of the two, S k represents the pixel coverage rate of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere in the screen space;

[0033] Filter out the deformed 3D Gaussian spheres with pixel coverage rates less than the pixel coverage rate threshold from the set of deformed 3D Gaussian spheres, obtain the filtered Gaussian spheres, and construct a set of filtered Gaussian spheres;

[0034] Divide all the filtered Gaussian spheres in the set of filtered Gaussian spheres into voxel grids according to their spatial positions. Multiple filtered Gaussian spheres within each voxel of the voxel grid generate an aggregated Gaussian sphere through average pooling, as shown in the following formula:

[0035] ;

[0036] ;

[0037] ;

[0038] Among them, G m is the set of filtered Gaussian spheres within the m-th voxel, represents m the number of filtered Gaussian spheres in G , and are respectively the center position, scaling information, and color information of the filtered Gaussian spheres within G m , , and are respectively the center position, scaling information, and color information of the aggregated Gaussian sphere, represents the pixel coverage rate threshold, is the average pixel coverage rate of the filtered Gaussian sphere set in each voxel;

[0039] Take the aggregated Gaussian spheres or the deformed 3D Gaussian spheres with pixel coverage rate greater than or equal to the pixel coverage rate threshold as multi-scale Gaussian spheres, and construct a multi-scale Gaussian sphere set . Preferably, perform Gaussian screening based on pixel coverage rate on the multi-scale Gaussian sphere set to obtain an optimized multi-scale Gaussian sphere set, specifically including:

[0040] Filter out the multi-scale Gaussian spheres that meet the conditions from the multi-scale Gaussian sphere set according to the current rendering resolution. The filtering conditions are:

[0041] ;

[0042] where, represents the multi-scale Gaussian sphere set the pixel coverage rate of a single multi-scale Gaussian sphere in represents the pixel coverage rate threshold, and are the maximum pixel coverage rate and the minimum pixel coverage rate of the multi-scale Gaussian spheres in the multi-scale Gaussian sphere set at different rendering resolutions respectively, is the relative maximum pixel coverage rate threshold, is the relative minimum pixel coverage rate threshold, and represent "AND" and "OR" respectively;

[0043] Use the filtering conditions to filter out the optimized multi-scale Gaussian spheres suitable for rendering at the current resolution and construct an optimized multi-scale Gaussian sphere set.

[0044] Preferably, project the optimized multi-scale Gaussian sphere set onto a two-dimensional plane to generate the distribution of the corresponding 2D Gaussian on the two-dimensional plane, and perform Alpha blending on the optimized multi-scale Gaussian sphere set included in a single pixel point of the 2D Gaussian determined by the distribution of the 2D Gaussian on the two-dimensional plane to reconstruct an anti-aliased dynamic rendering scene image, specifically including:

[0045] Perform 3D to 2D projection on each optimized multi-scale Gaussian sphere in the optimized multi-scale Gaussian sphere set through the projection formula to obtain the center position and covariance matrix of the 2D Gaussian corresponding to the optimized multi-scale Gaussian sphere, and determine the distribution of the 2D Gaussian corresponding to the optimized multi-scale Gaussian sphere on the two-dimensional plane, as shown in the following formula:

[0046] ;

[0047] ;

[0048] Among them, is the center position of the 2D Gaussian corresponding to the optimized multi-scale Gaussian sphere, is the covariance matrix of the 2D Gaussian corresponding to the optimized multi-scale Gaussian sphere, P is the projection matrix, W is the view transformation matrix, and J is the Jacobian matrix of the affine approximation, represents the center position of a single optimized multi-scale Gaussian sphere in the set of optimized multi-scale Gaussian spheres, represents the covariance matrix of a single optimized multi-scale Gaussian sphere in the set of optimized multi-scale Gaussian spheres, and T represents the transpose;

[0049] Determine the set of optimized multi-scale Gaussian spheres contained in a single pixel point of the 2D Gaussian on the two-dimensional plane according to the distribution of the 2D Gaussian corresponding to the optimized multi-scale Gaussian sphere on the two-dimensional plane;

[0050] Input the color and opacity of each optimized multi-scale Gaussian sphere in the set of optimized multi-scale Gaussian spheres contained in a single pixel point of the 2D Gaussian on the two-dimensional plane into the Alpha blending formula to obtain the final color information of a single pixel point of the 2D Gaussian on the two-dimensional plane, as shown in the following formula:

[0051] ;

[0052] Among them, C is the final color information of a single pixel point, is the color of the p-th optimized multi-scale Gaussian sphere in the set of optimized multi-scale Gaussian spheres contained in a single pixel point of the 2D Gaussian on the two-dimensional plane, is the opacity of the p-th optimized multi-scale Gaussian sphere in the set of optimized multi-scale Gaussian spheres contained in a single pixel point of the 2D Gaussian on the two-dimensional plane, b is the number of optimized multi-scale Gaussian spheres in the set of optimized multi-scale Gaussian spheres contained in a single pixel point of the 2D Gaussian on the two-dimensional plane, represents the cumulative transmittance of all the optimized multi-scale Gaussian spheres before the p-th optimized multi-scale Gaussian sphere in the set of optimized multi-scale Gaussian spheres contained in a single pixel point of the 2D Gaussian on the two-dimensional plane, j is the index of all the optimized multi-scale Gaussian spheres before the p-th optimized multi-scale Gaussian sphere in the set of optimized multi-scale Gaussian spheres contained in a single pixel point of the 2D Gaussian on the two-dimensional plane, is the opacity of the j-th optimized multi-scale Gaussian sphere in the set of optimized multi-scale Gaussian spheres contained in a single pixel point of the 2D Gaussian on the two-dimensional plane;

[0053] Construct the anti-aliased dynamic rendering scene image through the final color information of all pixel points.

[0054] In a second aspect, the present invention provides a dynamic scene reconstruction device based on multi-scale Gaussian spheres, comprising:

[0055] An initialization module, configured to obtain a video frame sequence to be reconstructed, process the video frame sequence to be reconstructed by using a structure from motion algorithm to generate a sparse point cloud, and initialize the sparse point cloud to generate a 3D Gaussian sphere set;

[0056] A deformation module, configured to process the 3D Gaussian sphere set by using a dual-domain deformation model and an adaptive timestamp to obtain a deformed 3D Gaussian sphere set;

[0057] A multi-scale Gaussian module, configured to perform multi-scale Gaussian processing on the deformed 3D Gaussian sphere set to generate a multi-scale Gaussian sphere set; perform Gaussian screening based on pixel coverage rate on the multi-scale Gaussian sphere set to obtain an optimized multi-scale Gaussian sphere set;

[0058] A reconstruction module, configured to project the optimized multi-scale Gaussian sphere set onto a two-dimensional plane to generate a corresponding 2D Gaussian distribution on the two-dimensional plane, and perform Alpha blending on the optimized multi-scale Gaussian sphere set included in a single pixel point of the 2D Gaussian on the two-dimensional plane determined by the 2D Gaussian distribution on the two-dimensional plane to reconstruct an anti-aliased dynamic rendering scene image.

[0059] In a third aspect, the present invention provides an electronic device, comprising one or more processors; a storage device for storing one or more programs, which when executed by the one or more processors cause the one or more processors to implement the method described in any implementation manner of the first aspect.

[0060] In a fourth aspect, the present invention provides a computer-readable storage medium, having stored thereon a computer program, which when executed by a processor implements the method described in any implementation manner of the first aspect.

[0061] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which when executed by a processor implements the method described in any implementation manner of the first aspect.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] (1) The dynamic scene reconstruction method based on multi-scale Gaussian spheres proposed by the present invention uses a dual-domain deformation module to model the attributes (such as position, rotation, and color) of each Gaussian sphere simultaneously in the time domain and the frequency domain, thereby reducing the computational overhead. And through the combination of polynomials and Fourier series, the dual-domain deformation model can capture complex motion trajectories without adding too much computational complexity.

[0064] (2) The dynamic scene reconstruction method based on multi-scale Gaussian spheres proposed by the present invention performs multi-scale Gaussian processing to select an optimized set of multi-scale Gaussian spheres that are most suitable for the current rendering resolution for rendering, thereby avoiding too many Gaussian spheres in a pixel, so that the color information of the pixel is only determined by the color information of the Gaussian spheres ranked in the front, resulting in an aliasing effect, and enabling a relatively good reconstruction effect to be shown at different rendering resolutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0066] Figure 1 It is a schematic flowchart of the dynamic scene reconstruction method based on multi-scale Gaussian spheres according to the embodiment of the present application;

[0067] Figure 2 It is a schematic diagram of the dynamic scene reconstruction device based on multi-scale Gaussian spheres according to the embodiment of the present application;

[0068] Figure 3 It is a schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0070] Figure 1 The following shows a dynamic scene reconstruction method based on multi-scale Gaussian spheres provided by the embodiment of the present application, including the following steps:

[0071] S1. Obtain the video frame sequence to be reconstructed, process the video frame sequence to be reconstructed using the Structure from Motion (SFM) algorithm to generate a sparse point cloud, and initialize the sparse point cloud to generate a 3D Gaussian sphere set.

[0072] In a specific embodiment, processing the video frame sequence to be reconstructed using the Structure from Motion (SFM) algorithm to generate a sparse point cloud, and initializing the sparse point cloud to generate a 3D Gaussian sphere set specifically includes:

[0073] Input the video frame sequence to be reconstructed and the known camera parameters into the Structure from Motion (SFM) algorithm, and output a set of sparse point cloud P cloud with position information and color information,

[0074] and its corresponding camera poses; cloud Initialize each point in the sparse point cloud P

[0075] ;

[0076] where G(x) represents the function corresponding to the 3D Gaussian sphere, x represents a three-dimensional space point in the 3D Gaussian sphere, T represents the transpose, is the center position of the 3D Gaussian sphere, is the covariance matrix of the 3D Gaussian sphere, and the covariance matrix of the 3D Gaussian sphere can be parameterized by the scaling vector s and the rotation quaternion q, expressed as: ; is the scaling matrix transformed by the scaling vector s, and R is the rotation matrix transformed by the rotation quaternion q;

[0077] Finally, construct a 3D Gaussian sphere set where represents the i-th 3D Gaussian sphere.

[0078] Specifically, input the video frame sequence and the known camera parameters into the Structure from Motion (SFM) algorithm, output a set of sparse point cloud P cloud with position information and color information and the corresponding camera poses, initialize the sparse point cloud P cloud to generate a 3D Gaussian sphere set, and initialize each point in the sparse point cloud P cloud to a 3D Gaussian sphere. The function corresponding to the 3D Gaussian sphere is used to describe the probability density of the point x in the Gaussian distribution; x in the function corresponding to the 3D Gaussian sphere is used to describe the position of a single point in the above-mentioned sparse point cloud P cloud ; the rotation quaternion q is a mathematical tool used to represent three-dimensional rotation, and the rotation quaternion can be expressed as , where w q ​represents half of the cosine value of the rotation angle, and x q represents the x-axis component of the rotation axis, y q represents the y-axis component of the rotation axis, z q represents the z-axis component of the rotation axis, w q is the real part, is the imaginary part. The formula for converting the rotation quaternion q to the rotation matrix R is as follows:

[0079] .

[0080] In step S2, a dual-domain deformation model and an adaptive timestamp are used to process the 3D Gaussian sphere set, and a deformed 3D Gaussian sphere set is obtained.

[0081] In a specific embodiment, step S2 specifically includes:

[0082] An adaptive timestamp is introduced to scale the normalized frame index time of each 3D Gaussian sphere in the 3D Gaussian sphere set, and the scaled time is obtained as shown in the following formula:

[0083] ;

[0084] where, represents the scaled time, represents the normalized frame index time, and respectively represent the time scaling factor and the basic factor of the 3D Gaussian sphere;

[0085] Based on the time-varying residual after the input of the scaled time and the dual-domain deformation model, the basic attributes of each 3D Gaussian sphere at the reference time are dynamically modeled, and the deformed dynamic attributes after the scaled time are obtained as shown in the following formula:

[0086] ;

[0087] where S(t s ) is the deformed dynamic attribute of the 3D Gaussian sphere after the scaled time , S 0 is the basic attribute of the 3D Gaussian sphere at the reference time t 0 , and the attributes include the center position, rotation information, and color information. The time-varying residual D(t s ) after the input of the scaled time is jointly fitted by a time-domain polynomial and a frequency-domain Fourier series as shown in the following formula:

[0088] ;

[0089] where, is an Nth-order polynomial, and n represents the nth order in the polynomial, are the polynomial coefficients of the nth order, is the L-order Fourier series, l represents the l-th order in the Fourier series, , are the sine Fourier coefficients and cosine Fourier coefficients of the lth order;

[0090] Through the 3D Gaussian sphere with time after scaling The dynamic properties after deformation get the deformed 3D Gaussian sphere , and construct a deformed 3D Gaussian sphere set , represents the i-th deformed 3D Gaussian sphere.

[0091] Specifically, in the embodiment of the present application, a dual-domain deformation model is used to transform each 3D Gaussian sphere The basic properties of S 0 (Center location , rotation information q 0 、Color informationc 0 ) is used to model dynamic changes. The dual-domain deformation model decomposes motion into low-frequency trend terms and high-frequency detail terms. As a low-frequency trend term, the time domain polynomial uses a low-order polynomial to characterize the overall trajectory of the object's motion, while as a high-frequency detail term, the frequency domain Fourier series captures periodic or non-rigid deformations through the superposition of fundamental frequency and harmonics. This frequency-division modeling method significantly improves the expressiveness of complex motion trajectories and reduces computational overhead compared to the original sampling of each video frame. At the same time, in order to balance the overfitting problem of violent motion, an adaptive timestamp is introduced to scale the time input of each 3D Gaussian sphere.

[0092] S3, performing multi-scale Gaussian processing on the deformed 3D Gaussian sphere set to generate a multi-scale Gaussian sphere set; performing Gaussian screening based on pixel coverage on the multi-scale Gaussian sphere set to obtain an optimized multi-scale Gaussian sphere set.

[0093] In a specific embodiment, performing multi-scale Gaussian processing on the deformed 3D Gaussian sphere set to generate a multi-scale Gaussian sphere set specifically includes:

[0094] Project each deformed 3D Gaussian sphere in the deformed 3D Gaussian sphere set into the 2D Gaussian corresponding to the deformed 3D Gaussian sphere , and calculate the pixel coverage of the 2D Gaussian corresponding to the deformed 3D Gaussian ball in the screen space, as shown in the following formula:

[0095] ;

[0096] ;

[0097] ;

[0098] Among them, μ k represents the center position of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere, V k represents the covariance matrix of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere, u represents the horizontal axis length of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere in the plane, and v represents the vertical axis length of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere in the plane. represents the opacity of the deformed 3D Gaussian sphere, and respectively represent the abscissa and ordinate components of the projection of the deformed 3D Gaussian sphere onto the two-dimensional plane. represents the preset opacity threshold of the Gaussian, represents taking the minimum of the two, S k represents the pixel coverage rate of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere in the screen space;

[0099] Filter out the deformed 3D Gaussian spheres with pixel coverage rates less than the pixel coverage rate threshold in the set of deformed 3D Gaussian spheres, obtain the filtered Gaussian spheres, and construct a set of filtered Gaussian spheres;

[0100] Divide all the filtered Gaussian spheres in the set of filtered Gaussian spheres into voxel grids according to their spatial positions. Multiple filtered Gaussian spheres in each voxel of the voxel grid generate an aggregated Gaussian sphere through average pooling, as shown in the following formula:

[0101] ;

[0102] ;

[0103] ;

[0104] Among them, G m is the set of filtered Gaussian spheres in the m-th voxel, represents m the number of filtered Gaussian spheres in G , and are respectively the center position, scaling information, and color information of the filtered Gaussian spheres in G m , , and are respectively the center position, scaling information, and color information of the aggregated Gaussian sphere, represents the pixel coverage rate threshold, is the average pixel coverage rate of the set of filtered Gaussian spheres in each voxel;

[0105] The aggregated Gaussian spheres or the deformed 3D Gaussian spheres with a pixel coverage rate greater than or equal to the pixel coverage rate threshold are used as multi-scale Gaussian spheres, and a multi-scale Gaussian sphere set is constructed. . Specifically, in one embodiment, the preset opacity threshold of the Gaussian . In other embodiments, other values can be selected. The horizontal axis length u and the vertical axis length v are calculated through the above formula, and the smaller of the two is the pixel coverage rate S. k .

[0106] Set a pixel coverage rate threshold S. T Filter out all the deformed 3D Gaussian spheres that satisfy S k < S T , and denote them as G. small . In one embodiment, the pixel coverage rate threshold S T is set to 2px. In other embodiments, other values can be selected. The filtered Gaussian sphere set is divided into voxel grids according to the spatial position, and multiple small filtered Gaussian spheres in each voxel are used to generate a large aggregated Gaussian sphere through average pooling. The aggregated Gaussian spheres in this part are combined with the original deformed Gaussian spheres with a pixel coverage rate greater than or equal to the pixel coverage rate threshold S T to form a multi-scale Gaussian sphere set. Through the above steps, the original Gaussian sphere set is transformed into a multi-scale Gaussian sphere set, so as to facilitate the selection of suitable multi-scale Gaussian spheres for rendering according to the real-time rendering resolution in the subsequent steps.

[0107] In a specific embodiment, Gaussian screening based on the pixel coverage rate is performed on the multi-scale Gaussian sphere set to obtain an optimized multi-scale Gaussian sphere set, which specifically includes:

[0108] Select the multi-scale Gaussian spheres that meet the conditions from the multi-scale Gaussian sphere set according to the current rendering resolution. The selection conditions are:

[0109] ;

[0110] where, represents the pixel coverage rate of a single multi-scale Gaussian sphere in the multi-scale Gaussian sphere set , represents the pixel coverage rate threshold, and respectively represent the maximum pixel coverage rate and the minimum pixel coverage rate of the multi-scale Gaussian spheres in the multi-scale Gaussian sphere set under different rendering resolutions, is the relative maximum pixel coverage rate threshold, is the relative minimum pixel coverage rate threshold, and respectively represent "AND" and "OR";

[0111] Use the screening conditions to screen out the optimized multi-scale Gaussian spheres suitable for rendering at the current resolution, and construct an optimized multi-scale Gaussian sphere set.

[0112] Specifically, in one embodiment, the relative maximum pixel coverage threshold is set to 1.5, and the relative minimum pixel coverage threshold is set to 0.5. Other values can be selected in other embodiments. Using the above screening conditions, an optimized multi-scale Gaussian sphere set suitable for rendering at the current rendering resolution can be screened out.

[0113] S4. Project the optimized multi-scale Gaussian sphere set onto a two-dimensional plane to generate the distribution of the corresponding 2D Gaussian on the two-dimensional plane, and perform Alpha blending on the optimized multi-scale Gaussian sphere set included in a single pixel point of the 2D Gaussian determined by the distribution of the 2D Gaussian on the two-dimensional plane to reconstruct an anti-aliased dynamic rendering scene image.

[0114] In a specific embodiment, step S4 specifically includes:

[0115] Perform 3D to 2D projection on each optimized multi-scale Gaussian sphere in the optimized multi-scale Gaussian sphere set through the projection formula to obtain the center position and covariance matrix of the corresponding 2D Gaussian of the optimized multi-scale Gaussian sphere, and determine the distribution of the corresponding 2D Gaussian of the optimized multi-scale Gaussian sphere on the two-dimensional plane, as shown in the following formula:

[0116] ;

[0117] ;

[0118] Among them, is the center position of the 2D Gaussian corresponding to the optimized multi-scale Gaussian sphere, is the covariance matrix of the 2D Gaussian corresponding to the optimized multi-scale Gaussian sphere, P is the projection matrix, W is the view transformation matrix, J is the Jacobian matrix of the affine approximation, represents the center position of a single optimized multi-scale Gaussian sphere in the optimized multi-scale Gaussian sphere set, represents the covariance matrix of a single optimized multi-scale Gaussian sphere in the optimized multi-scale Gaussian sphere set, and T represents the transpose;

[0119] Determine the optimized multi-scale Gaussian sphere set included in a single pixel point of the 2D Gaussian on the two-dimensional plane according to the distribution of the corresponding 2D Gaussian of the optimized multi-scale Gaussian sphere on the two-dimensional plane;

[0120] The color and opacity of each optimized multi-scale Gaussian sphere in the optimized multi-scale Gaussian sphere set contained in a single pixel of the 2D Gaussian on the two-dimensional plane are input into the Alpha blending formula to obtain the final color information of the single pixel of the 2D Gaussian on the two-dimensional plane, as shown in the following formula:

[0121] ;

[0122] where C is the final color information of a single pixel, is the color of the p-th optimized multi-scale Gaussian sphere in the optimized multi-scale Gaussian sphere set contained in a single pixel of the 2D Gaussian on the two-dimensional plane, is the opacity of the p-th optimized multi-scale Gaussian sphere in the optimized multi-scale Gaussian sphere set contained in a single pixel of the 2D Gaussian on the two-dimensional plane, b is the number of optimized multi-scale Gaussian spheres in the optimized multi-scale Gaussian sphere set contained in a single pixel of the 2D Gaussian on the two-dimensional plane, represents the cumulative light transmittance of all optimized multi-scale Gaussian spheres before the p-th optimized multi-scale Gaussian sphere in the optimized multi-scale Gaussian sphere set contained in a single pixel of the 2D Gaussian on the two-dimensional plane, j is the index of all optimized multi-scale Gaussian spheres before the p-th optimized multi-scale Gaussian sphere in the optimized multi-scale Gaussian sphere set contained in a single pixel of the 2D Gaussian on the two-dimensional plane, is the opacity of the j-th optimized multi-scale Gaussian sphere in the optimized multi-scale Gaussian sphere set contained in a single pixel of the 2D Gaussian on the two-dimensional plane;

[0123] An anti-aliased dynamic rendering scene image is constructed through the final color information of all pixels.

[0124] Specifically, the projection matrix, view transformation matrix, and Jacobian matrix of the affine approximation used in the process of projecting the optimized multi-scale Gaussian sphere set are determined by the internal and external camera parameters, and the internal and external camera parameters include the camera pose determined in step S1. Since the color of the final single pixel needs to be on the two-dimensional plane, it is necessary to first project the optimized multi-scale Gaussian sphere set onto the two-dimensional plane to determine the center position and covariance matrix of the optimized multi-scale Gaussian sphere set projected onto the two-dimensional plane. These two parameters determine the position and shape of the 2D Gaussian, so as to determine the distribution of the 2D Gaussian on the two-dimensional plane, and then the optimized multi-scale Gaussian sphere set contained in a single pixel of the 2D Gaussian on the two-dimensional plane can be determined. The final color information of a single pixel of the 2D Gaussian on the two-dimensional plane is calculated through the optimized multi-scale Gaussian sphere set contained in a single pixel of the 2D Gaussian on the two-dimensional plane.

[0125] The position of a single pixel in the two-dimensional plane is fixed, and each pixel has color information. The final reconstructed anti-aliased dynamic rendering scene image needs to be constructed from the colors of all pixels. Through Alpha blending, the colors of the optimized multi-scale Gaussian spheres within a pixel can be superimposed to obtain the final color information of the pixel.

[0126] For further reference Figure 2 As an implementation of the methods shown in the above figures, an embodiment of a dynamic scene reconstruction device based on multi-scale Gaussian spheres is provided in this application. This device embodiment corresponds to Figure 1 the method embodiment shown, and this device can be specifically applied to various electronic devices.

[0127] An embodiment of a dynamic scene reconstruction device based on multi-scale Gaussian spheres is provided in this application embodiment, including:

[0128] Initialization module 1, configured to obtain a video frame sequence to be reconstructed, process the video frame sequence to be reconstructed using a structure from motion algorithm, generate a sparse point cloud, and initialize the sparse point cloud to generate a set of 3D Gaussian spheres;

[0129] Deformation module 2, configured to process the set of 3D Gaussian spheres using a dual-domain deformation model and an adaptive timestamp to obtain a deformed set of 3D Gaussian spheres;

[0130] Multi-scale Gaussian module 3, configured to perform multi-scale Gaussian processing on the deformed set of 3D Gaussian spheres to generate a set of multi-scale Gaussian spheres; perform Gaussian screening based on pixel coverage on the set of multi-scale Gaussian spheres to obtain an optimized set of multi-scale Gaussian spheres;

[0131] Reconstruction module 4, configured to project the optimized set of multi-scale Gaussian spheres onto a two-dimensional plane to generate the corresponding distribution of 2D Gaussians on the two-dimensional plane, and perform Alpha blending on the optimized set of multi-scale Gaussian spheres included in a single pixel point determined by the distribution of 2D Gaussians on the two-dimensional plane to reconstruct an anti-aliased dynamic rendering scene image.

[0132] Figure 3 is a schematic hardware structure diagram of an electronic device provided in an embodiment of the present invention. As Figure 3 shown, the electronic device in this embodiment includes: a processor 301 and a memory 302; wherein the memory 302 is used to store computer execution instructions; the processor 301 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. Specifically, reference can be made to the relevant descriptions in the foregoing method embodiment.

[0133] Optionally, the memory 302 can be either independent or integrated with the processor 301.

[0134] When the memory 302 is independently provided, the electronic device further includes a bus 303 for connecting the memory 302 and the processor 301.

[0135] An embodiment of the present invention further provides a computer storage medium, in which computer-executable instructions are stored. When the processor 301 executes the computer-executable instructions, the above method is implemented.

[0136] An embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 301, the above method is implemented.

[0137] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or modules can be in electrical, mechanical or other forms.

[0138] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to implement the solution of this embodiment.

[0139] In addition, in each embodiment of the present invention, the various functional modules can be integrated in a processing unit, or each module can exist physically alone, or two or more modules can be integrated in one unit. The units formed by the above modules can be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0140] The above integrated modules implemented in the form of software functional modules can be stored in a computer-readable storage medium. The above software functional modules are stored in a storage medium, including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or the processor 301 to execute some steps of the methods in various embodiments of the present application.

[0141] It should be understood that the above-mentioned processor 301 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), etc. The general-purpose processor may be a microprocessor or the processor 301 may also be any conventional processor 301, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being executed and completed by the hardware processor 301, or by a combination of the hardware and software modules in the processor 301.

[0142] The memory 302 may include high-speed RAM memory and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disc, etc.

[0143] The bus 303 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus 303 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, the bus 303 in the drawings of this application is not limited to only one bus 303 or one type of bus 303.

[0144] The above-mentioned storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0145] An exemplary storage medium is coupled to the processor 301, enabling the processor 301 to read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 301. The processor 301 and the storage medium can be located in an Application Specific Integrated Circuits (ASIC). Of course, the processor 301 and the storage medium can also exist as discrete components in an electronic device or a master device.

[0146] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dynamic scene reconstruction method based on multi-scale Gaussian sphere, characterized in that: The following steps are involved: Acquire a video frame sequence to be reconstructed, process the video frame sequence to be reconstructed using a structure-from-motion algorithm to generate a sparse point cloud, initialize the sparse point cloud, and generate a 3D Gaussian sphere set; The 3D Gaussian sphere set is processed using a dual-domain deformation model and an adaptive timestamp to obtain a deformed 3D Gaussian sphere set, specifically including: An adaptive timestamp is introduced to scale the normalized frame index time of each 3D Gaussian sphere in the 3D Gaussian sphere set to obtain the scaled time, as shown in the following formula: t s =λ s t+λ b ; Among them, t s represents the scaled time, t∈[0,1] represents the normalized frame index time, λ s and λ b They represent the time scaling factor and basic factor of the 3D Gaussian sphere respectively; Based on the time-varying residual after inputting the scaled time and the dual-domain deformation model, the basic properties of each 3D Gaussian sphere at the reference time are dynamically modeled to obtain the dynamic properties deformed with the scaled time, as shown in the following formula: S(t s )=S0+D(t s ); Among them, S(t s ) is the time t of the 3D Gaussian ball after the scaling s The dynamic properties after deformation, S0 is the basic properties of the 3D Gaussian ball at the reference time t0, including the center position, rotation information and color information, and the time-varying residual D(t s ) is fitted by the time domain polynomial and the frequency domain Fourier series, as shown in the following formula: D(t s )=P N (t s )+F L (t s ); in, is an N-order polynomial, n represents the nth order in the polynomial, are the polynomial coefficients of the nth order, is the L-order Fourier series, l represents the l-th order in the Fourier series, are the sine Fourier coefficients and cosine Fourier coefficients of the lth order; The 3D Gaussian sphere passes through the scaled time t s The deformed dynamic properties are used to obtain the deformed 3D Gaussian sphere G′, and the deformed 3D Gaussian sphere set is constructed. G i ′ represents the i-th deformed 3D Gaussian sphere; Performing multi-scale Gaussian processing on the deformed 3D Gaussian sphere set to generate a multi-scale Gaussian sphere set; performing Gaussian screening based on pixel coverage on the multi-scale Gaussian sphere set to obtain an optimized multi-scale Gaussian sphere set; The optimized multi-scale Gaussian sphere set is projected onto a two-dimensional plane to generate a corresponding 2D Gaussian distribution on the two-dimensional plane, and Alpha blending is performed on the optimized multi-scale Gaussian sphere set contained in a single pixel point of the 2D Gaussian on the two-dimensional plane determined by the distribution of the 2D Gaussian on the two-dimensional plane to reconstruct an anti-aliased dynamic rendering scene image.

2. The dynamic scene reconstruction method based on multi-scale Gaussian sphere according to claim 1, characterized in that: The video frame sequence to be reconstructed is processed by using a structure-from-motion algorithm to generate a sparse point cloud, and the sparse point cloud is initialized to generate a 3D Gaussian sphere set, specifically including: The video frame sequence to be reconstructed and the known camera parameters are input into the motion recovery structure algorithm, and a set of sparse point clouds P with position information and color information is output. cloud and its corresponding camera pose; For the sparse point cloud P cloud Each point in is initialized as a 3D Gaussian ball G, and each 3D Gaussian ball is defined as: Wherein, G(x) represents the function corresponding to the 3D Gaussian sphere, x represents a three-dimensional space point in the 3D Gaussian sphere, T represents transposition, μ is the center position of the 3D Gaussian sphere, ∑ is the covariance matrix of the 3D Gaussian sphere, and the covariance matrix of the 3D Gaussian sphere can be obtained by parameterizing the scaling vector s and the rotation quaternion q, and is expressed as: ∑ = RΛ(s)Λ(s) T R T ; Λ(s) is the scaling matrix converted by scaling vector s, and R is the rotation matrix converted by rotating quaternion q; Finally, a 3D Gaussian sphere set is constructed. G i represents the i-th 3D Gaussian sphere.

3. The dynamic scene reconstruction method based on multi-scale Gaussian sphere according to claim 1, characterized in that: Performing multi-scale Gaussian processing on the deformed 3D Gaussian sphere set to generate a multi-scale Gaussian sphere set specifically includes: Project each deformed 3D Gaussian sphere in the deformed 3D Gaussian sphere set into a 2D Gaussian G corresponding to the deformed 3D Gaussian sphere 2d (μ k ,V k ), and calculate the pixel coverage of the 2D Gaussian corresponding to the deformed 3D Gaussian ball in the screen space, as shown in the following formula: S k =min(u,v); Among them, μ k Indicates the center position of the 2D Gaussian corresponding to the deformed 3D Gaussian ball, V k represents the covariance matrix of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere, u represents the horizontal axis length of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere in the plane, v represents the vertical axis length of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere in the plane, σ k represents the opacity of the deformed 3D Gaussian sphere, and They represent the horizontal and vertical coordinate components of the deformed 3D Gaussian sphere projected onto the two-dimensional plane, σ T Indicates the preset Gaussian opacity threshold, min means taking the minimum value of the two, S k Indicates the pixel coverage of the 2D Gaussian corresponding to the deformed 3D Gaussian sphere in screen space; Screening out deformed 3D Gaussian spheres whose pixel coverage is less than a pixel coverage threshold from the deformed 3D Gaussian sphere set, obtaining screened Gaussian spheres and constructing the screened Gaussian sphere set; All the filtered Gaussian balls in the filtered Gaussian ball set are divided into voxel grids according to spatial positions, and multiple filtered Gaussian balls in each voxel of the voxel grid are averaged and pooled to generate an aggregated Gaussian ball, as shown in the following formula: Among them, G m is the set of Gaussian balls after screening in the mth voxel, |G m | indicates G m The number of Gaussian balls after screening, μ k 、s k and c k G m The center position, scaling information and color information of the filtered Gaussian ball, μ new 、s new and c new They are the center position, scaling information and color information of the aggregated Gaussian sphere, S T represents the pixel coverage threshold, S avg is the average pixel coverage of the set of filtered Gaussian spheres in each voxel; The aggregated Gaussian sphere or the deformed 3D Gaussian sphere whose pixel coverage is greater than or equal to the pixel coverage threshold is used as a multi-scale Gaussian sphere, and a multi-scale Gaussian sphere set is constructed.

4. The dynamic scene reconstruction method based on multi-scale Gaussian sphere according to claim 1, characterized in that: Performing Gaussian screening based on pixel coverage on the multi-scale Gaussian sphere set to obtain an optimized multi-scale Gaussian sphere set specifically includes: According to the current rendering resolution, a multi-scale Gaussian sphere that meets the conditions is screened out from the multi-scale Gaussian sphere set, and the screening conditions are: Among them, S k ′ represents a multi-scale Gaussian ball set The pixel coverage of a single multi-scale Gaussian sphere in S T represents the pixel coverage threshold, and A collection of multi-scale Gaussian spheres at different rendering resolutions The maximum pixel coverage and minimum pixel coverage of the multi-scale Gaussian sphere, is the relative maximum pixel coverage threshold, is the relative minimum pixel coverage threshold, ∧ and ∨ represent "and" and "or" respectively; The screening condition is used to screen out optimized multi-scale Gaussian spheres suitable for rendering at the current resolution and to construct an optimized multi-scale Gaussian sphere set.

5. The dynamic scene reconstruction method based on multi-scale Gaussian sphere according to claim 1, characterized in that: The optimized multi-scale Gaussian ball set is projected onto a two-dimensional plane to generate a corresponding distribution of 2D Gaussians on the two-dimensional plane, and an alpha blending process is performed on the optimized multi-scale Gaussian ball set contained in a single pixel point of a 2D Gaussian on the two-dimensional plane determined by the distribution of the 2D Gaussians on the two-dimensional plane to reconstruct an anti-aliased dynamic rendering scene image, specifically including: Each optimized multi-scale Gaussian ball in the optimized multi-scale Gaussian ball set is projected from 3D to 2D by the projection formula to obtain the center position and covariance matrix of the 2D Gaussian corresponding to the optimized multi-scale Gaussian ball, and determine the distribution of the 2D Gaussian corresponding to the optimized multi-scale Gaussian ball on the two-dimensional plane, as shown in the following formula: m k ′=PWμ multi ; ∑′ k =JW∑ multi W T J T ; Among them, μ k ′ is the center position of the 2D Gaussian corresponding to the optimized multi-scale Gaussian ball, ∑′ k is the covariance matrix of the 2D Gaussian corresponding to the optimized multi-scale Gaussian sphere, P is the projection matrix, W is the view transformation matrix, J is the Jacobian matrix of the affine approximation, μ multi represents the center position of a single optimized multi-scale Gaussian ball in the optimized multi-scale Gaussian ball set, ∑ multi represents the covariance matrix of a single optimized multi-scale Gaussian sphere in the optimized multi-scale Gaussian sphere set, and T represents transpose; Determine the optimized multi-scale Gaussian sphere set contained in a single pixel point of the 2D Gaussian on the two-dimensional plane according to the distribution of the 2D Gaussian corresponding to the optimized multi-scale Gaussian sphere on the two-dimensional plane; The color and opacity of each optimized multi-scale Gaussian ball in the set of optimized multi-scale Gaussian balls contained in a single pixel point of the 2D Gaussian on the two-dimensional plane are input into the Alpha blending formula to obtain the final color information of a single pixel point of the 2D Gaussian on the two-dimensional plane, as shown in the following formula: Among them, C is the final color information of a single pixel, c p is the color of the pth optimized multi-scale Gaussian ball in the set of optimized multi-scale Gaussian balls contained in a single pixel point of the 2D Gaussian on the two-dimensional plane, α p is the opacity of the pth optimized multi-scale Gaussian ball in the set of optimized multi-scale Gaussian balls contained in a single pixel point of the 2D Gaussian on the two-dimensional plane, b is the number of optimized multi-scale Gaussian balls in the set of optimized multi-scale Gaussian balls contained in a single pixel point of the 2D Gaussian on the two-dimensional plane, represents the cumulative transmittance of all optimized multiscale Gaussian balls before the pth optimized multiscale Gaussian ball in the set of optimized multiscale Gaussian balls contained in a single pixel point of the 2D Gaussian on the two-dimensional plane, j is the index of all optimized multiscale Gaussian balls before the pth optimized multiscale Gaussian ball in the set of optimized multiscale Gaussian balls contained in a single pixel point of the 2D Gaussian on the two-dimensional plane, α j is the opacity of the jth optimized multi-scale Gaussian ball in the set of optimized multi-scale Gaussian balls contained in a single pixel point of the 2D Gaussian on the two-dimensional plane; The anti-aliased dynamic rendering scene image is constructed through the final color information of all pixels.

6. A dynamic scene reconstruction device based on a multi-scale Gaussian sphere, using the dynamic scene reconstruction method based on a multi-scale Gaussian sphere according to any one of claims 1 to 5, characterized in that: include: An initialization module is configured to obtain a video frame sequence to be reconstructed, process the video frame sequence to be reconstructed using a structure-from-motion algorithm to generate a sparse point cloud, initialize the sparse point cloud, and generate a 3D Gaussian sphere set; A deformation module is configured to process the 3D Gaussian sphere set using a dual-domain deformation model and an adaptive timestamp to obtain a deformed 3D Gaussian sphere set; The multi-scale Gaussian module is configured to perform multi-scale Gaussian processing on the deformed 3D Gaussian sphere set to generate a multi-scale Gaussian sphere set; perform Gaussian screening based on pixel coverage on the multi-scale Gaussian sphere set to obtain an optimized multi-scale Gaussian sphere set; The reconstruction module is configured to project the optimized multi-scale Gaussian sphere set onto a two-dimensional plane, generate a corresponding 2D Gaussian distribution on the two-dimensional plane, and perform Alpha blending processing on the optimized multi-scale Gaussian sphere set contained in a single pixel point of the 2D Gaussian on the two-dimensional plane determined by the distribution of the 2D Gaussian on the two-dimensional plane, so as to reconstruct an anti-aliased dynamic rendering scene image.

7. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images