4D gaussian reconstruction method and system based on self-heterogeneity

Abstract

The application discloses a 4D Gaussian reconstruction method and system based on self-heterogeneity, and belongs to the field of computer vision processing.The method comprises the following steps: parameterizing input timestamps and observation angles, and combining corresponding pictures as training data; and cutting the Gaussian renderable area in advance before Gaussian rasterization, so as to avoid redundant rasterization calculation.A learnable parameter is set for each Gaussian kernel, and adaptive learning is performed during training of the 4D Gaussian model, so as to realize self-heterogeneous change of the Gaussian kernel.The trained 4D Gaussian model is directly input with time and angle parameters, and the corresponding rendering picture can be obtained.Through the above steps, the method can reconstruct a high-quality dynamic multi-view scene.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision processing, and particularly relates to a 4D Gaussian reconstruction method and system based on self-heterogeneity. BACKGROUND

[0002] Existing dynamic three-dimensional scene reconstruction methods, especially Gaussian Splatting-based techniques, still face challenges in handling temporal dimension changes and maintaining high-quality rendering. The parameters of traditional Gaussian kernels are usually fixed or only undergo simple interpolation, making it difficult to adaptively express complex and non-uniform dynamic deformation and transparency changes in the scene. In addition, redundant rasterization calculations can also affect rendering efficiency.

[0003] Although recent years have seen significant progress in 3D Gaussian Splatting (3DGS) methods based on deformation field driving in 4D reconstruction tasks, a core problem still exists in practical applications: severe motion blur in dynamic content. Especially when rendering from new perspectives, the edges of objects in the scene often appear blurred and trailing, disrupting spatiotemporal consistency. This blur not only reduces visual quality but also limits the practical application value of related methods in high-quality 4D scene modeling. This problem is widespread in multiple public datasets and real data, showing that it is not an isolated phenomenon but a systematic flaw common to current mainstream methods.

[0004] Most current work attributes the motion blur problem to the quality of the deformation field itself, focusing on improving the expression ability of the deformation field, such as designing more complex motion models and introducing temporal attention mechanisms. These methods do improve the inter-frame alignment to some extent, but research has found that they generally ignore the limitations of the representation ability of 3DGS. Specifically, existing methods rely too much on external motion models when dealing with time-varying scenes, without delving into the expression conflict in the appearance-motion joint modeling of 3DGS itself under dynamic scenes. This "headache medicine" strategy is difficult to solve the dynamic blur problem from the root.

[0005] Through in-depth analysis, it can be found that the key of the problem lies in the alpha blending mechanism adopted by the 3DGS. In the traditional 3DGS, the rendering of each Gaussian on the image plane is a "template type" structure with high density in the center and gradually transparent at the edge. The original design intention of this structure is to realize the smooth transition of color and the high-fidelity reconstruction of appearance through the overlap between Gaussians in the static scene. However, in the dynamic scene, each Gaussian needs to complete two conflicting tasks at the same time: the center area is used to fit the local high-frequency motion or texture boundary, and the edge area is responsible for the low-frequency smooth transition across time frames. This dual role causes serious gradient conflict in the optimization process, leading to the deviation of the motion direction prediction of the Gaussian kernel from the real trajectory, and finally presenting obvious motion blur in the new view. It can be said that the space-frequency coupling in the alpha blending mechanism is the structural bottleneck of the current 3DGS applied to dynamic scenes. SUMMARY

[0006] The purpose of the present application is to solve the problems of quality, efficiency and adaptability of dynamic Gaussian scene reconstruction in the prior art, and to provide a 4D Gaussian reconstruction method and system based on self-heterogeneity. The method realizes the reconstruction of high-quality dynamic multi-view scenes by introducing a learnable Alpha-self-heterogeneity parameter and combining an efficient rasterization clipping strategy.

[0007] In order to achieve the above-mentioned application purpose, the present application specifically adopts the following technical solutions:

[0008] In a first aspect, the present application provides a 4D Gaussian reconstruction method based on self-heterogeneity, which includes the following steps:

[0009] S1: accurately parameterize the input original timestamp and original observation view angle respectively, combine the processed timestamp, the processed observation view angle with the original image corresponding to the timestamp and the observation view angle to form training data;

[0010] S2: design a corresponding region identification function for each Gaussian kernel in the 4D Gaussian model, each region identification function uses a respective rendering discrimination condition to discriminate the renderable region of the Gaussian kernel; for any point on the 2D image plane, the 2D coordinates of the point are successively discriminated to see whether they meet each rendering discrimination condition: if a rendering discrimination condition is met, the value of the region identification function corresponding to the rendering discrimination condition at the point is 1 and the Gaussian kernel needs to render the point, otherwise the value of the region identification function at the point is 0 and the point is not rendered; until all points on the 2D image plane are discriminated, the set of points to be rendered corresponding to each Gaussian kernel is output;

[0011] S3: training the 4D Gaussian model using the training data, setting a learnable Alpha parameter for each Gaussian kernel for self-heterogeneity learning during the training process, rendering each Gaussian kernel on its corresponding point set to be rendered, using a segmented activation function controlled by the Alpha parameter to adjust the opacity of each Gaussian kernel when calculating the rendering transparency, and forming a rendered image by the renderer in the 4D Gaussian model from all the adjusted rendering pixel opacities, constructing a total loss from the mean square error loss and the SSIM loss between the rendered image and the original image, updating the 4D Gaussian model parameters based on minimizing the total loss, and obtaining the 4D Gaussian model for reconstruction after multiple iterations;

[0012] S4: inputting the specified time parameters and viewing angle parameters into the trained 4D Gaussian model to generate a corresponding dynamic scene rendering image, and completing the 4D Gaussian reconstruction.

[0013] On the basis of the above scheme, each step can be implemented in the following preferred specific manner.

[0014] As a preferred embodiment of the first aspect, the specific process of accurately parameterizing the original timestamp and the original observation viewing angle in step S1 is as follows:

[0015] S11: processing the original observation viewing angle by a projection matrix to transform it from the world coordinate system to the image plane coordinate system, forming a processed observation viewing angle;

[0016] S12: mapping the discrete original timestamp to the [0, 1] interval using a normalization method to form a processed timestamp.

[0017] As a preferred embodiment of the first aspect, in step S2, the specific process of determining whether the 2D coordinates of any one point on the 2D image plane satisfy a rendering determination condition is as follows: obtaining the size of the long axis of the ellipse formed by the projection of a Gaussian kernel on the 2D image plane and the size of the short axis of the ellipse , and taking the feature vector corresponding to the long axis of the ellipse as the first feature vector , and taking the feature vector corresponding to the short axis of the ellipse as the second feature vector The dot product result of the 2D coordinate and the first eigenvector is taken as a first dot product result, a ratio of the first dot product result to the size of the long axis of the ellipse is taken as a first ratio, the dot product result of the 2D coordinate and the second eigenvector is taken as a second dot product result, a ratio of the second dot product result to the size of the short axis of the ellipse is taken as a second ratio, the size of the long axis of the ellipse and the negative of the size of the long axis of the ellipse are taken as two end points of a first discriminant closed interval, the size of the short axis of the ellipse and the negative of the size of the short axis of the ellipse are taken as two end points of a second discriminant closed interval, when the first ratio is located in the first discriminant closed interval and the second ratio is located in the second discriminant closed interval, the 2D coordinate of the point satisfies the rendering discriminant condition, the value of the region identification function corresponding to the rendering discriminant condition at the point is 1, and the Gaussian kernel corresponding to the region identification function needs to render the point; otherwise, the 2D coordinate of the point does not satisfy the rendering discriminant condition, the value of the region identification function corresponding to the rendering discriminant condition at the point is 0, and the Gaussian kernel corresponding to the region identification function does not need to render the point.

[0018] Further, in step S2, the region identification function is specifically represented as follows:

[0019]

[0020] wherein, represents a dot product operation of two vectors; represents a value of the region identification function at a point with a 2D coordinate of .

[0021] As a preferred embodiment of the first aspect, the process of calculating the size of the long axis of the ellipse and the size of the short axis of the ellipse formed by projecting the Gaussian kernel is as follows:

[0022] S21: calculating, for each 3D covariance matrix of the Gaussian kernel, to obtain a 2D covariance matrix of the Gaussian kernel projected onto the image plane;

[0023] S22: solving two eigenvalues corresponding to the 2D covariance matrix of each Gaussian kernel, and calculating based on the obtained two eigenvalues to obtain the size of the long axis of the ellipse and the size of the short axis of the ellipse projected onto the 2D image plane.

[0024] As a preferred embodiment of the first aspect, in step S21, the specific process of obtaining the 2D covariance matrix of a Gaussian kernel is as follows: multiplying the Jacobian matrix , the view angle matrix , the 3D covariance matrix of the Gaussian kernel , the transpose of the view angle matrix, and the transpose of the Jacobian matrix to obtain the 2D covariance matrix of the Gaussian kernel after projection. .

[0025] Further, in step S21, the 2D covariance matrix of the Gaussian kernel projected onto the image plane can be calculated as follows:

[0026]

[0027] in, This indicates the matrix transpose.

[0028] As a preferred embodiment of the first aspect mentioned above, in step S22, the size of the major axis of the ellipse is three times the arithmetic square root of the larger of the two eigenvalues, and the size of the minor axis of the ellipse is three times the arithmetic square root of the smaller of the two eigenvalues.

[0029] Furthermore, in step S22, the two eigenvalues ​​corresponding to the 2D covariance matrix Calculate using the following formula:

[0030]

[0031]

[0032] in, These correspond to the values ​​in the 2D covariance matrix.

[0033] Furthermore, the size of the major axis of the ellipse and the size of the minor axis of the ellipse They are represented as follows:

[0034]

[0035]

[0036] in, This represents the larger eigenvalue among the two eigenvalues ​​of the 2D covariance matrix; This represents the smaller eigenvalue among the two eigenvalues ​​of the 2D covariance matrix.

[0037] As a preferred embodiment of the first aspect mentioned above, in step S3, the first... Rendered pixel opacity adjusted by a Gaussian kernel By the The self-opacity of a Gaussian kernel and the function value of the piecewise activation function The function value is obtained by multiplying two parts, the first part being the coefficient of the piecewise activation function. With power terms calculated from the 2D covariance matrix The first part is the result of multiplication between the two parts, and the second part is the result of calculating the residual term in the piecewise activation function. .

[0038] Furthermore, the first Opacity adjusted by a Gaussian kernel Specifically, it is expressed as follows:

[0039]

[0040] As a preferred embodiment of the first aspect, the coefficients are all generated based on a learnable Alpha parameter; when the power term is greater than or equal to 0 and less than or equal to a preset first threshold, the residual term is 0; when the power term is greater than the first threshold and less than or equal to a preset second threshold, the residual term is generated based on the learnable Alpha parameter; when the power term is greater than the second threshold and less than or equal to 1, the residual term is generated based on the learnable Alpha parameter.

[0041] As a preferred embodiment of the first aspect, the learnable Alpha parameter is optimized using a backpropagation method.

[0042] In a second aspect, the present application provides a 4D Gaussian reconstruction system based on self-heterogeneity, which comprises:

[0043] A data acquisition module is configured to accurately parameterize the input original timestamps and original observation angles respectively, combine the processed timestamps, processed observation angles with the original images corresponding to the timestamps and observation angles to form training data;

[0044] A region identification module is configured to design a corresponding region identification function for each Gaussian kernel in the 4D Gaussian model, and each region identification function uses a respective rendering discrimination condition to discriminate the renderable region of the Gaussian kernel; for any point on a 2D image plane, the 2D coordinates of the point are sequentially discriminated to see if they satisfy each rendering discrimination condition: if a rendering discrimination condition is satisfied, the value of the region identification function corresponding to the rendering discrimination condition at the point is 1 and the Gaussian kernel needs to render the point, otherwise the value of the region identification function at the point is 0 and the point is not rendered; until all points on the 2D image plane are discriminated, the respective corresponding set of points to be rendered for each Gaussian kernel is outputted;

[0045] A model training module is configured to train the 4D Gaussian model using the training data, and set a learnable Alpha parameter for each Gaussian kernel for self-heterogeneity learning during the training process; each Gaussian kernel performs rendering on the respective set of points to be rendered, and when the rendering transparency is calculated, a segmented activation function controlled by the Alpha parameter is used to adjust the self-opacity of the respective Gaussian kernel, and all adjusted rendering pixel opacities form a rendered image through a renderer in the 4D Gaussian model; a total loss is constructed based on the mean square error loss and the SSIM loss between the rendered image and the original image, the 4D Gaussian model parameters are updated based on minimizing the total loss, and the 4D Gaussian model for reconstruction is obtained through multiple rounds of iteration;

[0046] The reconstruction module is used for inputting the specified time parameters and perspective parameters into the trained 4D Gaussian model to generate a corresponding dynamic scene rendering image, and completes 4D Gaussian reconstruction.

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] The present application proposes a 4D Gaussian reconstruction method based on self-heterogeneity for training with limited specific style pictures, which can quickly realize high-fidelity 4D reconstruction effect. In the present application, a learnable Alpha parameter is set for each Gaussian kernel for self-heterogeneity learning, and the core idea is to decouple the alpha transparency and the Gaussian shape, breaking the fixed structure of the traditional template alpha distribution. In addition, based on the above Alpha parameter, the present application also designs a segmented activation function controlled by the Alpha parameter, which allows each Gaussian to adaptively adjust its alpha profile according to the frequency characteristics of the region it is in. In the high-frequency region, the sharp alpha boundary is retained to capture clear motion or texture changes; while in the low-frequency region, a more flat alpha distribution is adopted to enhance spatial consistency and reduce unnecessary overlap. This decoupling in structure enables each Gaussian to perform functional division between different tasks, fundamentally alleviating the optimization conflict between appearance modeling and motion modeling. In order to further improve the representation efficiency and optimization effect, the present application introduces a region recognition function. The traditional 3DGS needs to calculate alpha for all Gaussians in the rendering process, including a large number of redundant operations that do not contribute to the final image. However, the method of the present application quickly eliminates these "ineffective Gaussians" before alpha calculation, thereby significantly reducing unnecessary calculation and memory overhead. This strategy not only improves the convergence speed in the training stage, but also provides "zero-cost improvement" in inference, further enhancing the practical applicability of the method of the present application.

[0049] The present application and the existing 4D Gaussian reconstruction method are widely experimented on multiple data sets, and the results show that the method of the present application significantly reduces the motion blur phenomenon without increasing the inference cost. Compared with the existing methods, the method of the present application achieves excellent performance in spatiotemporal consistency, boundary sharpness, reconstruction accuracy and other indicators. In addition, since the method of the present application is an improvement on the representation structure itself, it can be seamlessly integrated with existing motion modeling or rendering acceleration techniques, and has good scalability and engineering practical value. In summary, the present application systematically solves the problem of motion blur in current 4D reconstruction from the root of the problem, and provides a more efficient, accurate and universal solution for dynamic image modeling. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The step flowchart of the method of the present application;

[0051] Figure 2 This is a comparison chart of the rendering results using learnable Alpha parameters and not using learnable Alpha parameters in this embodiment;

[0052] Figure 3 This is a comparison chart of the rendering results of different methods on the DNeRF dataset in this embodiment;

[0053] Figure 4 This is a comparison chart of the rendering results of different methods on the Neu3D dataset in this embodiment;

[0054] Figure 5 This is a system block diagram of the present invention. Detailed Implementation

[0055] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0056] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0057] like Figure 1 As shown, in a preferred embodiment of the present invention, a 4D Gaussian reconstruction method based on self-heterogeneity is provided. This method is implemented based on invariant representation learning and can achieve fast and high-fidelity 4D reconstruction results. The above-mentioned 4D Gaussian reconstruction method based on self-heterogeneity includes the following steps S1 to S4. The specific implementation process of each step is described in detail below.

[0058] S1: The original timestamp and original viewing angle of the input are precisely parameterized, and the processed timestamp and processed viewing angle are combined with the original images of the corresponding timestamp and viewing angle to form training data.

[0059] It should be noted that step S1 provides temporally and spatially aligned supervision information for the subsequent training of the 4D Gaussian model.

[0060] In step S1 of the embodiment, the specific process of accurately parameterizing the original timestamp and the original observation perspective is as follows:

[0061] S11: The original observation perspective is processed by a projection matrix to be transformed from the world coordinate system to the image plane coordinate system, to form a processed observation perspective.

[0062] S12: The discrete original timestamp is mapped to the interval [0, 1] using a normalization method to form a processed timestamp.

[0063] S2: A corresponding region identification function is designed for each Gaussian kernel in the 4D Gaussian model, and each region identification function discriminates the renderable region of the Gaussian kernel using a respective rendering discrimination condition. For any point on the 2D image plane, the 2D coordinates of the point are sequentially discriminated to see whether they satisfy each rendering discrimination condition: if a rendering discrimination condition is satisfied, the value of the region identification function corresponding to the rendering discrimination condition at the point is 1 and the Gaussian kernel needs to render the point, otherwise the value of the region identification function at the point is 0 and the point is not rendered; until all points on the 2D image plane are discriminated, the respective corresponding set of points to be rendered for each Gaussian kernel is output.

[0064] It should be noted that in step S2, the rendering discrimination condition is used to clip the renderable region of the Gaussian kernel, which aims to identify and remove the redundant Gaussian kernel part that contributes little or nothing to the final rendering result, thereby avoiding unnecessary rasterization calculation and significantly improving rendering efficiency.

[0065] In step S2 of the present application, the specific process of determining whether the 2D coordinates of any point on the 2D image plane satisfy a rendering discrimination condition is as follows: Get the size of the major axis of the ellipse formed by the projection of a Gaussian kernel on the 2D image plane and the size of the minor axis of the ellipse , and take the feature vector corresponding to the major axis of the ellipse as the first feature vector , and take the feature vector corresponding to the minor axis of the ellipse as the second feature vector The point product result of the 2D coordinate and the first feature vector is taken as a first point product result, a ratio of the first point product result to the size of the long axis of the ellipse is taken as a first ratio, the point product result of the 2D coordinate and the second feature vector is taken as a second point product result, a ratio of the second point product result to the size of the short axis of the ellipse is taken as a second ratio, the size of the long axis of the ellipse and the opposite number thereof are taken as two end points of a first discriminant closed interval, the size of the short axis of the ellipse and the opposite number thereof are taken as two end points of a second discriminant closed interval, when the first ratio is located in the first discriminant closed interval and the second ratio is located in the second discriminant closed interval, the 2D coordinate of the point satisfies the rendering discriminant condition, the value of the region identification function corresponding to the rendering discriminant condition at the point is 1, and the Gaussian kernel corresponding to the region identification function needs to render the point; otherwise, the 2D coordinate of the point does not satisfy the rendering discriminant condition, the value of the region identification function corresponding to the rendering discriminant condition at the point is 0, and the Gaussian kernel corresponding to the region identification function does not need to render the point.

[0066] In step S2 of the embodiment, a corresponding region identification function is matched for each Gaussian kernel before Gaussian rasterization calculation. The region identification function is specifically represented as follows:

[0067]

[0068] wherein, represents a point product operation of two vectors; represents the value of the region identification function at the point with the 2D coordinate .

[0069] It should be noted that, in the present application, the 2D covariance matrix of the obtained Gaussian kernel can be used to calculate the ellipse formed by the projection of the Gaussian kernel on the 2D image plane and the range covered by the ellipse. Since the ellipse is obtained by the respective 2D projection of the Gaussian kernel, the visible range is within three times the eigenvalue of the Gaussian distribution, and accordingly, the long axis and the short axis of the obtained ellipse can be calculated according to the 2D covariance of the Gaussian kernel and the three times eigenvalue.

[0070] Specifically, the process of calculating the size of the long axis of the ellipse formed by the projection of the Gaussian kernel and the size of the short axis of the ellipse is as follows:

[0071] S21: the 3D covariance matrix of each Gaussian kernel is calculated respectively to obtain the 2D covariance matrix of each Gaussian kernel projected on the image plane.

[0072] In S21 of the present application, the specific process of obtaining the 2D covariance matrix of a Gaussian kernel is as follows: the Jacobian matrix , the view angle matrix , the 3D covariance matrix of the Gaussian kernel , the transpose of the view angle matrix and the transpose of the Jacobian matrix are multiplied to obtain the 2D covariance matrix of the Gaussian kernel after projection .

[0073] In the embodiment S21, the 2D covariance matrix of the Gaussian kernel projected onto the image plane can be calculated as follows:

[0074]

[0075] wherein, denotes the matrix transpose; denotes the view matrix, which is a parameter recording the view angle of the shooting device when shooting.

[0076] In addition, the center coordinates of the Gaussian kernel can be multiplied by the obtained view matrix and projection matrix to obtain the 2D coordinates of the image plane after the projection of the Gaussian kernel :

[0077]

[0078] S22: Solve the two eigenvalues corresponding to the 2D covariance matrix of each Gaussian kernel, and calculate based on the obtained two eigenvalues to obtain the long axis size and the short axis size of the ellipse of the Gaussian kernel projected onto the 2D image plane.

[0079] In the present application S22, the long axis size of the ellipse is 3 times the arithmetic square root of the larger eigenvalue, and the short axis size of the ellipse is 3 times the arithmetic square root of the smaller eigenvalue.

[0080] In the embodiment S22, the two eigenvalues corresponding to the 2D covariance matrix are calculated as follows:

[0081]

[0082]

[0083] wherein, respectively correspond to the values in the 2D covariance matrix. Thus, the long axis size of the ellipse and the short axis size of the ellipse are respectively represented as:

[0084]

[0085]

[0086] wherein, denotes the larger eigenvalue of the two eigenvalues; denotes the smaller eigenvalue of the two eigenvalues.

[0087] In the present application, the first feature vector and the second feature vector are obtained by the following formula:

[0088]

[0089] wherein, denotes the eigenvector formed by two eigenvalues of the 2D covariance matrix ; denotes the unit matrix; denotes the feature matrix composed of the first feature vector and the second feature vector, the first row of the feature matrix being the first feature vector and the second row being the second feature vector.

[0090] S3: training the 4D Gaussian model using the training data, setting a learnable Alpha parameter for each Gaussian kernel for self-heterogeneity learning during the training process, and performing rendering on the corresponding point set to be rendered by each Gaussian kernel, when the rendering transparency is calculated, using the segmented activation function controlled by the Alpha parameter to adjust the self-opacity of the respective Gaussian kernel, and all adjusted rendering pixel opacities forming a rendering image through the renderer in the 4D Gaussian model, constructing a total loss from the mean square error loss and the SSIM loss between the rendering image and the original image, updating the 4D Gaussian model parameters based on minimizing the total loss, and obtaining the 4D Gaussian model for reconstruction through multiple iterations;

[0091] It should be noted that in step S3 of the present application, the Alpha parameter is adaptively learned in the training process of the 4D Gaussian model to realize self-heterogeneity change of the Gaussian kernel, so that it can accurately adjust the transparency or opacity characteristics according to the scene dynamics and its own contribution.

[0092] In the present application, the first feature vector and the second feature vector are obtained by the following formula: the adjusted rendering pixel opacity of the first Gaussian kernel is obtained by multiplying the self-opacity of the first Gaussian kernel and the function value of the segmented activation function , the function value being obtained by adding two parts, the first part being the multiplication result between the coefficient of the segmented activation function and the power term calculated according to the 2D covariance matrix, and the second part being the calculation result of the residual term in the segmented activation function, denoted by .

[0093] ​Furthermore, all coefficients are generated based on learnable alpha parameters; when the power term is greater than or equal to 0 and less than or equal to a preset first threshold, the residual term is 0; when the power term is greater than the first threshold and less than or equal to a preset second threshold, the residual term is generated based on learnable alpha parameters; when the power term is greater than the second threshold and less than or equal to 1, the residual term is generated based on learnable alpha parameters.

[0094] In this embodiment, significant blurring exists when reconstructing dynamic 3D scenes using existing methods. Taking a classic scene from dynamic 3D reconstruction as an example, such as... Figure 2 As shown, the alpha-free method exhibits significant blurring in the finger portion, and this blurring worsens from time 1 to time 2. Previous methods attributed this to inaccurate Gaussian kernel driving by the deformation mesh. However, our analysis reveals that the instability of the Gaussian kernel's alpha value during rendering interferes with the deformation mesh's control over the kernel's motion. Specifically, since the error between the rendered and ground truth images is calculated using mean squared error for optimization, the deformation mesh needs to drive the Gaussian kernel to different positions at each time step to render the image. When a gap exists between the two Gaussian kernels at time 1 (i.e., the pixel's alpha value is too small, appearing transparent), the optimization method based on minimizing mean squared error loss forces the deformation mesh to not only learn the true motion direction but also manipulate the Gaussian kernels to stack to fill the gap (even if the pixel's alpha becomes larger, appearing more opaque). Since the vector and direction (i.e., the deformation mesh's driving direction) do not conform to the true motion direction, this is the reason for the significant blurring in dynamic 3D reconstruction using the alpha-free method.

[0095] To address this, this invention proposes an alpha-self-heterogeneity method, which manipulates the alpha values ​​rendered by Gaussian kernels to enable the deformation mesh to learn correct motion. The core idea is to calculate the alpha value of each Gaussian kernel at a pixel during rendering and then classify them into two types: one type consistently used to represent pixels with small alpha values, and the other consistently used to represent pixels with large alpha values. This allows the deformation mesh to focus on the positional mapping between Gaussian kernels representing pixels with the same alpha value, i.e., the correct motion relationship.

[0096] Based on the above analysis, this embodiment generalizes the existing Gaussian rasterization formula to obtain the corresponding general rendering equation:

[0097]

[0098] in, This indicates the rendering pixel opacity used for rasterization (i.e., the adjusted rendering pixel opacity). denotes a piecewise activation function controlled by an Alpha parameter; denotes a self opacity of a denotes a power term calculated according to a 2D covariance matrix; denotes a coefficient in a piecewise activation function; denotes a residual term in a piecewise activation function.

[0099] In the embodiment, the piecewise activation function is used to remap the rasterization formula. When the first threshold is set to 0.25 and the second threshold is set to 0.75, the function form of the piecewise activation function is:

[0100]

[0101] wherein, is a learnable Alpha parameter introduced in the present application. When the power term is greater than or equal to 0 and less than or equal to 0.25, the coefficient is , and the residual term is 0; when the power term is greater than 0.25 and less than or equal to 0.75, the coefficient is , and the residual term is ; when the power term is greater than 0.75 and less than or equal to 1, the coefficient is , and the residual term is .

[0102] It should be noted that in step S3 of the present application, the learnable Alpha parameter is optimized by using the method of backpropagation. In the embodiment, according to the function form of the piecewise activation function, the learnable Alpha parameter corresponding partial derivative is:

[0103]

[0104] In the embodiment, when the 4D Gaussian model is trained by using the training data, a training data (containing a processed timestamp, a processed observation angle and an original image) is given and input into the 4D Gaussian model to obtain a rendered image corresponding to the processed observation angle and the processed timestamp. Then, the rendered image and the original image are calculated to obtain a mean square error loss and an SSIM loss, and after multiple iterations, the 4D Gaussian model used for reconstruction is obtained.

[0105] S4: input the specified time parameter and angle parameter into the trained 4D Gaussian model to generate a corresponding dynamic scene rendering image, and complete the 4D Gaussian reconstruction.

[0106] ​It should be noted that the 4D Gaussian model trained through the above steps supports efficient and high-quality rendering. Specifically, by directly inputting the specified time parameter and view angle parameter, the corresponding dynamic scene rendering image can be quickly generated.

[0107] The 4D Gaussian reconstruction method based on self-heterogeneity in the above embodiment will be applied to a specific data set for classification testing. The specific steps are as described in S1-S4, which will not be repeated, and the main purpose is to show the specific parameters and technical effects.

[0108] Embodiment

[0109] This embodiment follows the implementation process of the foregoing S1-S4 steps. For quantitative indicators, this embodiment performs qualitative and quantitative experiments on the DNeRF and Neu3D data sets, and compares the method of the present application with the prior art 4D-GS, DeformGS, SC-GS, Grid4D, Tensor-4D, HexPlane, and TiNeuVox-B. The comparison results on the DNeRF data set are shown in Table 1, the comparison results on the Neu3D data set are shown in Table 2, and the quantitative evaluation results on the DNeRF data set are shown in Table 3. Figure 3 Figure 4 Figure 3 In the above tables, the first comparison method refers to the prior art 4D-GS, the second comparison method refers to the prior art DeformGS, the third comparison method refers to the prior art SC-GS, and the fourth comparison method refers to the prior art Grid4D.

[0110] Table 1. Test data PSNR quantitative results

[0111]

[0112] Table 2. Test data SSIM quantitative results

[0113]

[0114] Table 3. Test data LPIPS quantitative results

[0115]

[0116] From Tables 1-3, it can be clearly observed that in terms of fidelity, consistency, and visual perception, the present application has achieved the best quantitative indicators, which reflects the present application's ability to accurately reconstruct dynamic 4D scenes. From Figure 3 and Figure 4 It can be clearly observed that the present application has a detailed advantage in reconstruction details, which is due to the Gaussian kernel proposed by the present application, which can maintain stability in motion, thereby accurately reconstructing the set and texture information in training. ​​

[0117] It should be further noted that the 4D Gaussian reconstruction method based on self-heterogeneity in the above embodiments can be essentially executed by a computer program or module. Therefore, based on the same inventive concept, another preferred embodiment of the present application also provides a 4D Gaussian reconstruction system based on self-heterogeneity corresponding to the 4D Gaussian reconstruction method based on self-heterogeneity provided by the above embodiments, as shown in Figure 5 which comprises:

[0118] A data acquisition module for accurately parameterizing each of the input original timestamp and original observation angle, combining the processed timestamp, the processed observation angle with the original image corresponding to the timestamp and the observation angle to form training data;

[0119] A region identification module for designing a corresponding region identification function for each Gaussian kernel in the 4D Gaussian model, each region identification function using a respective rendering discrimination condition to discriminate the renderable region of the Gaussian kernel; for any one point on the 2D image plane, the 2D coordinates of the point are discriminated in turn to see if they meet each rendering discrimination condition: if they meet a rendering discrimination condition, the value of the region identification function corresponding to the rendering discrimination condition at the point is 1 and the Gaussian kernel needs to render the point, otherwise the value of the region identification function at the point is 0 and the point is not rendered; until all points on the 2D image plane are discriminated, the respective corresponding set of points to be rendered for each Gaussian kernel is output;

[0120] A model training module for training the 4D Gaussian model using the training data, and setting a learnable Alpha parameter for each Gaussian kernel for self-heterogeneity learning during the training process, each Gaussian kernel rendering on its corresponding set of points to be rendered, when the rendering transparency is calculated, using the Alpha parameter controlled segmented activation function to adjust the self opacity of the respective Gaussian kernel, all adjusted rendering pixel opacities forming a rendered image through the renderer in the 4D Gaussian model, the total loss being constructed from the mean square error loss and the SSIM loss between the rendered image and the original image, the 4D Gaussian model parameters being updated based on minimizing the total loss, and the 4D Gaussian model for reconstruction being obtained through multiple iterations;

[0121] A reconstruction module for inputting specified time parameters and viewing angle parameters to the trained 4D Gaussian model to generate a corresponding dynamic scene rendering image, completing the 4D Gaussian reconstruction.

[0122] In addition, it should be noted that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the foregoing method embodiments, and will not be described here. In the embodiments provided in the present application, the division of steps or modules in the system and method described is only a logical functional division, and there can be another division manner in actual implementation, for example, multiple modules or steps can be combined or integrated together, or a module or step can be split.

[0123] The above-described embodiments are only a preferred scheme of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical scheme obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present application.

Claims

1. A 4D Gaussian reconstruction method based on self-heterogeneity, characterized in that, Includes the following steps: S1: The original timestamp and the original viewing angle are precisely parameterized, and the processed timestamp and the processed viewing angle are combined with the original images of the corresponding timestamp and viewing angle to form training data. S2: Design a corresponding region identification function for each Gaussian kernel in the 4D Gaussian model. Each region identification function uses its own rendering discrimination condition to determine the renderable region of the Gaussian kernel. For any point on the 2D image plane, determine whether the 2D coordinates of the point satisfy each rendering discrimination condition in turn: if a rendering discrimination condition is satisfied, the value of the region identification function corresponding to the rendering discrimination condition at the point is 1 and the Gaussian kernel needs to render the point; otherwise, the value of the region identification function at the point is 0 and the point is not rendered. Continue until all points on the 2D image plane have been identified, and output the set of points to be rendered corresponding to each Gaussian kernel. S3: Train the 4D Gaussian model using training data, and set a learnable Alpha parameter for each Gaussian kernel during the training process to perform self-heterogeneity learning. Each Gaussian kernel renders on its corresponding set of points to be rendered. When the rendering opacity is calculated, the piecewise activation function controlled by the Alpha parameter is used to adjust the opacity of each Gaussian kernel. All the adjusted rendering pixel opacities are processed by the renderer in the 4D Gaussian model to form a rendering image. The total loss is constructed by the mean squared error loss and SSIM loss between the rendering image and the original image. The 4D Gaussian model parameters are updated based on minimizing the total loss. After multiple iterations, a 4D Gaussian model for reconstruction is obtained. S4: Input the specified time parameters and viewpoint parameters into the trained 4D Gaussian model to generate the corresponding dynamic scene rendering map and complete the 4D Gaussian reconstruction.

2. The 4D Gaussian reconstruction method based on self-heterogeneity as described in claim 1, characterized in that, In step S1, the specific process of precisely parameterizing the original timestamp and the original observation perspective is as follows: S11: The original viewing angle is processed by a projection matrix to transform it from the world coordinate system to the image plane coordinate system, thus forming the processed viewing angle. S12: Normalize the discrete original timestamps and map them to the interval [0,1] to form the processed timestamps.

3. The 4D Gaussian reconstruction method based on self-heterogeneity as described in claim 1, characterized in that, In step S2, the specific process of determining whether the 2D coordinates of any point on the 2D image plane satisfy a rendering discrimination condition is as follows: Obtain the size of the major axis and minor axis of the ellipse formed by projecting a Gaussian kernel onto the 2D image plane; simultaneously, use the eigenvector corresponding to the major axis of the ellipse as the first eigenvector, and the eigenvector corresponding to the minor axis of the ellipse as the second eigenvector; take the dot product of the 2D coordinates and the first eigenvector as the first dot product result; take the ratio of the first dot product result to the size of the major axis of the ellipse as the first ratio; and take the ratio of the 2D coordinates and the second eigenvector... The dot product result is used as the second dot product result. The ratio of the second dot product result to the size of the minor axis of the ellipse is used as the second ratio. The size of the major axis of the ellipse and its opposite are used as the two endpoints of the first discriminative closed interval. The size of the minor axis of the ellipse and its opposite are used as the two endpoints of the second discriminative closed interval. When the first ratio is within the first discriminative closed interval and the second ratio is within the second discriminative closed interval, the 2D coordinates of the point satisfy the rendering discrimination condition. The value of the region identification function corresponding to the rendering discrimination condition is 1 at the point. The Gaussian kernel corresponding to the region identification function needs to render the point. Otherwise, the 2D coordinates of the point do not meet the rendering criteria, the value of the region identification function corresponding to the rendering criteria is 0 at the point, and the Gaussian kernel corresponding to the region identification function does not need to render the point.

4. The 4D Gaussian reconstruction method based on self-heterogeneity as described in claim 3, characterized in that, The process of calculating the major and minor axes of the ellipse formed by the Gaussian kernel projection is as follows: S21: Calculate the 3D covariance matrix of each Gaussian kernel separately to obtain the 2D covariance matrix of each Gaussian kernel projected onto the image plane; S22: Solve for the two eigenvalues ​​corresponding to the 2D covariance matrix of each Gaussian kernel, and calculate based on the obtained two eigenvalues ​​to obtain the major axis and minor axis of the ellipse projected onto the 2D image plane by the Gaussian kernel.

5. The 4D Gaussian reconstruction method based on self-heterogeneity as described in claim 4, characterized in that, In step S21, the specific process of obtaining a 2D covariance matrix of a Gaussian kernel is as follows: multiply the five parts, namely the Jacob matrix, the view matrix, the 3D covariance matrix of the Gaussian kernel, the transpose of the view matrix, and the transpose of the Jacob matrix, to obtain the 2D covariance matrix after the projection of the Gaussian kernel.

6. The 4D Gaussian reconstruction method based on self-heterogeneity as described in claim 4, characterized in that, In step S22, the major axis of the ellipse is three times the square root of the larger of the two eigenvalues, and the minor axis of the ellipse is three times the square root of the smaller of the two eigenvalues.

7. The 4D Gaussian reconstruction method based on self-heterogeneity as described in claim 1, characterized in that, In step S3, the opacity of the rendered pixel after the Gaussian kernel adjustment is obtained by multiplying the opacity of the Gaussian kernel itself by the function value of the piecewise activation function. The function value is obtained by adding two parts: the first part is the result of multiplying the coefficient of the piecewise activation function with the power term calculated based on the 2D covariance matrix, and the second part is the result of calculating the residual term in the piecewise activation function.

8. The 4D Gaussian reconstruction method based on self-heterogeneity as described in claim 7, characterized in that, The coefficients are all generated based on learnable Alpha parameters; when the power term is greater than or equal to 0 and less than or equal to a preset first threshold, the residual term is 0; when the power term is greater than the first threshold and less than or equal to a preset second threshold, the residual term is generated based on learnable Alpha parameters; when the power term is greater than the second threshold and less than or equal to 1, the residual term is generated based on learnable Alpha parameters.

9. The 4D Gaussian reconstruction method based on self-heterogeneity as described in claim 8, characterized in that, The learnable alpha parameter is optimized using backpropagation.

10. A 4D Gaussian reconstruction system based on self-heterogeneity, characterized in that, include: The data acquisition module is used to precisely parameterize the input raw timestamps and raw viewing angles, and combine the processed timestamps and processed viewing angles with the original images of the corresponding timestamps and viewing angles to form training data. The region identification module is used to design a corresponding region identification function for each Gaussian kernel in the 4D Gaussian model. Each region identification function uses its own rendering discrimination conditions to determine the renderable region of the Gaussian kernel. For any point on the 2D image plane, it sequentially checks whether the 2D coordinates of the point satisfy each rendering discrimination condition: if a rendering discrimination condition is satisfied, the value of the region identification function corresponding to that rendering discrimination condition at that point is 1 and the Gaussian kernel needs to render that point; otherwise, the value of the region identification function at that point is 0 and the point is not rendered. This process continues until all points on the 2D image plane have been identified, and then the set of points to be rendered corresponding to each Gaussian kernel is output. The model training module is used to train the 4D Gaussian model using training data. During the training process, a learnable Alpha parameter is set for each Gaussian kernel to perform self-heterogeneity learning. Each Gaussian kernel renders on its corresponding set of points to be rendered. When the rendering opacity is calculated, the opacity of each Gaussian kernel is adjusted using a piecewise activation function controlled by the Alpha parameter. All the adjusted rendering pixel opacities are processed by the renderer in the 4D Gaussian model to form a rendering image. The total loss is constructed by the mean squared error loss and SSIM loss between the rendering image and the original image. The 4D Gaussian model parameters are updated based on minimizing the total loss. After multiple iterations, a 4D Gaussian model for reconstruction is obtained. The reconstruction module is used to input the specified time parameters and viewpoint parameters into the trained 4D Gaussian model, generate the corresponding dynamic scene rendering map, and complete the 4D Gaussian reconstruction.

Images