Image rendering method and system, computer equipment, storage medium and product
By constructing spatial domain and frequency domain regular terms and loop constraints of the three-dimensional Gaussian splatter model, the problem of low rendering quality of the three-dimensional Gaussian splatter technology is solved, and higher rendering quality and fidelity are achieved.
Patent Information
- Application Number
- CN202411584199.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-07
- Publication Date
- 2025-03-11
AI Technical Summary
When rendering images by 3D Gaussian splattering technology, there is still a big difference between the generated image and the real image, resulting in low rendering quality.
By obtaining the rendered image and reference image output from the three-dimensional Gaussian splatter model, constructing spatial domain regular terms and frequency domain regular terms, loop constraints the three-dimensional Gaussian splatter model to generate intermediate images until the number of times threshold is reached, and the rendered image is optimized.
Improve the rendering quality of 3D Gaussian splattering technology, making the generated image closer to the reference image and has higher restore and fidelity.
Smart Images
Figure CN119600167A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image rendering method, system, computer device, storage medium and product. Background Art
[0002] 3D Gaussian splatting is a technique used to create point cloud rendering in computer graphics. Specifically, 3D Gaussian splatting can process large-scale point cloud data and generate realistic images or videos.
[0003] However, although the three-dimensional Gaussian splashing technology can render realistic images, the rendered images are still quite different from the real images.
[0004] Therefore, how to improve the rendering quality of the three-dimensional Gaussian splattering technology is a technical problem that needs to be solved by those skilled in the art. Summary of the invention
[0005] The main purpose of this application is to provide an image rendering method, system, computer device, storage medium and product, aiming to solve the technical problem of how to improve the rendering quality of three-dimensional Gaussian splattering technology.
[0006] To achieve the above object, the present application proposes an image rendering method, the method comprising:
[0007] Acquire a rendered image output by a three-dimensional Gaussian splash model, and acquire a reference image, wherein the rendered image and the reference image belong to the same visual scene;
[0008] constructing a spatial domain regularization term according to the pixel gradient of the rendered image, and constructing a frequency domain regularization term according to the frequency domain information of the reference image and the frequency domain information of the rendered image;
[0009] Constraining the three-dimensional Gaussian splatter model according to the spatial domain regularization term and the frequency domain regularization term to generate an intermediate image based on the rendered image, and cyclically executing the steps of using the intermediate image as the rendered image and constructing the spatial domain regularization term according to the pixel gradient of the rendered image until the number of times the intermediate image is generated reaches a number threshold;
[0010] The image to be rendered is rendered by using the three-dimensional Gaussian splash model to obtain an image rendering result.
[0011] In one embodiment, the step of constructing a spatial domain regularization term according to the pixel gradient of the rendered image comprises:
[0012] Inputting pixel gradients corresponding to respective positions in the rendered image into a preset pixel weight model to obtain pixel weights corresponding to respective positions, wherein the pixel weights in the pixel weight model are proportional to the pixel gradients;
[0013] An L1 loss between the rendered image and the reference image is calculated, and a spatial domain regularization term is constructed according to the L1 loss and each pixel weight.
[0014] In one embodiment, before the step of constructing a frequency domain regularization term according to the frequency domain information of the reference image and the frequency domain information of the rendered image, the method further includes:
[0015] Extracting high-frequency frequency domain information in the frequency domain information of the reference image and high-frequency frequency domain information in the frequency domain information of the rendered image through a high-pass filter;
[0016] The step of constructing a frequency domain regularization term according to the frequency domain information of the reference image and the frequency domain information of the rendered image comprises:
[0017] A frequency domain regularization term is constructed according to the high frequency frequency domain information of the reference image and the high frequency frequency domain information of the rendered image.
[0018] In one embodiment, the high-frequency frequency domain information of the reference image includes a first amplitude value and a first phase value, and the high-frequency frequency domain information of the rendered image includes: a second amplitude value and a second phase value;
[0019] The step of constructing a frequency domain regularization term according to the high-frequency frequency domain information of the reference image and the high-frequency frequency domain information of the rendered image comprises:
[0020] determining an amplitude difference between the first amplitude value and the second amplitude value, and determining a phase difference between the first phase value and the second phase value;
[0021] A frequency domain regularization term is constructed based on the amplitude difference and the phase difference.
[0022] In one embodiment, the step of obtaining a rendered image output by a three-dimensional Gaussian splatter model and obtaining a reference image includes:
[0023] Acquire a preset data set, wherein the data set includes a plurality of images;
[0024] Using a preset number of images in the data set as reference images, and using the remaining images in the data set as model training images;
[0025] Determine camera parameters and a sparse point cloud corresponding to the model training image, and initialize a three-dimensional Gaussian splash model based on the sparse point cloud;
[0026] A rendered image is obtained by using the camera parameters and the three-dimensional Gaussian splash model.
[0027] In one embodiment, the step of constraining the three-dimensional Gaussian splatter model according to the spatial domain regularization term and the frequency domain regularization term to generate an intermediate image based on the rendered image includes:
[0028] calculating a structural similarity loss between the rendered image and the reference image;
[0029] Based on the spatial domain regularization term, the frequency domain regularization term and the structural similarity loss, gradient backpropagation is performed on the attributes of the three-dimensional Gaussian splatter model, and an intermediate image is generated based on the rendered image through the three-dimensional Gaussian splatter model after gradient backpropagation.
[0030] In addition, to achieve the above purpose, the present application also proposes an image rendering system, the image rendering system comprising:
[0031] An image acquisition module, used to acquire a rendered image output by a three-dimensional Gaussian splash model and to acquire a reference image, wherein the rendered image and the reference image belong to the same visual scene;
[0032] A constraint term construction module, used to construct a spatial domain regularization term according to the pixel gradient of the rendered image, and to construct a frequency domain regularization term according to the frequency domain information of the reference image and the frequency domain information of the rendered image;
[0033] a loop module, configured to constrain the three-dimensional Gaussian splatter model to generate an intermediate image based on the rendered image according to the spatial domain regularization term and the frequency domain regularization term, and to cyclically execute the steps of using the intermediate image as the rendered image and constructing the spatial domain regularization term according to the pixel gradient of the rendered image until the number of times the intermediate image is generated reaches a number threshold;
[0034] The output module is used to render the image to be rendered by using the three-dimensional Gaussian splash model to obtain an image rendering result.
[0035] In addition, to achieve the above objectives, the present application also proposes a computer device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image rendering method described above.
[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image rendering method described above are implemented.
[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the image rendering method described above are implemented.
[0038] In an embodiment of the present application, a rendered image output by a three-dimensional Gaussian splash model is obtained, and a reference image is obtained, wherein the rendered image and the reference image belong to the same visual scene, and a basis for further optimizing the rendered image can be obtained, that is, a reference image is obtained; then a spatial domain regularization term is constructed according to the pixel gradient of the rendered image, and a frequency domain regularization term is constructed according to the frequency domain information of the reference image and the frequency domain information of the rendered image, and a basis for constraining the three-dimensional Gaussian splash model to generate an image closer to the reference image can be obtained according to the differences between the reference image and the rendered image in the spatial domain and the frequency domain; then the three-dimensional Gaussian splash model is constrained according to the spatial domain regularization term and the frequency domain regularization term to generate an intermediate image based on the rendered image, and the steps of taking the intermediate image as the rendered image and constructing the spatial domain regularization term according to the pixel gradient of the rendered image are cyclically executed until the number of times the intermediate image is generated reaches a number threshold, and the three-dimensional Gaussian splash model can be prompted to continuously optimize the rendered image according to the continuously updated spatial domain regularization term and the frequency domain regularization term in a cyclic manner; finally, the image to be rendered is rendered by the three-dimensional Gaussian splash model to obtain an image rendering result.
[0039] Therefore, the present application can constrain the three-dimensional Gaussian splash model based on the spatial domain regularization term and the frequency domain regularization term to generate an image with high restoration and realism based on the reference image. Therefore, for the image that needs to be rendered, the image rendering result can be generated by the constrained three-dimensional Gaussian splash model, so as to achieve the purpose of improving the rendering quality of the three-dimensional Gaussian splash technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0042] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the image rendering method of the present application;
[0043] Figure 2 A schematic diagram of a flow chart provided for the second embodiment of the image rendering method of the present application;
[0044] Figure 3 A brief flowchart of a specific embodiment of the image rendering method of the present application;
[0045] Figure 4 This is a schematic diagram of the module structure of the image rendering system according to an embodiment of the present application;
[0046] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the image rendering method in the embodiment of the present application.
[0047] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0048] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0049] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0050] 3D Gaussian splatting is a technique used to create point cloud rendering in computer graphics. Specifically, 3D Gaussian splatting can process large-scale point cloud data and generate realistic images or videos.
[0051] However, although the three-dimensional Gaussian splattering technology can render realistic images, the rendered images are still quite different from the real images.
[0052] Therefore, how to improve the rendering quality of the three-dimensional Gaussian splattering technology is a technical problem that needs to be solved by those skilled in the art.
[0053] To solve the above problems, in an embodiment of the present application, a rendered image output by a three-dimensional Gaussian splash model is obtained, and a reference image is obtained, wherein the rendered image and the reference image belong to the same visual scene, and a basis for further optimizing the rendered image can be obtained, that is, a reference image is obtained; then a spatial domain regularization term is constructed according to the pixel gradient of the rendered image, and a frequency domain regularization term is constructed according to the frequency domain information of the reference image and the frequency domain information of the rendered image, and according to the differences between the reference image and the rendered image in the spatial domain and the frequency domain, a basis for constraining the three-dimensional Gaussian splash model to generate an image closer to the reference image is obtained; then, according to the spatial domain regularization term and the frequency domain regularization term, the three-dimensional Gaussian splash model is constrained to generate an intermediate image based on the rendered image, and the steps of taking the intermediate image as the rendered image and constructing the spatial domain regularization term according to the pixel gradient of the rendered image are cyclically executed until the number of times the intermediate image is generated reaches a number threshold, and the three-dimensional Gaussian splash model can be prompted to continuously optimize the rendered image according to the continuously updated spatial domain regularization term and the frequency domain regularization term in a cyclic manner; finally, the image to be rendered is rendered by the three-dimensional Gaussian splash model to obtain an image rendering result.
[0054] Therefore, the present application can constrain the three-dimensional Gaussian splash model based on the spatial domain regularization term and the frequency domain regularization term to generate an image with high restoration and realism based on the reference image. Therefore, for the image that needs to be rendered, the image rendering result can be generated by the constrained three-dimensional Gaussian splash model, so as to achieve the purpose of improving the rendering quality of the three-dimensional Gaussian splash technology.
[0055] It should be noted that the execution subject of the image rendering method of the present application may be a computer device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a server, etc. The following takes the server as an example to illustrate various embodiments of the image rendering method of the present application.
[0056] Based on this, the present application embodiment provides an image rendering method, referring to Figure 1 , Figure 1 This is a flowchart diagram of the first embodiment of the image rendering method of the present application.
[0057] In this embodiment, the image rendering method includes steps S10 to S40:
[0058] Step S10, obtaining a rendering image output by a three-dimensional Gaussian splash model, and obtaining a reference image, wherein the rendering image and the reference image belong to the same visual scene;
[0059] It should be noted that the three-dimensional Gaussian splash model is a technology that describes voxels in a scene through 3D Gaussian distribution.
[0060] In this embodiment, the rendered image refers to an image rendered by a three-dimensional Gaussian splash model based on a pre-given image. In the rendered image, the details of the image are relatively blurred. Therefore, it is necessary to optimize the rendered image based on a reference image belonging to the same scene so that the image quality of the rendered image is higher.
[0061] In one embodiment, the above step S10 includes:
[0062] Step S101, obtaining a preset data set, wherein the data set includes a plurality of images;
[0063] It should be noted that the data set is a set of pre-set images used to improve the rendering quality of the three-dimensional Gaussian splatter model.
[0064] Step S102, using a preset number of images in the data set as reference images, and using the remaining images in the data set as model training images;
[0065] It should be noted that the model training image refers to the image used to train the three-dimensional Gaussian splash model.
[0066] In one embodiment, for each image in the data set, thirty percent of the images may be used as reference images, and seventy percent of the images may be used as model training images.
[0067] Step S103, determining the camera parameters and sparse point cloud corresponding to the model training image, and initializing the three-dimensional Gaussian splash model based on the sparse point cloud;
[0068] It is understandable that after obtaining the image used for training the model, the camera parameters and sparse point cloud of the model training image can be obtained through the SFM algorithm (Structure from Motion, a 3D reconstruction technology), and then the 3D Gaussian in the 3D Gaussian splash model is initialized through the point cloud. Specifically, the initialization process is actually to define the properties of each 3D Gaussian model through parameters such as position, rotation matrix, scaling matrix, opacity and color, so that the 3D Gaussian splash model has the ability to generate rendered images.
[0069] Step S104, obtaining a rendered image through the camera parameters and the three-dimensional Gaussian splash model.
[0070] It should be noted that the camera parameters include the position, orientation, focal length, etc. of the camera.
[0071] In this embodiment, after initializing the three-dimensional Gaussian splash model, each three-dimensional Gaussian model can be projected onto the image plane based on the camera parameters to obtain a two-dimensional Gaussian distribution. Then, the two-dimensional Gaussian distribution projected onto the image plane can be converted into pixel values, and then a rendered image can be generated according to the pixel values.
[0072] Thus, based on the preset data set, a rendering image to be optimized and a reference image serving as a basis for optimization can be obtained.
[0073] Step S20, constructing a spatial domain regularization term according to the pixel gradient of the rendered image, and constructing a frequency domain regularization term according to the frequency domain information of the reference image and the frequency domain information of the rendered image;
[0074] It should be noted that the pixel gradient is a vector consisting of two components: the horizontal component and the vertical component. The horizontal component represents the rate of change of the pixel brightness in the horizontal direction, while the vertical component represents the rate of change of the pixel brightness in the vertical direction. Therefore, the pixel gradient describes the direction and rate of change of the pixel brightness in the image. The spatial domain regularization term refers to the constraint formula derived from the spatial domain for constraining the three-dimensional Gaussian splash model to generate images. Correspondingly, the frequency domain regularization term refers to the constraint formula derived from the frequency domain for constraining the three-dimensional Gaussian splash model to generate images.
[0075] Since the image quality of the rendered image is poorer than the image quality actually captured by the camera, the present application re-renders or reconstructs the rendered image based on real images such as reference images to improve the rendering quality of the rendered image.
[0076] In one embodiment, the step S20 includes:
[0077] Step S201, inputting pixel gradients corresponding to respective positions in the rendered image into a preset pixel weight model to obtain pixel weights corresponding to respective positions, wherein the pixel weights in the pixel weight model are proportional to the pixel gradients;
[0078] It is understandable that the larger the pixel gradient of a certain area, the more dramatic the change in pixel brightness representing the area. The more dramatic the pixel brightness change, the more details representing the area. Therefore, for this part, the 3D Gaussian splash model can be controlled to focus on learning, so that the 3D Gaussian splash model can give a rendered image with better reconstruction effect of the detail part.
[0079] In this embodiment, by setting the pixel weight to be proportional to the pixel gradient, when training the three-dimensional Gaussian splash model based on the pixel weight, the three-dimensional Gaussian splash model can focus on learning the details of the image (such as the texture area), thereby obtaining a better image reconstruction effect.
[0080] Step S202, calculating the L1 loss between the rendered image and the reference image, and constructing a spatial domain regularization term according to the L1 loss and each pixel weight.
[0081] It should be noted that the L1 loss is the sum of the absolute differences between the predicted value and the true value. In this embodiment, the L1 loss refers to the sum of the absolute values of the differences between the RGB pixel values at each position in the reference image and the rendered image.
[0082] In this embodiment, after obtaining the pixel weights of each region or each position in the image, a spatial domain regularization term may be constructed using the pixel weights and the L1 loss.
[0083] Exemplarily, the gradient of a pixel at (x, y) in the rendered image l is It can be expressed as:
[0084]
[0085] Among them G x and G y is the partial derivative of the image in the x and y directions, and the Sobel operator can be used to calculate G x and G y The Sobel operator consists of two 3×3 convolution kernels K x and K y They are used to calculate the horizontal and vertical gradients respectively. At this time, G x and G y The expressions are:
[0086] G x (x, y) = I (x, y) * K x ;
[0087] G y (x, y) = I (x, y) * K y ;
[0088] At this time, the gradient size of the pixel at (x, y) in the rendered image I is Through G x and G y The calculation results are:
[0089]
[0090] Then, based on the pre-set hyperparameter T, the expression for the pixel weight at the (x, y) position when calculating the loss can be written as:
[0091]
[0092] Here, ρ is initialized to 0 and gradually increases to 1 with the number of training iterations, so that the learning of high-frequency texture information can be gradually enhanced when the model tends to be stable in the later stage of training.
[0093] In one embodiment, the expression of the spatial domain regularization term Lspatial is:
[0094]
[0095] Among them, ||I(x i ,y i )-I gt (x i ,y i )||1 is the L1 loss between the reference image and the rendered image. The spatial domain regularization term is obtained by performing a Hadamard product between the weight weight and the L1 Loss, and N is the total number of pixels.
[0096] In one embodiment, before the above step S20, the method further includes:
[0097] Step S100, extracting high-frequency frequency domain information in the frequency domain information of the reference image and high-frequency frequency domain information in the frequency domain information of the rendered image through a high-pass filter;
[0098] It is understandable that after the reference image and the rendered image are converted to the frequency domain by Fourier transform, the rendered image can be further optimized according to the difference between the reference image and the rendered image in the frequency domain, so that the reconstruction quality of the rendered image is higher. Specifically, since there is more detailed information in the high-frequency area in the frequency domain, the frequency domain information of the high frequency in the reference image and the rendered image can be extracted by a high-pass filter.
[0099] Based on this, the above step S20 further includes:
[0100] Step S203: construct a frequency domain regularization term according to the high frequency frequency domain information of the reference image and the high frequency frequency domain information of the rendered image.
[0101] Therefore, the present application constructs frequency domain regularization terms by using the high-frequency frequency domain information of the reference image and the high-frequency frequency domain information of the rendered image, so that the three-dimensional Gaussian splatter model can pay more attention to the characteristics of the high-frequency area when reconstructing the image, thereby making the details of the reconstructed image (i.e., the rendered image) closer to the real image.
[0102] In one embodiment, the high-frequency frequency domain information of the reference image includes a first amplitude value and a first phase value, and the high-frequency frequency domain information of the rendered image includes: a second amplitude value and a second phase value;
[0103] The above step S203 includes:
[0104] Step S2031, determining an amplitude difference between the first amplitude value and the second amplitude value, and determining a phase difference between the first phase value and the second phase value;
[0105] Step S2032: construct a frequency domain regularization term based on the amplitude difference and the phase difference.
[0106] In this embodiment, the difference between the reference image and the rendered image can be characterized by the amplitude difference and phase difference between the reference image and the rendered image. Therefore, a frequency domain regularization term can be constructed according to the amplitude difference and phase difference.
[0107] For example, the rendered image and the reference image may first be converted to the frequency domain using Fourier transform, and then the high-frequency frequency domain information corresponding to the rendered image and the reference image in the frequency domain may be extracted using a high-pass filter, and then the frequency domain regularization term may be constructed based on the amplitude difference and the phase difference. In a feasible implementation, the frequency domain regularization term Lfre may be:
[0108]
[0109] Among them, A and Represent the amplitude and phase angle of the rendered image I, H and W represent the height and width of the image, respectively; x and y represent the horizontal and vertical coordinates of the pixel, respectively, where x ranges from 0 to H-1, and y ranges from 0 to W-1, indicating that all pixels of the image are traversed; the ρ here has the same meaning as the ρ that appears in the previous spatial domain regularization term, and it gradually increases with the increase in the number of iterations, A gt and Represents the reference image I gt The amplitude and phase angle of the frequency domain; through the above-mentioned frequency domain regularization term, the high-frequency components of the rendered image can be constrained to be closer to the high-frequency components of the reference image, thereby improving the rendering quality of high-frequency details.
[0110] Step S30, constraining the three-dimensional Gaussian splatter model based on the rendered image according to the spatial domain regularization term and the frequency domain regularization term, and cyclically executing the steps of taking the intermediate image as the rendered image and constructing the spatial domain regularization term according to the pixel gradient of the rendered image until the number of times the intermediate image is generated reaches a number threshold;
[0111] It should be noted that the intermediate image refers to an image generated by the three-dimensional Gaussian splash model based on the rendered image.
[0112] In this embodiment, after obtaining the space domain regularization term and the frequency domain regularization term, the three-dimensional Gaussian splatter model can be constrained to focus on learning the details in the image, thereby generating an intermediate image based on the rendered image (the details of the intermediate image are better than those of the rendered image). In order to make the intermediate image rendered by the three-dimensional Gaussian splatter model closer to the reference image, the steps of using the intermediate image as the rendered image, calculating new space domain regularization terms and frequency domain regularization terms based on the reference image and the new rendered image, and using the new space domain regularization terms and frequency domain regularization terms to generate the intermediate image for the new rendered image can be executed repeatedly until the number of cycles meets the requirements.
[0113] Step S40, rendering the image to be rendered by using the three-dimensional Gaussian splash model to obtain an image rendering result.
[0114] Therefore, when the number of cycles reaches the target, it can be considered that the three-dimensional Gaussian splash model has been optimized. Then, the image to be rendered can be rendered based on the optimized three-dimensional Gaussian splash model to obtain an image rendering result.
[0115] Exemplarily, after the 3D Gaussian splash model is optimized, the camera parameters can be input into the 3D Gaussian splash model, so that the 3D Gaussian splash model can render the image to be rendered based on the camera parameters and output a high-quality image rendering result.
[0116] In an embodiment of the present application, a rendered image output by a three-dimensional Gaussian splash model is obtained, and a reference image is obtained, wherein the rendered image and the reference image belong to the same visual scene, and a basis for further optimizing the rendered image can be obtained, that is, a reference image is obtained; then a spatial domain regularization term is constructed according to the pixel gradient of the rendered image, and a frequency domain regularization term is constructed according to the frequency domain information of the reference image and the frequency domain information of the rendered image, and a basis for constraining the three-dimensional Gaussian splash model to generate an image closer to the reference image can be obtained according to the differences between the reference image and the rendered image in the spatial domain and the frequency domain; then the three-dimensional Gaussian splash model is constrained according to the spatial domain regularization term and the frequency domain regularization term to generate an intermediate image based on the rendered image, and the steps of taking the intermediate image as the rendered image and constructing the spatial domain regularization term according to the pixel gradient of the rendered image are cyclically executed until the number of times the intermediate image is generated reaches a number threshold, and the three-dimensional Gaussian splash model can be prompted to continuously optimize the rendered image according to the continuously updated spatial domain regularization term and the frequency domain regularization term in a cyclic manner; finally, the image to be rendered is rendered by the three-dimensional Gaussian splash model to obtain an image rendering result.
[0117] Therefore, the present application can constrain the three-dimensional Gaussian splash model based on the spatial domain regularization term and the frequency domain regularization term to generate an image with high restoration and realism based on the reference image. Therefore, for the image that needs to be rendered, the image rendering result can be generated by the constrained three-dimensional Gaussian splash model, so as to achieve the purpose of improving the rendering quality of the three-dimensional Gaussian splash technology.
[0118] Furthermore, based on the above-mentioned first embodiment of the image rendering method of the present application, a second embodiment of the image rendering method of the present application is proposed.
[0119] Please refer to Figure 2 In this embodiment, the above step S30 includes:
[0120] Step S301, calculating the structural similarity loss between the rendered image and the reference image;
[0121] It should be noted that structural similarity loss is usually used to evaluate the similarity between two images. Specifically, structural similarity loss combines three factors: brightness similarity, contrast similarity, and structural similarity, and can provide a comprehensive similarity measure. It is understood that the calculation method of structural similarity loss is not limited in this application.
[0122] Step S302: performing gradient backpropagation on the attributes of the three-dimensional Gaussian splatter model based on the spatial domain regularization term, the frequency domain regularization term and the structural similarity loss, and generating an intermediate image based on the rendered image through the three-dimensional Gaussian splatter model after gradient backpropagation.
[0123] It can be understood that in the first embodiment, the purpose of improving the reconstruction quality of the rendered image can be achieved by only constraining the three-dimensional Gaussian splash model by the space domain regularization term and the frequency domain regularization term. In this embodiment, the structural similarity loss is also used as a constraint term of the three-dimensional Gaussian splash model, so that the reconstruction quality of the rendered image can be further improved on the basis of the first embodiment.
[0124] Furthermore, based on the various embodiments of the above-mentioned image rendering method, a specific embodiment of the image rendering method of the present application is proposed.
[0125] Please refer to Figure 3, for the rendered image I, it can be converted to the frequency domain and the spatial domain respectively; similarly, for the reference image Igt, it can also be converted to the frequency domain and the spatial domain respectively; then, in the spatial domain, each pixel-level weight can be obtained according to the pixel gradient at each position in the rendered image, and the rendered image can be subtracted from the reference image to obtain the L1 loss, and then the pixel-level weight and the L1 loss can be Hadamard product to obtain the spatial domain regularization term Lspatial; for the frequency domain, for the expression F(t) of the rendered image in the frequency domain and the expression Fgt(t) of the reference image in the frequency domain, the high-frequency component can be extracted by a high-pass filter, and then the frequency domain regularization term Lfre can be calculated by the phase difference and amplitude difference in the high-frequency component. At this time, the three-dimensional Gaussian splash model can be constrained by the frequency domain regularization term Lfre and the spatial domain regularization term Lspatial to optimize the rendered image I, and when the number of optimizations reaches the set number, the rendered image corresponding to the set number is output as the rendering result. Alternatively, the structural similarity loss LSSIM between the rendered image I and the reference image may be calculated, and then the rendered image I may be optimized based on the structural similarity loss, the frequency domain regularization term, and the space domain regularization term.
[0126] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the image rendering method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.
[0127] This application also provides an image rendering system, please refer to Figure 4 , the image rendering system comprises:
[0128] An image acquisition module 10, used to acquire a rendered image output by a three-dimensional Gaussian splash model, and to acquire a reference image, wherein the rendered image and the reference image belong to the same visual scene;
[0129] A constraint term construction module 20, configured to construct a spatial domain regularization term according to a pixel gradient of the rendered image, and to construct a frequency domain regularization term according to frequency domain information of the reference image and frequency domain information of the rendered image;
[0130] A loop module 30 is used to constrain the three-dimensional Gaussian splash model to generate an intermediate image based on the rendered image according to the spatial domain regularization term and the frequency domain regularization term, and to cyclically execute the steps of using the intermediate image as the rendered image and constructing the spatial domain regularization term according to the pixel gradient of the rendered image until the number of times the intermediate image is generated reaches a number threshold;
[0131] The output module 40 is used to render the image to be rendered by using the three-dimensional Gaussian splash model to obtain an image rendering result.
[0132] In one embodiment, the constraint item construction module 20 is further used for:
[0133] Inputting pixel gradients corresponding to respective positions in the rendered image into a preset pixel weight model to obtain pixel weights corresponding to respective positions, wherein the pixel weights in the pixel weight model are proportional to the pixel gradients;
[0134] An L1 loss between the rendered image and the reference image is calculated, and a spatial domain regularization term is constructed according to the L1 loss and each pixel weight.
[0135] In one embodiment, the image rendering system further includes:
[0136] A high-frequency component extraction module, used for extracting high-frequency frequency domain information in the frequency domain information of the reference image and high-frequency frequency domain information in the frequency domain information of the rendered image through a high-pass filter;
[0137] Based on this, the constraint item building module 20 is also used for:
[0138] A frequency domain regularization term is constructed according to the high frequency frequency domain information of the reference image and the high frequency frequency domain information of the rendered image.
[0139] In one embodiment, the high-frequency frequency domain information of the reference image includes a first amplitude value and a first phase value, and the high-frequency frequency domain information of the rendered image includes: a second amplitude value and a second phase value; the constraint item construction module 20 is further used to:
[0140] determining an amplitude difference between the first amplitude value and the second amplitude value, and determining a phase difference between the first phase value and the second phase value;
[0141] A frequency domain regularization term is constructed based on the amplitude difference and the phase difference.
[0142] In one embodiment, the image acquisition module 10 is further used for:
[0143] Acquire a preset data set, wherein the data set includes a plurality of images;
[0144] Using a preset number of images in the data set as reference images, and using the remaining images in the data set as model training images;
[0145] Determine camera parameters and a sparse point cloud corresponding to the model training image, and initialize a three-dimensional Gaussian splash model based on the sparse point cloud;
[0146] A rendered image is obtained by using the camera parameters and the three-dimensional Gaussian splash model.
[0147] In one embodiment, the circulation module 30 is further used for:
[0148] calculating a structural similarity loss between the rendered image and the reference image;
[0149] Based on the spatial domain regularization term, the frequency domain regularization term and the structural similarity loss, gradient backpropagation is performed on the attributes of the three-dimensional Gaussian splatter model, and an intermediate image is generated based on the rendered image through the three-dimensional Gaussian splatter model after gradient backpropagation.
[0150] The image rendering system provided by the present application adopts the image rendering method in the above embodiment, which can solve the technical problem of how to improve the rendering quality of the three-dimensional Gaussian splashing technology. Compared with the prior art, the beneficial effects of the image rendering system provided by the present application are the same as the beneficial effects of the image rendering method provided by the above embodiment, and other technical features of the image rendering system are the same as the features disclosed in the above embodiment method, which will not be repeated here.
[0151] The present application provides a computer device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image rendering method in the above-mentioned embodiment 1.
[0152] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer device suitable for implementing an embodiment of the present application. Figure 5 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0153] like Figure 5As shown, the computer device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the computer device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the computer device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a computer device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.
[0154] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0155] The computer device provided by the present application adopts the image rendering method in the above embodiment, which can solve the technical problem of how to improve the rendering quality of the three-dimensional Gaussian splashing technology. Compared with the prior art, the beneficial effects of the computer device provided by the present application are the same as the beneficial effects of the image rendering method provided by the above embodiment, and the other technical features in the computer device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0156] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0157] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0158] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the image rendering method in the above-mentioned embodiment.
[0159] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.
[0160] The computer-readable storage medium may be included in the computer device; or may exist independently without being installed in the computer device.
[0161] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a computer device, the computer device: obtains a rendered image output by a three-dimensional Gaussian splash model, and obtains a reference image, wherein the rendered image and the reference image belong to the same visual scene; constructs a spatial domain regularization term according to the pixel gradient of the rendered image, and constructs a frequency domain regularization term according to the frequency domain information of the reference image and the frequency domain information of the rendered image; constrains the three-dimensional Gaussian splash model to generate an intermediate image based on the rendered image according to the spatial domain regularization term and the frequency domain regularization term, and cyclically executes the steps of using the intermediate image as the rendered image, and constructing a spatial domain regularization term according to the pixel gradient of the rendered image until the number of times the intermediate image is generated reaches a number threshold; renders the image to be rendered by the three-dimensional Gaussian splash model to obtain an image rendering result.
[0162] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0163] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0164] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0165] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned image rendering method, and can solve the technical problem of how to improve the rendering quality of the three-dimensional Gaussian splattering technology. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the image rendering method provided in the above-mentioned embodiment, and will not be repeated here.
[0166] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned image rendering method when executed by a processor.
[0167] The computer program product provided by this application can solve the technical problem of how to improve the rendering quality of the three-dimensional Gaussian splattering technology. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the image rendering method provided by the above embodiment, which will not be repeated here.
[0168] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. An image rendering method, characterized in that: The image rendering method comprises: Acquire a rendered image output by a three-dimensional Gaussian splash model, and acquire a reference image, wherein the rendered image and the reference image belong to the same visual scene; constructing a spatial domain regularization term according to the pixel gradient of the rendered image, and constructing a frequency domain regularization term according to the frequency domain information of the reference image and the frequency domain information of the rendered image; Constraining the three-dimensional Gaussian splatter model according to the spatial domain regularization term and the frequency domain regularization term to generate an intermediate image based on the rendered image, and cyclically executing the steps of using the intermediate image as the rendered image and constructing the spatial domain regularization term according to the pixel gradient of the rendered image until the number of times the intermediate image is generated reaches a number threshold; The image to be rendered is rendered by using the three-dimensional Gaussian splash model to obtain an image rendering result.
2. The image rendering method according to claim 1, characterized in that: The step of constructing a spatial domain regularization term according to the pixel gradient of the rendered image comprises: Inputting pixel gradients corresponding to respective positions in the rendered image into a preset pixel weight model to obtain pixel weights corresponding to respective positions, wherein the pixel weights in the pixel weight model are proportional to the pixel gradients; An L1 loss between the rendered image and the reference image is calculated, and a spatial domain regularization term is constructed according to the L1 loss and each pixel weight.
3. The image rendering method according to claim 1, characterized in that: Before the step of constructing a frequency domain regularization term according to the frequency domain information of the reference image and the frequency domain information of the rendered image, the method further includes: Extracting high-frequency frequency domain information in the frequency domain information of the reference image and high-frequency frequency domain information in the frequency domain information of the rendered image through a high-pass filter; The step of constructing a frequency domain regularization term according to the frequency domain information of the reference image and the frequency domain information of the rendered image comprises: A frequency domain regularization term is constructed according to the high frequency frequency domain information of the reference image and the high frequency frequency domain information of the rendered image.
4. The image rendering method according to claim 3, characterized in that: The high-frequency frequency domain information of the reference image includes a first amplitude value and a first phase value, and the high-frequency frequency domain information of the rendered image includes: a second amplitude value and a second phase value; The step of constructing a frequency domain regularization term according to the high-frequency frequency domain information of the reference image and the high-frequency frequency domain information of the rendered image comprises: determining an amplitude difference between the first amplitude value and the second amplitude value, and determining a phase difference between the first phase value and the second phase value; A frequency domain regularization term is constructed based on the amplitude difference and the phase difference.
5. The image rendering method according to claim 1, characterized in that: The step of obtaining a rendered image output by a three-dimensional Gaussian splash model and obtaining a reference image comprises: Acquire a preset data set, wherein the data set includes a plurality of images; Using a preset number of images in the data set as reference images, and using the remaining images in the data set as model training images; Determine camera parameters and a sparse point cloud corresponding to the model training image, and initialize a three-dimensional Gaussian splash model based on the sparse point cloud; A rendered image is obtained by using the camera parameters and the three-dimensional Gaussian splash model.
6. The image rendering method according to claim 1, characterized in that: The step of constraining the three-dimensional Gaussian splatter model according to the spatial domain regularization term and the frequency domain regularization term to generate an intermediate image based on the rendered image comprises: calculating a structural similarity loss between the rendered image and the reference image; Based on the spatial domain regularization term, the frequency domain regularization term and the structural similarity loss, gradient backpropagation is performed on the attributes of the three-dimensional Gaussian splatter model, and an intermediate image is generated based on the rendered image through the three-dimensional Gaussian splatter model after gradient backpropagation.
7. An image rendering system, characterized in that: The image rendering system comprises: An image acquisition module, used to acquire a rendered image output by a three-dimensional Gaussian splash model and to acquire a reference image, wherein the rendered image and the reference image belong to the same visual scene; A constraint term construction module, used to construct a spatial domain regularization term according to the pixel gradient of the rendered image, and to construct a frequency domain regularization term according to the frequency domain information of the reference image and the frequency domain information of the rendered image; a loop module, configured to constrain the three-dimensional Gaussian splatter model to generate an intermediate image based on the rendered image according to the spatial domain regularization term and the frequency domain regularization term, and to cyclically execute the steps of using the intermediate image as the rendered image and constructing the spatial domain regularization term according to the pixel gradient of the rendered image until the number of times the intermediate image is generated reaches a number threshold; The output module is used to render the image to be rendered by using the three-dimensional Gaussian splash model to obtain an image rendering result.
8. A computer device, characterized in that: The computer device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image rendering method according to any one of claims 1 to 6.
9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image rendering method according to any one of claims 1 to 6 are implemented.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the image rendering method according to any one of claims 1 to 6 are implemented.