A 3D scene reconstruction and rendering method based on 3D Gaussian splatting and continuous learning

Abstract

The present invention discloses a 3D scene reconstruction and rendering method based on 3D Gaussian splatting and continuous learning, comprising: confirming a 3D Gaussian field at the Nth moment; the 3D Gaussian field comprising: a 3D Gaussian representation, a visibility pool, and an appearance model; obtaining a structure-invariant mask; setting a training set; training the appearance model with fixed parameters of a pre-trained 3D Gaussian representation; setting the parameters of the appearance model, taking a sparse point cloud of a scene as an initial 3D Gaussian and adding it to the 3D Gaussian representation, training the initial 3D Gaussian and learning a removal factor; setting the 3D Gaussian that should be removed from the pre-trained 3D Gaussian representation to be invisible in the visibility pool at the N+1th moment based on the removal factor; jointly optimizing the appearance model and the initialized 3D Gaussian to obtain an updated 3D Gaussian field; inputting the pose of the image to be rendered and the corresponding time into the updated 3D Gaussian field, and outputting a new perspective image rendered at the moment.

Description

Technical Field

[0001] The present invention belongs to the field of augmented reality and virtual reality, and in particular relates to a three-dimensional scene reconstruction and rendering method based on three-dimensional Gaussian splashing and continuous learning. Background Art

[0002] High-quality scene reconstruction and realistic image rendering are crucial for many applications, such as augmented reality, virtual reality, and robotics. In recent years, neural scene representations, represented by neural radiance fields, have shown excellent performance in novel view synthesis. Although these rendering results are impressive, the same scene often undergoes many changes over time. For example, the same room may have different objects, furniture placement, and lighting conditions at different times. Therefore, it is crucial to update the learned neural scene representation to render images that align with the current scene appearance. In addition, knowledge from previous time steps is valuable, and after updating the learned neural scene representation, information from different times can be retained, allowing high-quality images of the same scene at different times to be rendered when needed.

[0003] Existing techniques combine neural radiance fields and continuous learning for scene reconstruction and image rendering. Although these techniques can update the model as the scene changes and prevent catastrophic forgetting, they still have the following problems:

[0004] 1. The image rendering speed is slow and the image rendering quality is not high;

[0005] 2. Unable to detect changes in the geometric structure of the scene. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the present invention provides a three-dimensional scene reconstruction and rendering method based on three-dimensional Gaussian splashing and continuous learning.

[0007] An embodiment of the present invention provides a method for reconstructing and rendering a three-dimensional scene based on three-dimensional Gaussian splatting and continuous learning, the method comprising:

[0008] Determine the three-dimensional Gaussian field at the Nth moment; the three-dimensional Gaussian field includes: a three-dimensional Gaussian representation, a visibility pool, and an appearance model;

[0009] The RGB image dataset at the Nth moment is obtained based on the three-dimensional Gaussian field at the Nth moment and the pose rendering corresponding to each image in the RGB image dataset at the N+1th moment; the structure-invariant mask is obtained based on the segmentation mask set corresponding to the RGB image dataset at the N+1th moment and the segmentation mask set corresponding to the RGB image dataset at the Nth moment;

[0010] The pose of the RGB image dataset at time 0-N is input into the three-dimensional Gaussian field rendering at time N to obtain the RGB image data corresponding to time 0-N; the RGB image and its pose corresponding to time 0-N, and the RGB image and its pose at time N+1 are used as training sets;

[0011] Fix the parameters of the pre-trained 3D Gaussian representation, set the first loss function, and train the appearance model based on the first loss function, the structure-invariant mask, and the training set to learn the global illumination changes in the geometrically consistent area;

[0012] The parameters of the appearance model are fixed; a second loss function is set, and the acquired sparse point cloud of the scene is used as the initial 3D Gaussian and added to the 3D Gaussian representation. The initial 3D Gaussian is trained based on the second loss function and the training set, and a removal factor is learned. Based on the removal factor, the 3D Gaussian that should be removed from the pre-trained 3D Gaussian representation is set to be invisible in the visibility pool at the N+1th moment, thereby learning the geometric structure changes of the scene;

[0013] Setting a third loss function, jointly optimizing the appearance model and the initialized three-dimensional Gaussian based on the third loss function and the training set, filtering redundant three-dimensional Gaussians, and obtaining an updated three-dimensional Gaussian field;

[0014] The pose of the image to be rendered and the corresponding time are input into the updated three-dimensional Gaussian field, and the new perspective image rendered at that moment is output.

[0015] The beneficial effects of the present invention are:

[0016] The present invention provides a 3D scene reconstruction and rendering method based on 3D Gaussian splatting and continuous learning. In the process of updating the 3D Gaussian field, the present invention first fixes the parameters of a pre-trained 3D Gaussian representation, trains an appearance model to learn global illumination changes in geometrically consistent areas, then fixes the parameters of the appearance model, uses the acquired scene sparse point cloud as the initial 3D Gaussian and adds it to the 3D Gaussian representation, trains the initial 3D Gaussian and learns a removal factor. Based on the removal factor, the 3D Gaussian that should be removed from the pre-trained 3D Gaussian representation is set to be invisible in the visibility pool at the N+1th moment, thereby learning the geometric structure changes of the scene. This allows the present invention to detect geometric structure changes of the scene, and after the scene changes, new data can be collected to train and update the trained 3D Gaussian representation without forgetting existing scene information. The present invention can achieve real-time rendering speed and higher new perspective synthesis quality, and can also obtain changes in scene geometry over time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a schematic flow diagram of the present invention;

[0019] Figure 2 It is a schematic diagram of the overall network framework of the present invention;

[0020] Figure 3 is a schematic diagram of obtaining a structure-invariant mask according to the present invention;

[0021] Figure 4 This is a comparison chart of the new perspective synthesis results of the present invention on the WAT public dataset and the existing public methods;

[0022] Figure 5 This is a diagram showing the actual effect of detecting changes in the geometric structure of a scene according to the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0025] like Figure 1 and Figure 2 As shown, a 3D scene reconstruction and rendering method based on 3D Gaussian splatting and continuous learning includes the following steps:

[0026] Step S1, confirming the three-dimensional Gaussian field at the Nth moment; the three-dimensional Gaussian field includes: a three-dimensional Gaussian representation, a visibility pool, and an appearance model;

[0027] The three-dimensional Gaussian representation is composed of a number of three-dimensional Gaussians (generally composed of hundreds of thousands to millions of three-dimensional Gaussians), and only the three-dimensional Gaussian representation at time 0 is obtained by training with three-dimensional Gaussian splashing;

[0028] The visibility pool is used to store the mapping relationship between each moment and the visible three-dimensional Gaussian corresponding to the moment;

[0029] The appearance model includes a multi-layer hash table and a multi-layer perceptron; the multi-layer hash table takes the position of a three-dimensional Gaussian and the time of the image to be rendered as input and outputs feature values; after all the features are spliced ​​together, they are input into the multi-layer perceptron to obtain the spherical harmonic coefficient correction value and scale correction value of the three-dimensional Gaussian.

[0030] Step S2: Rendering is performed based on the three-dimensional Gaussian field at the Nth moment and the pose corresponding to each image in the RGB image dataset at the N+1th moment to obtain the RGB image dataset at the Nth moment; and the segmentation mask set corresponding to the RGB image dataset at the N+1th moment and the segmentation mask set corresponding to the RGB image dataset at the Nth moment are processed by a dual threshold detection method to obtain a structure-invariant mask.

[0031] The process of obtaining the structure-invariant mask includes:

[0032] In step S201, the RGB image dataset at the N+1th moment and the RGB image dataset at the Nth moment are respectively input into the Segment Anything Model (SAM) to obtain a set of overlapping segmentation masks for each image in the RGB image dataset at the N+1th moment and a set of overlapping segmentation masks for each image in the RGB image dataset at the Nth moment.

[0033] Step S202: overlapping each segmentation mask in the set of overlapping segmentation masks for each image in the RGB image dataset at the N+1th time on the corresponding image plane in descending order of area, to obtain a set of non-overlapping segmentation masks for each image in the RGB image dataset at the N+1th time;

[0034] Each segmentation mask in the set of overlapping segmentation masks of each image in the RGB image dataset at the Nth moment is overlapped on the corresponding image plane in descending order of area, to obtain a set of non-overlapping segmentation masks for each image in the RGB image dataset at the Nth moment.

[0035] In step S203, a non-overlapping segmentation mask set of each image in the RGB image dataset at the N+1th moment is compared based on a dual threshold detection method, and a non-overlapping segmentation mask set corresponding to each image in the RGB image dataset at the Nth moment is obtained to obtain a structurally invariant mask for each image in the RGB image dataset at the N+1th moment.

[0036] Specifically, if Figure 3 As shown, step S203 includes:

[0037] In step S20301, an intersection-over-union (IoU) is calculated for each segmentation mask in the set of non-overlapping segmentation masks for each image in the RGB image dataset at the N+1th moment and the segmentation mask at the corresponding position in the corresponding image in the RGB image dataset at the Nth moment. A provisional structural invariant mask for each image in the RGB image dataset at the N+1th moment is obtained by combining the segmentation masks in the set of non-overlapping segmentation masks for each image in the RGB image dataset at the N+1th moment whose IoU ratio is greater than a threshold (in this example, the threshold is 0.6).

[0038] The intersection-and-union (IoU) of each segmentation mask in the set of non-overlapping segmentation masks for each image in the RGB image dataset at the Nth moment is calculated with the segmentation mask at the corresponding position in the corresponding image in the RGB image dataset at the N+1th moment. A tentative structure-invariant mask for each image in the RGB image dataset at the Nth moment is obtained by merging the segmentation masks in the set of non-overlapping segmentation masks for each image in the RGB image dataset at the Nth moment whose IoU ratio is greater than a threshold (in this example, the threshold is 0.6).

[0039] Step S20302 : Obtain a structure-invariant mask for each image in the RGB image dataset at the N+1th moment by intersecting the tentative structure-invariant mask for each image in the RGB image dataset at the N+1th moment with the tentative structure-invariant mask for each corresponding image in the RGB image dataset at the Nth moment.

[0040] Step S3: The RGB images and their poses corresponding to time 0-N, and the RGB image and its pose at time N+1 are used as training sets.

[0041] Among them, the RGB images corresponding to time 0-N are generated by a generative replay strategy; specifically, the generative replay strategy generates images of past time moments (i.e., replays the RGB images corresponding to time 0-N), saves the RGB image poses of past time moments, and uses the three-dimensional Gaussian representation of time N to render the RGB images of the past time moments.

[0042] Step S4, fixing the parameters of the three-dimensional Gaussian represented by the pre-trained three-dimensional Gaussian; setting a first loss function, and training the appearance model based on the structure-invariant mask to learn the global illumination changes of the geometrically consistent area.

[0043] Among them, the expression of the first loss function is as follows:

[0044]

[0045]

[0046]

[0047] Where C represents the RGB image, C n Represents the pixel value in the RGB image, represents an RGB image rendered with a 3D Gaussian representation, Represents the pixel value in the RGB image rendered by the three-dimensional Gaussian representation, M f represents the structure-invariant mask, ⊙ represents element-wise multiplication, and λ is the coefficient of L1 distance loss and SSIM loss.

[0048] Step S5, fix the parameters of the appearance model; set a second loss function, use the scene sparse point cloud as the initial three-dimensional Gaussian and add it to the three-dimensional Gaussian representation, train the initial three-dimensional Gaussian based on the second loss function and learn the removal factor; based on the removal factor, set the three-dimensional Gaussian that should be removed from the pre-trained three-dimensional Gaussian representation to be invisible in the visibility pool at the N+1th moment, thereby learning the geometric structure changes of the scene.

[0049] Among them, the expression of the second loss function is as follows:

[0050] L nd =(1-λ)L1+λL D-SSIM +L reg

[0051]

[0052]

[0053] L reg =λ2(1-ψ(m))ψ(m)+λ3BCE(ψ(m),1)

[0054] Where C represents the RGB image provided by the training set, C n Represents the pixel value in the RGB image provided by the training set, represents a 3D Gaussian representation of the rendered RGB image, represents the pixel value in the RGB image rendered by the three-dimensional Gaussian representation, λ is the coefficient of L1 distance loss and SSIM loss; regularization loss L reg The purpose of the first term is to make the removal factor after activation tend to 0 or 1, λ2 is the coefficient of the first term; the regularization term loss L reg The purpose of the second term is to prevent unnecessary three-dimensional Gaussians from being removed, and λ3 is the coefficient of the second term.

[0055] The removal factor is activated by the activation function and multiplied by the opacity of the pre-trained three-dimensional Gaussian during the rendering process; the activation function ψ(m) of the removal factor m is as follows:

[0056]

[0057] This activation function can make the removal factor m more likely to tend to 0 or 1, thereby achieving the purpose of removing the 3D Gaussians that should disappear at the current moment; after the current step training is completed, the 3D Gaussians whose activated removal factor is less than the threshold (in this example, the threshold is 0.01) are used as candidate 3D Gaussians, and these candidate 3D Gaussians are clustered and filtered using the DBSCAN algorithm. Then, the 3D bounding box of each clustered 3D Gaussian point cloud is calculated, and finally, all 3D Gaussians in these bounding boxes are set to invisible in the visibility pool at the N+1th moment.

[0058] It should be noted that learning the global illumination changes in the geometrically consistent areas in step S4 is the basis for adjusting the geometric structure. By learning the global illumination changes in the geometrically consistent areas, the impact of the global illumination changes of the scene on the learned removal factor can be reduced while allowing the initialized three-dimensional Gaussian to be mainly concentrated in the area where the geometric structure changes.

[0059] It should be noted that for the three-dimensional Gaussian that has been trained at the Nth moment and before, no further training will be performed at the N+1 moment and subsequent moments. This method divides the scene changes into changes in global illumination and changes in the scene geometry, and learns the changes in global illumination by training the appearance model; the changes in the scene geometry can be divided into new objects and removed objects, and the changes in the scene geometry are learned by learning the initial three-dimensional Gaussian and the removal factor respectively.

[0060] Step S6: setting a third loss function, jointly optimizing the appearance model and the initialized three-dimensional Gaussian, and filtering redundant three-dimensional Gaussians using a three-dimensional Gaussian filtering strategy based on importance to obtain an updated three-dimensional Gaussian field;

[0061] The expression of the third loss function is as follows:

[0062] L rd =(1-λ)L1+λL D-SSIM

[0063] The importance-based three-dimensional Gaussian filtering strategy traverses the pose corresponding to the RGB image trained at the N+1th moment and calculates the product of the maximum opacity and transmittance of each three-dimensional Gaussian during the rendering process as the importance of each three-dimensional Gaussian. Then, the three-dimensional Gaussian with an importance lower than 0.05 is filtered to remove redundant three-dimensional Gaussians.

[0064] The specific method of training the three-dimensional Gaussian field is to iteratively update the parameters according to the gradient backpropagation method and use GPU for acceleration until the number of network iterations meets the requirements and then stop training.

[0065] Step S7: input the pose of the image to be rendered and the corresponding time into the updated three-dimensional Gaussian field, and output the new perspective image rendered at the time.

[0066] The final results of the above method and the results of the existing methods are as follows Figure 4 As shown in the figure, the present invention can render new perspective synthetic images at different times, and is closer to the true RGB image than the previous method. The structure change of the scene is detected as follows Figure 5 As shown, the black dots represent three-dimensional Gaussians whose structures have not changed, the red dots represent three-dimensional Gaussians that should disappear at the current moment compared to the previous moment, and the green dots represent three-dimensional Gaussians that are newly added at the current moment compared to the previous moment.

[0067] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change of the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A 3D scene reconstruction and rendering method based on 3D Gaussian splatting and continuous learning, characterized in that: The method comprises: Determine the three-dimensional Gaussian field at the Nth moment; the three-dimensional Gaussian field includes: a three-dimensional Gaussian representation, a visibility pool, and an appearance model; The RGB image dataset at the Nth moment is obtained based on the three-dimensional Gaussian field at the Nth moment and the pose rendering corresponding to each image in the RGB image dataset at the N+1th moment; the structure-invariant mask is obtained based on the segmentation mask set corresponding to the RGB image dataset at the N+1th moment and the segmentation mask set corresponding to the RGB image dataset at the Nth moment; The pose of the RGB image dataset at time 0-N is input into the three-dimensional Gaussian field rendering at time N to obtain the RGB image data corresponding to time 0-N; the RGB image and its pose corresponding to time 0-N, and the RGB image and its pose at time N+1 are used as training sets; Fix the parameters of the pre-trained 3D Gaussian representation, set the first loss function, and train the appearance model based on the first loss function, the structure-invariant mask, and the training set to learn the global illumination changes in the geometrically consistent area; The parameters of the appearance model are fixed; a second loss function is set, and the acquired sparse point cloud of the scene is used as the initial 3D Gaussian and added to the 3D Gaussian representation. The initial 3D Gaussian is trained based on the second loss function and the training set, and a removal factor is learned. Based on the removal factor, the 3D Gaussian that should be removed from the pre-trained 3D Gaussian representation is set to be invisible in the visibility pool at the N+1th moment, thereby learning the geometric structure changes of the scene; Setting a third loss function, jointly optimizing the appearance model and the initialized three-dimensional Gaussian based on the third loss function and the training set, filtering redundant three-dimensional Gaussians, and obtaining an updated three-dimensional Gaussian field; The pose of the image to be rendered and the corresponding time are input into the updated three-dimensional Gaussian field, and the new perspective image rendered at that moment is output.

2. The 3D scene reconstruction and rendering method based on 3D Gaussian splatting and continuous learning according to claim 1, characterized in that: The process of obtaining a structure-invariant mask based on the segmentation mask set corresponding to the RGB image dataset at the N+1th moment and the segmentation mask set corresponding to the RGB image dataset at the Nth moment includes: The RGB image dataset at the N+1th moment and the RGB image dataset at the Nth moment are respectively input into the image segmentation model to obtain a set of overlapping segmentation masks for each image in the RGB image dataset at the N+1th moment and a set of overlapping segmentation masks for each image in the RGB image dataset at the Nth moment; Each segmentation mask in the set of overlapping segmentation masks for each image in the RGB image dataset at the N+1th moment is sequentially overlapped on the corresponding image plane in descending order of area, to obtain a set of non-overlapping segmentation masks for each image in the RGB image dataset at the N+1th moment; Each segmentation mask in the set of overlapping segmentation masks of each image in the RGB image dataset at the Nth moment is sequentially overlapped on the corresponding image plane in descending order of area, to obtain a set of non-overlapping segmentation masks for each image in the RGB image dataset at the Nth moment; The structure-invariant mask of each image in the RGB image dataset at the N+1th moment is obtained by comparing the non-overlapping segmentation mask set of each image in the RGB image dataset at the N+1th moment and the non-overlapping segmentation mask set corresponding to each image in the RGB image dataset at the Nth moment.

3. The 3D scene reconstruction and rendering method based on 3D Gaussian splatting and continuous learning according to claim 2, characterized in that: The process of comparing the non-overlapping segmentation mask set of each image in the RGB image dataset at the N+1th time and obtaining the structure-invariant mask of each image in the RGB image dataset at the N+1th time by using the non-overlapping segmentation mask set corresponding to each image in the RGB image dataset at the Nth time includes: Calculate the intersection-and-union ratio (IoU) of each segmentation mask in the set of non-overlapping segmentation masks for each image in the RGB image dataset at the N+1th moment with the segmentation mask at the corresponding position in the corresponding image in the RGB image dataset at the Nth moment, and obtain a tentative structure-invariant mask for each image in the RGB image dataset at the N+1th moment by merging the segmentation masks in the set of non-overlapping segmentation masks for each image in the RGB image dataset at the N+1th moment whose IoU ratio is greater than a threshold; Calculate the intersection-and-union ratio (IoU) of each segmentation mask in the set of non-overlapping segmentation masks for each image in the RGB image dataset at the Nth moment with the segmentation mask at the corresponding position in the corresponding image in the RGB image dataset at the N+1th moment, and obtain a tentative structure-invariant mask for each image in the RGB image dataset at the Nth moment by merging the segmentation masks in the set of non-overlapping segmentation masks for each image in the RGB image dataset at the Nth moment with an IoU ratio greater than a threshold; The structure-invariant mask of each image in the RGB image dataset at the N+1th moment is obtained by intersecting the tentative structure-invariant mask of each image in the RGB image dataset at the N+1th moment with the tentative structure-invariant mask of each corresponding image in the RGB image dataset at the Nth moment.

4. The 3D scene reconstruction and rendering method based on 3D Gaussian splatting and continuous learning according to claim 1, characterized in that: The expression of the first loss function is as follows: Where C represents the RGB image, C n Represents the pixel value in the RGB image, represents an RGB image rendered with a 3D Gaussian representation, Represents the pixel value in the RGB image rendered by the three-dimensional Gaussian representation, M f represents the structure-invariant mask, ⊙ represents element-wise multiplication, and λ is the coefficient of L1 distance loss and SSIM loss.

5. The 3D scene reconstruction and rendering method based on 3D Gaussian splatting and continuous learning according to claim 1, characterized in that: The expression of the second loss function is as follows: THE nd =(1-λ)L1+λL D-SSIM +L reg L reg =λ2(1-ψ(m))ψ(m)+λ3BCE(ψ(m),1) Where C represents the RGB image provided by the training set, C n Represents the pixel value in the RGB image provided by the training set, represents a 3D Gaussian representation of the rendered RGB image, represents the pixel value in the RGB image rendered by the three-dimensional Gaussian representation, λ is the coefficient of L1 distance loss and SSIM loss; regularization loss L reg The purpose of the first term is to make the removal factor after activation tend to 0 or 1, λ2 is the coefficient of the first term; the regularization term loss L reg The purpose of the second term is to prevent unnecessary three-dimensional Gaussians from being removed, and λ3 is the coefficient of the second term.

6. The 3D scene reconstruction and rendering method based on 3D Gaussian splatting and continuous learning according to claim 1, characterized in that: The removal factor is activated by the activation function and multiplied by the opacity of the pre-trained three-dimensional Gaussian during the image rendering process; Among them, the expression of the activation function of the removed factor is as follows: In the formula, m represents the removal factor, and e represents the natural constant; The three-dimensional Gaussian whose removal factor after activation is less than the threshold is taken as the candidate three-dimensional Gaussian; All candidate 3D Gaussians are clustered and filtered, and the 3D bounding box of each clustered 3D Gaussian point cloud is calculated. All 3D Gaussians in these bounding boxes are set to be invisible at the N+1th moment in the visibility pool.

7. The 3D scene reconstruction and rendering method based on 3D Gaussian splatting and continuous learning according to claim 1, characterized in that: The expression of the third loss function is as follows: L rd =(1-λ)L1+λL D-SSIM In the formula, λ represents the weight coefficient, L1 represents the L1 distance loss, and L D-SSIM represents the SSIM loss.

8. The 3D scene reconstruction and rendering method based on 3D Gaussian splatting and continuous learning according to claim 1, characterized in that: The process of updating the three-dimensional Gaussian field also includes: By traversing the pose corresponding to the RGB image trained at the N+1th moment, the product of the maximum opacity and transmittance of each 3D Gaussian in the rendering process is calculated as the importance of each 3D Gaussian; Filter the three-dimensional Gaussians whose importance is lower than the threshold, thereby removing redundant three-dimensional Gaussians.

Images