Three-dimensional static background reconstruction method, device, equipment and storage medium

Abstract

The application discloses a three-dimensional static background reconstruction method, device and equipment and a storage medium. Real images and camera parameters of a scene are acquired; initialized 3D Gaussian parameters and the camera parameters are input into a 3D GS renderer; a rendered image output by the 3D GS renderer and the real images are input into a mask predictor; the 3D Gaussian parameters and the mask predictor are trained; the trained 3D Gaussian parameters and the camera parameters are input into the 3D GS renderer, and a rendered image of a reconstructed static background is output. The application is based on the 3D GS renderer and the mask predictor, uses 3D GS to reconstruct a static background from scene data containing dynamic objects, and does not depend on artificial annotation information.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-dimensional scene reconstruction, and particularly relates to a three-dimensional static background reconstruction method, device, equipment and storage medium. BACKGROUND

[0002] Three-dimensional scene reconstruction refers to a process of reconstructing original three-dimensional information according to multiple pictures of different perspectives taken in a scene. The reconstruction result can provide support for fields such as visualization simulation, virtual reality and game modeling. The 3DGS method has great advantages in three-dimensional reconstruction fidelity and rendering efficiency, and has become a research hotspot. However, the 3DGS assumes that the scene is static, and is not applicable to dynamic scenes, i.e., scenes containing dynamic objects. Dynamic objects cause artifacts in the reconstructed scene after 3DGS.

[0003] The existing solution is to model the dynamic objects and the static background in the dynamic scene respectively, and then reconstruct the entire dynamic scene in a manner of merging and rendering, i.e., reconstructing both the static background and the dynamic objects.

[0004] Specifically, the scene is divided into 1+N blocks of Gaussian point clouds by using the scene point cloud and the bounding box (bbox) and trajectory of the dynamic object: 1 block represents the Gaussian point cloud of the dynamic background and N blocks represent the Gaussian point clouds of N dynamic objects. The parameters of the 1+N blocks of Gaussian point clouds are similar to those of 3DGS. The coordinates of the dynamic objects in the static background are calculated according to the tracking trajectory of the dynamic objects, the 1+N blocks of Gaussian point clouds are combined together, and then unified rendering is performed.

[0005] However, this technology requires the bbox and trajectory of the dynamic object to complete the modeling strategy of dynamic and static separation. However, the information of the bbox and trajectory of the dynamic object depends on manual annotation, which is time-consuming and laborious. SUMMARY

[0006] To solve the above technical problems, the present application provides a three-dimensional static background reconstruction method, device, equipment and storage medium.

[0007] To achieve the above purpose, the technical scheme of the present application is as follows:

[0008] In a first aspect, the present application discloses a three-dimensional static background reconstruction method, comprising:

[0009] Step S1: obtaining real images and camera parameters of a scene;

[0010] Step S2: inputting the initialized 3D Gaussian parameters and camera parameters into a 3DGS renderer, the 3DGS renderer being used to output a reconstructed rendering image;

[0011] Step S3: Input the rendered image and the real image output by the 3DGS renderer into the mask predictor. The mask predictor is used to output a mask image. The value of each pixel on the mask image can reflect the reconstruction effect of the rendered image and distinguish between static backgrounds and dynamic objects.

[0012] Step S4: Train the 3D Gaussian parameters and mask predictor;

[0013] Step S5: Input the trained 3D Gaussian parameters and camera parameters into the 3DGS renderer and output the reconstructed static background rendering image.

[0014] Based on the above technical solution, the following improvements can be made:

[0015] As a preferred embodiment, in step S3, the value of each pixel in the mask image is between 0 and 1.

[0016] Furthermore, the lower the value of a pixel, the more it indicates that the pixel is occupied by a dynamic object;

[0017] The higher the pixel value, the more it indicates that the pixel is occupied by a static background.

[0018] As a preferred approach, step S4 calculates the loss through the following steps:

[0019] Step A: Calculate the Hadamard product of the rendered image and the real image with the mask image, respectively;

[0020]

[0021] Among them: I render and I real These are rendered images and real images, respectively.

[0022] M is the mask image;

[0023] ⊙ is for Hadamarda accumulation;

[0024] Step B: Based on the calculation results of Step A, obtain the loss using the following formula;

[0025]

[0026] in: For L1 loss;

[0027] For SSIM loss;

[0028] λ SSIM This is the loss coefficient for SSIM.

[0029] As a preferred approach, during the training of the 3D Gaussian parameters and the mask predictor, the gradient of the mask predictor is not propagated to the rendered image during backpropagation.

[0030] Secondly, the present invention also discloses a three-dimensional static background reconstruction device, comprising:

[0031] The acquisition module is used to acquire real images of the scene and camera parameters;

[0032] The rendering module is used to input the initialized 3D Gaussian parameters and camera parameters into the 3DGS renderer, which is used to output the reconstructed rendered image.

[0033] The prediction module is used to input the rendered image and the real image output by the 3DGS renderer into the mask predictor. The mask predictor is used to output a mask image. The value of each pixel on the mask image can reflect the reconstruction effect of the rendered image and distinguish between static backgrounds and dynamic objects.

[0034] The training module is used to train the 3D Gaussian parameters and mask predictor;

[0035] The reconstruction module is used to input the trained 3D Gaussian parameters and camera parameters into the 3DGS renderer and output the reconstructed static background rendered image.

[0036] As a preferred approach, in the prediction module, the value of each pixel on the mask image is between 0 and 1.

[0037] Furthermore, the lower the value of a pixel, the more it indicates that the pixel is occupied by a dynamic object;

[0038] The higher the pixel value, the more it indicates that the pixel is occupied by a static background.

[0039] As a preferred option, the training module includes:

[0040] The first computing unit is used to calculate the Hadamard product of the rendered image and the real image with the mask image, respectively.

[0041]

[0042] Among them: I render and I real These are rendered images and real images, respectively.

[0043] M is the mask image;

[0044] ⊙ is for Hadamarda accumulation;

[0045] The second calculation unit is used to obtain the loss based on the calculation results of the first calculation unit using the following formula;

[0046]

[0047] in: For L1 loss;

[0048] For SSIM loss;

[0049] λ SSIM This is the loss coefficient for SSIM.

[0050] As a preferred approach, during the training of the 3D Gaussian parameters and the mask predictor, the gradient of the mask predictor is not propagated to the rendered image during backpropagation.

[0051] Thirdly, the present invention also discloses a computing device, comprising:

[0052] One or more processors;

[0053] Memory;

[0054] And one or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors, and the one or more programs include instructions for any of the three-dimensional static background reconstruction methods described above.

[0055] Fourthly, the present invention also discloses a storage medium storing one or more computer-readable programs, the one or more programs including instructions adapted to be loaded by a memory and executed by any of the above-described three-dimensional static background reconstruction methods.

[0056] This invention discloses a method, apparatus, device, and storage medium for three-dimensional static background reconstruction, which has the following beneficial effects:

[0057] This invention is based on the 3DGS renderer and mask predictor, and uses 3DGS to reconstruct static backgrounds from scene data containing dynamic objects without relying on manual annotation information. Attached Figure Description

[0058] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A flowchart of a three-dimensional static background reconstruction method provided in an embodiment of the present invention.

[0060] Figure 2A flowchart illustrating the three-dimensional static background reconstruction method provided in this embodiment of the invention.

[0061] Figure 3 A block diagram of a three-dimensional static background reconstruction device provided in an embodiment of the present invention.

[0062] Figure 4 A block diagram of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0063] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Using ordinal numbers such as “first,” “second,” “third,” etc. to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, sequence, or any other way.

[0066] Furthermore, the expression "includes" is an "open-ended" expression, which means only that there is a corresponding component or step, and should not be interpreted as excluding additional components or steps.

[0067] To achieve the objectives of this invention, in some embodiments of the three-dimensional static background reconstruction method, the three-dimensional static background reconstruction method is based on 3DGS, such as... Figures 1-2 As shown, it includes:

[0068] Step S101: Obtain the real image of the scene and camera parameters;

[0069] Step S102: Input the initialized 3D Gaussian parameters and camera parameters into the 3DGS renderer. The 3DGS renderer is used to output the reconstructed rendered image.

[0070] Step S103: Input the rendered image and real image output by the 3DGS renderer into the mask predictor. The mask predictor is used to output a mask image. The value of each pixel on the mask image can reflect the reconstruction effect of the rendered image and distinguish between static backgrounds and dynamic objects.

[0071] Step S104: Train the 3D Gaussian parameters and mask predictor;

[0072] Step S105: Input the trained 3D Gaussian parameters and camera parameters into the 3DGS renderer and output the reconstructed static background rendering image.

[0073] Each of the above steps will be explained in detail below.

[0074] Step S101: Obtain the real image of the scene and camera parameters. Camera parameters include, but are not limited to, information such as camera position, tilt angle, focal length, and optical center offset. These parameters determine the projection method from the 3D scene to the 2D image.

[0075] In step S102, the 3D Gaussian parameters include, but are not limited to, information such as the position, shape, opacity, and color of the Gaussian points. Multiple Gaussian points constitute the 3D representation of the entire scene.

[0076] The 3DGS renderer takes 3D Gaussian parameters and camera parameters as input to render a 3D scene image under a projection method determined by the camera parameters. Specifically, the 3DGS renderer first projects the 3D Gaussian onto a 2D plane according to the projection method, and then superimposes its color information based on the occlusion relationship and opacity of the Gaussian to calculate the RGB value of each pixel in the rendered image.

[0077] The 3DGS renderer is essentially a rendering algorithm that renders 3D Gaussian parameters into a 2D image according to camera parameters. This invention uses real images of the scene and corresponding camera parameters as training data to train the 3D Gaussian parameters. It is worth noting that the rendering process of the 3DGS renderer is well known to those skilled in the art and will not be described in detail here.

[0078] Step S103 mainly involves a mask predictor, which consists of a neural network that takes a rendered image and a real image as input and outputs a mask image. The size of the mask image is the same as the size of the input images (rendered image and real image), specifically (H, W), and the value of each pixel on the mask image is between 0 and 1.

[0079] The lower the value of a pixel (indicating that the 3D scene area corresponding to that pixel location is reconstructed worse), the more likely that the pixel is occupied by a dynamic object;

[0080] The higher the value of a pixel (indicating that the 3D scene area corresponding to that pixel location is reconstructed better), the more it represents that the pixel is occupied by the static background.

[0081] In the mask image, the whiter the color, the closer the value is to 0.

[0082] To ensure stability during 3D Gaussian parameter training, the mask predictor does not propagate gradients to the rendered image during backpropagation.

[0083] It is worth noting that the neural network architecture of the mask predictor can be, but is not limited to, using U-Net, whose output image can be interpolated to obtain a final mask image of the same size as the input image.

[0084] Step S104 calculates the loss through the following steps:

[0085] Step A: Calculate the Hadamard product of the rendered image and the real image with the mask image, respectively;

[0086]

[0087] Among them: I render and I real These are rendered images and real images, respectively.

[0088] M is the mask image;

[0089] ⊙ is for Hadamarda accumulation;

[0090] Step B: Based on the calculation results of Step A, obtain the loss using the following formula;

[0091]

[0092] in: For L1 loss;

[0093] The loss is SSIM (StructuralSimilarityIndexMeasure).

[0094] λ SSIM The loss coefficient for SSIM can be set to 0.2.

[0095] This invention does not directly calculate the L1 and SSIM losses for the rendered and ground images. Instead, it first calculates the Hadamard product of these two images and the mask image, and then uses the result of the Hadamard product to calculate the L1 and SSIM losses. Since the pixel values ​​in the mask image range from 0 to 1, smaller values ​​indicate that the corresponding location in the image is more likely to be a dynamic object. Using the Hadamard product result with the mask image to calculate the loss reduces the contribution of dynamic objects to the gradient, allowing training to focus more on the static background, thereby achieving better reconstruction results.

[0096] The mask predictor described above was trained without using additional labels and used the same loss as the 3DGS training.

[0097] In other embodiments, the present invention also discloses a three-dimensional static background reconstruction device, such as... Figure 3 As shown, it includes:

[0098] The acquisition module 201 is used to acquire real images of the scene and camera parameters;

[0099] The rendering module 202 is used to input the initialized 3D Gaussian parameters and camera parameters into the 3DGS renderer, which is used to output the reconstructed rendered image.

[0100] The prediction module 203 is used to input the rendered image and the real image output by the 3DGS renderer into the mask predictor. The mask predictor is used to output a mask image. The value of each pixel on the mask image can reflect the reconstruction effect of the rendered image and distinguish between static backgrounds and dynamic objects.

[0101] Training module 204 is used to train the 3D Gaussian parameters and mask predictor;

[0102] The reconstruction module 205 is used to input the trained 3D Gaussian parameters and camera parameters into the 3DGS renderer and output the reconstructed static background rendered image.

[0103] Furthermore, in the prediction module, the value of each pixel on the mask image is between 0 and 1.

[0104] Furthermore, the lower the value of a pixel, the more it indicates that the pixel is occupied by a dynamic object;

[0105] The higher the pixel value, the more it indicates that the pixel is occupied by a static background.

[0106] Furthermore, the training module includes:

[0107] The first computing unit is used to calculate the Hadamard product of the rendered image and the real image with the mask image, respectively.

[0108]

[0109] Among them: I render and I real These are rendered images and real images, respectively.

[0110] M is the mask image;

[0111] ⊙ is for Hadamarda accumulation;

[0112] The second calculation unit is used to obtain the loss based on the calculation results of the first calculation unit using the following formula;

[0113]

[0114] in: For L1 loss;

[0115] For SSIM loss;

[0116] λ SSIM This is the loss coefficient for SSIM.

[0117] Furthermore, during the training of the 3D Gaussian parameters and the mask predictor, the gradient of the mask predictor does not propagate to the rendered image during backpropagation.

[0118] Furthermore, it should be noted that the three-dimensional static background reconstruction device provided in the above embodiments is only illustrated by the division of the above functional modules when performing static background reconstruction. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the three-dimensional static background reconstruction device can be divided into different functional modules to complete all or part of the functions described above.

[0119] Furthermore, the embodiments of the three-dimensional static background reconstruction device and the three-dimensional static background reconstruction method provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0120] Furthermore, in some other embodiments, such as Figure 4 As shown, the present invention also discloses a computing device, comprising:

[0121] One or more processors 301;

[0122] Memory 302;

[0123] And one or more programs, wherein the one or more programs are stored in memory 302 and configured to be executed by one or more processors 301, the one or more programs including instructions for the three-dimensional static background reconstruction method disclosed in the above embodiments.

[0124] Processor 301 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 301 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 301 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 301 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0125] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 302 are used to store at least one instruction, which is executed by the processor 301 to implement the three-dimensional static background reconstruction method provided in the method embodiments of the present invention.

[0126] In addition, the computing device may optionally include: a peripheral device interface and at least one peripheral device. The processor 301, memory 302, and peripheral device interface can be connected via a bus or signal lines. Each peripheral device can be connected to the peripheral device interface via a bus, signal lines, or a circuit board. Illustratively, peripheral devices include, but are not limited to: radio frequency circuitry, a touchscreen display, audio circuitry, and a power supply.

[0127] Of course, the computing device may also include fewer or more components, and this embodiment does not limit this.

[0128] Furthermore, in some other embodiments, the present invention also discloses a storage medium storing one or more computer-readable programs, the one or more programs including instructions adapted to be loaded by a memory and executed the three-dimensional static background reconstruction method disclosed in the above embodiments.

[0129] This invention discloses a method, apparatus, device, and storage medium for three-dimensional static background reconstruction, which has the following beneficial effects:

[0130] This invention is based on the 3DGS renderer and mask predictor, and uses 3DGS to reconstruct static backgrounds from scene data containing dynamic objects without relying on manual annotation information.

[0131] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the present invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope. All such changes and modifications fall within the scope of the present invention as claimed, which is defined by the appended claims and their equivalents.

Claims

1. A three-dimensional static background reconstruction method, characterized in that, include: Step S1: Obtain the real image of the scene and camera parameters; Step S2: Input the initialized 3D Gaussian parameters and camera parameters into the 3DGS renderer, which is used to output the reconstructed rendered image; Step S3: Input the rendered image and the real image output by the 3DGS renderer into the mask predictor. The mask predictor is used to output a mask image. The value of each pixel on the mask image can reflect the reconstruction effect of the rendered image and distinguish between static backgrounds and dynamic objects. Step S4: Train the 3D Gaussian parameters and mask predictor; Step S5: Input the trained 3D Gaussian parameters and camera parameters into the 3DGS renderer, and output the reconstructed static background rendered image; in step S3, the value of each pixel on the mask image is between 0 and 1. Furthermore, the lower the value of a pixel, the more it indicates that the pixel is occupied by a dynamic object; The higher the pixel value, the more it indicates that the pixel is occupied by a static background; Step S4 calculates the loss through the following steps: Step A: Calculate the Hadamard product of the rendered image and the real image with the mask image, respectively; ； ； in: and These are rendered images and real images, respectively. For masked images; For Hadamah accumulation; Step B: Based on the calculation results of Step A, obtain the loss using the following formula. ; ； in: For L1 loss; For SSIM loss; This is the loss coefficient for SSIM.

2. The three-dimensional static background reconstruction method according to claim 1, characterized in that, During the training of 3D Gaussian parameters and the mask predictor, the gradient of the mask predictor does not propagate to the rendered image during backpropagation.

3. A three-dimensional static background reconstruction device, characterized in that, include: The acquisition module is used to acquire real images of the scene and camera parameters; The rendering module is used to input the initialized 3D Gaussian parameters and camera parameters into the 3DGS renderer, which is used to output the reconstructed rendered image. The prediction module is used to input the rendered image and the real image output by the 3DGS renderer into the mask predictor. The mask predictor is used to output a mask image. The value of each pixel on the mask image can reflect the reconstruction effect of the rendered image and distinguish between static backgrounds and dynamic objects. The training module is used to train the 3D Gaussian parameters and mask predictor; The reconstruction module is used to input the trained 3D Gaussian parameters and camera parameters into the 3DGS renderer and output the reconstructed static background rendered image. In the prediction module, the value of each pixel on the mask image is between 0 and 1. Furthermore, the lower the value of a pixel, the more it indicates that the pixel is occupied by a dynamic object; The higher the pixel value, the more it indicates that the pixel is occupied by a static background; The training module includes: The first computing unit is used to calculate the Hadamard product of the rendered image and the real image with the mask image, respectively. ； ； in: and These are rendered images and real images, respectively. For masked images; For Hadamah accumulation; The second calculation unit is used to obtain the loss based on the calculation results of the first calculation unit using the following formula. ; ； in: For L1 loss; For SSIM loss; This is the loss coefficient for SSIM.

4. The three-dimensional static background reconstruction device according to claim 3, characterized in that, During the training of 3D Gaussian parameters and the mask predictor, the gradient of the mask predictor does not propagate to the rendered image during backpropagation.

5. A computing device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for the three-dimensional static background reconstruction method as described in claim 1 or 2 above.

6. A storage medium, characterized in that, The storage medium stores one or more computer-readable programs, the programs including instructions adapted to be loaded by memory and executed as described in claim 1 or 2.

Images