Panoramic reconstruction method, electronic device and storage medium based on 3DGS

Abstract

The present invention discloses a panoramic reconstruction method, electronic device and storage medium based on 3DGS, belonging to the field of three-dimensional reconstruction technology. The method comprises the following steps: collecting an original image containing a scene to be reconstructed, and processing it by a panoramic SfM method to obtain an initialization point cloud; initializing a set of 3D Gaussians using the initialization point cloud; predicting the changes in rotation, scaling and position by MLP, and updating the Gaussians; projecting according to the panoramic projection model based on the updated Gaussians, and using MLP to compensate for nonlinear errors; judging that the error is less than a threshold, or the number of iterations reaches an upper limit; using Gaussians for projection and rendering to obtain a clear deblurred image, and completing panoramic reconstruction. The invention can complete the data collection work by only using a panoramic camera to collect a small amount of data when reconstructing a scene, and the requirements for the collection quality are not high, which greatly reduces the dependence on accurate data sets compared to using the original 3DGS.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction, and in particular relates to a panoramic reconstruction method based on 3DGS, an electronic device and a storage medium. Background Art

[0002] 3D reconstruction is an important problem in computer vision. It aims to recover the geometric structure of a 3D scene from a 2D image and is usually used in autonomous driving, AR / VR and other fields. Traditional 3D reconstruction methods that do not use lasers, structured light, etc. are mainly Multi-View Stereo (MVS) and Structure-from-Motion (SfM). They perform feature matching, pose estimation and triangulation between multiple views based on multi-view geometry methods to obtain the restored 3D feature point cloud. However, for the reconstruction of large-scale scenes, the computing power consumed by matching is greatly increased, and it is not robust to situations such as inaccurate internal parameters and a large number of outliers. The recently proposed Neural Radiance Fields (NeRF) has shown amazing results in reconstruction. By training a multilayer perceptron (MLP), it can generate images from any new perspective. It optimizes the radiation field instead of detecting feature points by directly optimizing the photometric error between the rendered image pixels and the original pixels, which can avoid the impact of mismatching. However, since each rendering requires querying a large number of sampling points in space and generating corresponding parameters through MLP, NeRF is very time-consuming to train. In order to improve the training speed, 3D Gaussian Splatting (3DGS) was proposed, which replaces the point query in NeRF by projecting Gaussian onto the imaging plane, thereby greatly improving the training speed and being able to complete 3D reconstruction work efficiently and with high quality.

[0003] However, using the reconstruction results for autonomous driving or other data sets requires a large number of camera images, and good external parameter calibration is required between cameras. Therefore, using a panoramic camera to collect data is a relatively ideal way. Only one acquisition is required to obtain data equivalent to that collected by 6 pinhole cameras, and it can avoid inaccurate external parameter calibration between multiple cameras, different aperture light intake, etc. However, in the prior art, the projection step in 3DGS is designed for pinhole cameras and cannot be directly used for panoramic image reconstruction, and simple model modifications cannot adapt well to scenes with large distortion of panoramic cameras. In addition, due to the movement of the camera during the collection of data sets such as autonomous driving, motion blur, out-of-focus blur and other problems will occur, and the effect of directly using the original method for reconstruction is not good. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a panoramic reconstruction method, electronic device and storage medium based on 3DGS, which can complete the data collection work by only using a panoramic camera to collect a small amount of data when reconstructing a scene, and does not have high requirements on the collection quality. Compared with using the original 3DGS, it greatly reduces the dependence on accurate data sets, thereby solving at least one technical problem involved in the background technology.

[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides a panoramic reconstruction method based on 3DGS, comprising the following steps:

[0007] Step S1, collecting original images containing the scene to be reconstructed as a data set;

[0008] Step S2, processing the original images in the data set by the panoramic SfM method to obtain a set of initialization point clouds;

[0009] Step S3, for each point in the initialization point cloud, initialize the rotation, scaling, opacity and color to obtain a set of 3D Gaussians;

[0010] Step S4, taking the rotation, scaling and position of each Gaussian as the MLP input, obtaining n groups of blur amounts simulating motion blur, adding the blur amounts to the original Gaussian to perform blur operation, obtaining n groups of updated Gaussians, where n is a positive integer greater than or equal to 2;

[0011] Step S5, using a panoramic projection model to perform a projection operation on each group of updated Gaussians, and obtaining a value after compensating for nonlinear errors through MLP to obtain n panoramic images, and obtaining a blurred image by averaging the corresponding pixels of the n panoramic images;

[0012] Step S6, combining the photometric error and the structural similarity error, rendering the loss function between the blurred image and the original image, and training the loss function. If the loss function is less than the set threshold, or the number of iterations exceeds the threshold, the training ends and step S7 is executed, otherwise the training continues;

[0013] Step S7, directly use Gaussian for projection and rendering to obtain a clear deblurred image and complete panoramic reconstruction.

[0014] In a second aspect, an embodiment of the present invention provides an electronic device, including:

[0015] at least one processor;

[0016] At least one memory for storing at least one program;

[0017] When the at least one program is executed by the at least one processor, the at least one processor implements the steps of the method described in the first aspect.

[0018] In a third aspect, an embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0019] In a fourth aspect, an embodiment of the present invention provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1. Enable 3DGS to use panoramic camera data for data reconstruction, and use MLP to compensate for errors, which can effectively reduce the errors caused by linear approximation in large distortion scenarios such as panoramic cameras;

[0022] 2. The step of simulating blur during training improves the reconstruction accuracy when blur is caused by camera motion, so that the method can still perform well on datasets that have not been carefully processed, reducing the difficulty of collecting data for 3D reconstruction;

[0023] 3. The present invention can complete the data collection work by only collecting a small amount of data using a panoramic camera when reconstructing a scene, and does not have high requirements on the collection quality. Compared with using the original 3DGS, it greatly reduces the dependence on accurate data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0025] Figure 1 is a flow chart of a panoramic reconstruction method based on 3DGS provided by an embodiment of the present invention;

[0026] Figure 2 This is one of the hardware structure diagrams of the electronic device provided by the embodiment of the present invention;

[0027] Figure 3 This is the second schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The terms "first", "second", etc. in the specification and claims of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] The 3DGS-based panoramic reconstruction method provided by the embodiment of the present invention is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0031] See also Figure 1 , is a panoramic reconstruction method based on 3DGS provided by an embodiment of the present invention, comprising the following steps:

[0032] Step S1, using a panoramic camera to collect original images containing the scene to be reconstructed as a data set;

[0033] Step S2, processing the original images in the data set by the panoramic SfM method to obtain a set of initialization point clouds;

[0034] Step S3, for each point in the initialization point cloud, initialize the rotation, scaling, opacity and color to obtain a set of 3D Gaussians;

[0035] Step S4, taking the rotation, scaling and position of each Gaussian as the MLP input, obtaining n groups of blur amounts simulating motion blur, adding the blur amounts to the original Gaussian to perform blur operation, obtaining n groups of updated Gaussians, where n is a positive integer greater than or equal to 2;

[0036] Step S5, using a panoramic projection model to perform a projection operation on each group of updated Gaussians, and obtaining a value after compensating for nonlinear errors through MLP to obtain n panoramic images, and obtaining a blurred image by averaging the corresponding pixels of the n panoramic images;

[0037] Step S6, combining the photometric error and the structural similarity error, rendering the loss function between the blurred image and the original image, and training the loss function. If the loss function is less than the set threshold, or the number of iterations exceeds the threshold, the training ends and step S7 is executed, otherwise the training continues;

[0038] Step S7, directly use Gaussian for projection and rendering to obtain a clear deblurred image and complete panoramic reconstruction.

[0039] Step S5 specifically includes:

[0040] Step S51, determine the panoramic projection model, for any sampling point in the space , where is the coordinate of the space point in the camera coordinate system; T represents the transposed matrix; assuming that the panoramic camera is a fisheye camera, the panoramic projection model is expressed by the following formula:

[0041] ;

[0042] In the formula, is the focal length, is the angle between the line connecting the space point and the origin and the z-axis of the camera coordinate system, , is the distance from the imaging point to the origin on the imaging plane. The imaging point is converted from the imaging plane to the pixel coordinate system, which is expressed by the following formula:

[0043] ;

[0044] In the formula, is the coordinate of the projection point in the pixel coordinate system, is the coordinate under the imaging plane; are the length and width of the pixel respectively; is the coordinate of the center point of the pixel coordinate system; For space point The distance from the origin of the imaging coordinate system;

[0045] Step S52, for the Gaussian center , the projection process is For the panoramic imaging process, is the loss function; the back propagation process is ,in:

[0046] ;

[0047] In the formula, ;

[0048] Step S53, for Gaussian covariance , using the Jacobian matrix of the panoramic camera ,make , W is the rotation matrix from the world coordinate system to the camera coordinate system, is a collection of matrices of dimension 2×3, is the transformation matrix from the world coordinate system to the pixel coordinate system; then the two-dimensional Jacobian after projection is ; The back propagation process can be deduced as follows:

[0049] ;

[0050] ;

[0051] ;

[0052] S 1 to S 8 All represent intermediate calculation parameters:

[0053] ;

[0054] Step S54, the obtained Gaussian center , Depth of Field And sampling points Coordinates are input into the pre-trained dedistortion MLP to obtain sampling points The correction amount , at the corrected point Sampling is performed at the correction point to obtain the Gaussian sampling value at the correction point, which is expressed by the following formula:

[0055] ;

[0056] Step S55, rendering is performed according to the opacity α and the color c to obtain a panoramic image, which is expressed by the following formula:

[0057] ;

[0058] In the formula, is the pixel coordinate of the pixel, , and Respectively k Gaussian color, opacity and sampling value, is the transmittance of all Gaussians before the kth Gaussian, is the final color of the pixel.

[0059] like Figure 2As shown, an embodiment of the present invention further provides an electronic device 600, which includes a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned 3DGS-based panoramic reconstruction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0060] It should be noted that the first electronic device in the embodiment of the present invention includes the mobile electronic device and the non-mobile electronic device mentioned above.

[0061] Figure 3 The present invention is a schematic diagram of the hardware structure of an electronic device.

[0062] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0063] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 3 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0064] It should be understood that in the embodiment of the present invention, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static image or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 709 may be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 710.

[0065] An embodiment of the present invention further provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned 3DGS-based panoramic reconstruction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0066] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0067] An embodiment of the present invention further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned 3DGS-based panoramic reconstruction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0068] It should be understood that the chip mentioned in the embodiment of the present invention may also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0069] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0071] An embodiment of the present invention further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned 3DGS-based panoramic reconstruction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0072] It should be understood that the chip mentioned in the embodiment of the present invention may also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0073] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A panoramic reconstruction method based on 3DGS, characterized in that: The following steps are involved: Step S1, collecting original images containing the scene to be reconstructed as a data set; Step S2, processing the original images in the data set by the panoramic SfM method to obtain a set of initialization point clouds; Step S3, for each point in the initialization point cloud, initialize the rotation, scaling, opacity and color to obtain a set of 3D Gaussians; Step S4, taking the rotation, scaling and position of each Gaussian as the MLP input, obtaining n groups of blur amounts simulating motion blur, adding the blur amounts to the original Gaussian to perform blur operation, obtaining n groups of updated Gaussians, where n is a positive integer greater than or equal to 2; Step S5, using a panoramic projection model to perform a projection operation on each group of updated Gaussians, and obtaining a value after compensating for nonlinear errors through MLP to obtain n panoramic images, and obtaining a blurred image by averaging the corresponding pixels of the n panoramic images; Step S6, combining the photometric error and the structural similarity error, rendering the loss function between the blurred image and the original image, and training the loss function. If the loss function is less than the set threshold, or the number of iterations exceeds the threshold, the training ends and step S7 is executed, otherwise the training continues; Step S7, directly use Gaussian for projection and rendering to obtain a clear deblurred image and complete panoramic reconstruction.

2. The method according to claim 1, characterized in that In step S1, an original image is captured using a panoramic camera.

3. The method according to claim 2, characterized in that Step S5 specifically includes: Step S51, determine the panoramic projection model, for any sampling point in the space , where is the coordinate of the space point in the camera coordinate system; T represents the transposed matrix; assuming that the panoramic camera is a fisheye camera, the panoramic projection model is expressed as follows: ; In the formula, is the focal length, is the angle between the line connecting the space point and the origin and the z-axis of the camera coordinate system, is the distance from the imaging point to the origin on the imaging plane. The imaging point is converted from the imaging plane to the pixel coordinate system, which is expressed by the following formula: ; In the formula, is the coordinate of the projection point in the pixel coordinate system, is the coordinate under the imaging plane; are the length and width of the pixel respectively; is the coordinate of the center point of the pixel coordinate system; For space point The distance from the origin of the imaging coordinate system; Step S52, for the Gaussian center , the projection process is For the panoramic imaging process, is the loss function; the back propagation process is ,in: ; In the formula, ; Step S53, for Gaussian covariance , using the Jacobian matrix of the panoramic camera ,make , W is the rotation matrix from the world coordinate system to the camera coordinate system, is a collection of matrices of dimension 2×3, is the transformation matrix from the world coordinate system to the pixel coordinate system; then the two-dimensional Jacobian after projection is ; The back propagation process can be deduced as follows: ; ; ; S 1 to S 8 All represent intermediate calculation parameters: ; Step S54, the obtained Gaussian center , Depth of Field And sampling points Coordinates are input into the pre-trained dedistortion MLP to obtain sampling points The correction amount , at the corrected point Sampling is performed at the correction point to obtain the Gaussian sampling value at the correction point, which is expressed by the following formula: ; Step S55, rendering is performed according to the opacity α and the color c to obtain a panoramic image, which is expressed by the following formula: ; In the formula, is the pixel coordinate of the pixel, and Respectively k Gaussian color, opacity and sampling value, is the transmittance of all Gaussians before the kth Gaussian, is the final color of the pixel.

4. An electronic device, characterized in that: include: at least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the panoramic reconstruction method based on 3DGS as described in any one of claims 1-3.

5. A storage medium, characterized in that: Processor-executable instructions are stored, and when the processor executes the processor-executable instructions, the panoramic reconstruction method based on 3DGS according to any one of claims 1 to 3 is executed.

Images