A fisheye image 3D gaussian reconstruction method, system, terminal and storage medium

Abstract

This invention belongs to the field of visual 3D reconstruction technology and discloses a method, system, terminal, and storage medium for 3D Gaussian reconstruction of fisheye images. The method includes: acquiring a fisheye image; generating a sparse 3D point cloud corresponding to the fisheye image based on a motion recovery structure algorithm; optimizing the 3D Gaussian reconstruction algorithm based on a reversible residual network and an octahedral projection model; and reconstructing a 3D model of the fisheye image based on the optimized 3D Gaussian reconstruction algorithm and the sparse 3D point cloud. This invention reduces the 3D reconstruction error of fisheye images by simulating camera distortion parameters of a lens distortion model using a reversible residual network and incorporating these parameters into the 3D Gaussian reconstruction algorithm. Replacing the planar perspective model of the 3D Gaussian reconstruction algorithm with an octahedral projection model reduces stretching and image boundary distortion. The optimized 3D Gaussian reconstruction algorithm is used to generate the 3D model of the fisheye image, improving the realism of the 3D model reconstructed from the fisheye image.

Description

Technical Field

[0001] This invention relates to the field of visual 3D reconstruction technology, and in particular to a method, system, terminal and storage medium for 3D Gaussian reconstruction of fisheye images. Background Technology

[0002] Visual 3D reconstruction technology is a technique that uses image data to recover high-quality real-world 3D models. It is an indispensable key technology for applications such as digital twins and real-world 3D.

[0003] Conventional 3D reconstruction techniques are susceptible to interference from various environmental factors in complex scenes. Factors such as sparse textures and insufficient lighting can lead to visual feature degradation, resulting in poor accuracy, low fidelity, and incompleteness of the reconstructed model. 3D Gaussian Splatting (3DGS) technology has garnered significant attention in 3D reconstruction applications for complex scenes. 3DGS describes the reconstruction target by constructing a series of explicit Gaussian models with geometric and textural information. It achieves high-fidelity, real-time rendering of realistic scenes through visibility-aware anisotropic Gaussian models and fast rasterization. 3DGS technology can achieve pixel-level real-world 3D reconstruction and has broad application prospects in fields such as embodied intelligence, navigation and positioning, engineering surveying, and digital preservation of cultural relics.

[0004] However, existing 3DGS techniques, which construct the mapping relationship between pixels and Gaussians based on pinhole projection models, are not suitable for fisheye camera models with nonlinear distortions. Conventional 3DGS methods cannot model image distortion when processing fisheye images, leading to reduced stability and accuracy of Gaussian projection, thus affecting the realism of the model. Therefore, targeted improvements are needed to the existing 3D Gaussian framework to adapt to fisheye distortion; however, this research area is still in its early stages and has limitations, failing to achieve ideal modeling results.

[0005] Existing technologies still suffer from insufficient realism in 3D models reconstructed from fisheye images using Gaussian methods. Therefore, existing technologies need further improvement. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method, system, terminal and storage medium for 3D Gaussian reconstruction of fisheye images, in order to solve the problem of insufficient realism of the reconstructed 3D model in existing 3D Gaussian reconstruction methods for fisheye images.

[0007] The technical solution adopted by this invention to solve the technical problem is as follows:

[0008] In a first aspect, the present invention provides a 3D Gaussian reconstruction method for fisheye images, comprising:

[0009] Acquire a fisheye image and generate a sparse 3D point cloud corresponding to the fisheye image based on the structure-reconstruction-motion algorithm.

[0010] Optimize the 3D Gaussian reconstruction algorithm based on invertible residual network and octahedral projection model;

[0011] The three-dimensional model of the fisheye image is reconstructed based on the optimized 3D Gaussian reconstruction algorithm and the sparse three-dimensional point cloud.

[0012] In one implementation, the optimization of the 3D Gaussian reconstruction algorithm based on the invertible residual network and octahedral projection model includes:

[0013] Construct a lens distortion model;

[0014] The camera distortion parameters of the lens distortion model are simulated using an invertible residual network, and the camera distortion parameters are then incorporated into the 3D Gaussian reconstruction algorithm.

[0015] The octahedral projection model is used to replace the planar perspective model of the 3D Gaussian reconstruction algorithm to obtain the optimized 3D Gaussian reconstruction algorithm.

[0016] In one implementation, constructing the lens distortion model includes:

[0017] A lens distortion model is constructed, which is defined as a ray transformation from the undistorted direction to the distorted direction.

[0018] In one implementation, the step of using a reversible residual network to simulate the camera distortion parameters of the lens distortion model and incorporating the camera distortion parameters into the 3D Gaussian reconstruction algorithm includes:

[0019] Construct a reversible residual network;

[0020] The camera distortion parameters of the lens distortion model are simulated using the reversible residual network.

[0021] The camera distortion parameters are added to the 3D Gaussian reconstruction algorithm, and the camera distortion parameters are refined when the 3D Gaussian reconstruction algorithm optimizes the reconstruction parameters.

[0022] In one implementation, replacing the planar perspective model of the 3D Gaussian reconstruction algorithm with the octahedral projection model includes:

[0023] Based on the octahedral projection model, find the corresponding coordinates of each rendered pixel on the unit octahedral texture surface;

[0024] Calculate the mapping relationship between the coordinates of the rendered pixel and the corresponding coordinates;

[0025] The distortion rendering of the rendered pixels on the octahedral texture surface is achieved based on the mapping relationship.

[0026] In one implementation, calculating the mapping relationship between the coordinates of the rendered pixel and the corresponding coordinates includes:

[0027] Define the coordinates of the rendered pixel as follows: The corresponding distorted coordinates are The corresponding coordinates on the octahedral texture surface are ;

[0028] Calculate the mapping relationship based on the defined coordinates:

[0029] ;

[0030] Wherein, the corresponding coordinates of the octahedral texture surface satisfy... ; For the sign function.

[0031] In one implementation, reconstructing the 3D model of the fisheye image based on the optimized 3D Gaussian reconstruction algorithm and the sparse 3D point cloud includes:

[0032] Based on the optimized 3D Gaussian reconstruction algorithm, the projection relationship from point coordinates to pixel coordinates in three-dimensional coordinates is calculated;

[0033] Based on the projection relationship, the sparse 3D point cloud is modeled to reconstruct the 3D model of the fisheye image.

[0034] In a second aspect, the present invention provides a 3D Gaussian reconstruction system for fisheye images, comprising:

[0035] The point cloud generation module is used to acquire fisheye images and generate sparse 3D point clouds corresponding to the fisheye images based on the structure-reconstruction-motion algorithm.

[0036] The algorithm optimization module optimizes the 3D Gaussian reconstruction algorithm based on the invertible residual network and the octahedral projection model.

[0037] The 3D reconstruction module is used to reconstruct the 3D model of the fisheye image based on the optimized 3D Gaussian reconstruction algorithm and the sparse 3D point cloud.

[0038] Thirdly, the present invention provides a terminal comprising: a processor and a memory, the memory storing a 3D Gaussian reconstruction program for a fisheye image, wherein the 3D Gaussian reconstruction program for the fisheye image, when executed by the processor, is used to implement the operation of the 3D Gaussian reconstruction method for the fisheye image as described in the first aspect.

[0039] Fourthly, the present invention also provides a computer-readable storage medium storing a 3D Gaussian reconstruction program for a fisheye image, which, when executed by a processor, is used to implement the operation of the 3D Gaussian reconstruction method for a fisheye image as described in the first aspect.

[0040] The present invention, by employing the above technical solution, has the following effects:

[0041] This invention simulates camera distortion parameters of a lens distortion model using a reversible residual network, incorporates these parameters into a 3D Gaussian reconstruction algorithm, and effectively reduces the 3D reconstruction error of fisheye images. Furthermore, it replaces the planar perspective model of the 3D Gaussian reconstruction algorithm with an octahedral projection model, reducing stretching and image boundary distortion to achieve a more ideal modeling effect. Finally, the optimized 3D Gaussian reconstruction algorithm generates a 3D model of the fisheye image, improving the realism of the 3D model reconstructed from the fisheye image using a 3D Gaussian reconstruction algorithm. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0043] Figure 1 This is a flowchart of the 3D Gaussian reconstruction method for fisheye images in this invention.

[0044] Figure 2 This is the original fisheye image group obtained in one implementation of the present invention.

[0045] Figure 3 This is a schematic diagram of a conventional three-dimensional reconstruction model of the original fisheye image group in one implementation of the present invention.

[0046] Figure 4 This is a schematic diagram of a 3DGS three-dimensional reconstruction model of the original fisheye image group in one implementation of the present invention.

[0047] Figure 5 This is a schematic diagram of a camera parameter optimization process that includes an invertible residual network in one implementation of the present invention.

[0048] Figure 6 This is a flowchart illustrating the principle of the residual block in one implementation of the present invention.

[0049] Figure 7 This is a schematic diagram illustrating the process of three-dimensional reconstruction of the original fisheye image in one implementation of the present invention.

[0050] Figure 8 This is a schematic diagram of an optimized 3DGS three-dimensional reconstruction model of the original fisheye image group in one implementation of the present invention.

[0051] Figure 9 This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0052] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0054] Exemplary methods

[0055] Existing 3DGS technology is based on pinhole projection models to construct the mapping relationship between pixels and Gaussians, which is not suitable for fisheye camera models with nonlinear distortion. Conventional 3DGS methods cannot model image distortion when processing fisheye images, leading to reduced stability and accuracy of Gaussian projection, thus affecting the realism of the model. Therefore, it is necessary to make targeted improvements to the existing 3D Gaussian framework to adapt to fisheye distortion. However, this research field is still in its early stages and has limitations, failing to achieve ideal modeling results.

[0056] like Figure 2 As shown, this is the original fisheye image set obtained in this embodiment, from left to right: original fisheye image 1, original fisheye image 2, and original fisheye image 3. After performing 3D reconstruction on the original fisheye image set using conventional 3D reconstruction techniques, a schematic diagram of the conventional 3D reconstruction model of the original fisheye image set is obtained, as shown below. Figure 3 As shown, Figure 3 From left to right, the images show three conventional 3D reconstruction models: model 1 corresponding to the original fisheye image 1, model 2 corresponding to the original fisheye image 2, and model 3 corresponding to the original fisheye image 3. It is evident that conventional 3D reconstruction techniques generate reconstruction models with poor accuracy, low fidelity, and poor integrity. However, after using 3DGS technology to perform 3D reconstruction on the original fisheye image set, a schematic diagram of the 3DGS reconstruction model of the original fisheye image set is obtained, as shown below. Figure 4 As shown, Figure 4 From left to right, the images are: 3DGS 3D reconstruction model 1 corresponding to the original fisheye image 1, 3DGS 3D reconstruction model 2 corresponding to the original fisheye image 2, and 3DGS 3D reconstruction model 3 corresponding to the original fisheye image 3. It can be seen that when faced with fisheye images, 3DGS technology cannot handle the distortion of fisheye images, resulting in insufficient realism of the 3D reconstruction model.

[0057] To address the above technical problems, this invention provides a method for acquiring fisheye images, generating sparse 3D point clouds corresponding to the fisheye images based on a motion recovery structure algorithm, optimizing a 3D Gaussian reconstruction algorithm based on a reversible residual network and an octahedral projection model, and reconstructing a 3D model of the fisheye image based on the optimized 3D Gaussian reconstruction algorithm and the sparse 3D point clouds. This invention uses a reversible residual network to simulate camera distortion parameters of a lens distortion model, incorporating these parameters into the 3D Gaussian reconstruction algorithm to better reduce 3D reconstruction errors of fisheye images. Furthermore, by replacing the planar perspective model of the 3D Gaussian reconstruction algorithm with an octahedral projection model, stretching and image boundary distortion are reduced, achieving a more ideal modeling effect. Finally, the optimized 3D Gaussian reconstruction algorithm generates a 3D model of the fisheye image, improving the realism of the 3D model reconstructed from the fisheye image.

[0058] like Figure 1 As shown, this embodiment of the invention provides a 3D Gaussian reconstruction method for fisheye images, including the following steps:

[0059] Step S100: Obtain a fisheye image and generate a sparse 3D point cloud corresponding to the fisheye image based on the structure-reconstruction-motion algorithm.

[0060] In this embodiment, the Structure from Motion (SfM) algorithm is used to reconstruct the sparse 3D point cloud of the scene. The SfM algorithm can recover camera parameters and 3D scene structure by analyzing image sequences, thereby generating the sparse 3D point cloud corresponding to the fisheye image.

[0061] Specifically, the process of generating sparse 3D point clouds corresponding to fisheye images based on the structure-of-motion motion (SOG) algorithm includes the following steps:

[0062] Step S101: Perform feature extraction and matching on the fisheye image.

[0063] In this embodiment, key points are detected in the fisheye image. Key points are generally points with significant features in the fisheye image, such as corner points or points with rich textures. Feature extraction of the fisheye image is achieved by extracting key points.

[0064] In this embodiment, the extracted features are matched to determine which points in the image are projections of the same three-dimensional points.

[0065] Step S102: Perform initial camera pose estimation based on the fisheye image.

[0066] In this embodiment, the homography matrix is ​​calculated based on the matched keypoint pairs. :

[0067] ;

[0068] In the formula, Indicates camera intrinsic parameters. and These are the rotation and translation matrices of the camera in the real-world coordinate system, i.e., the camera extrinsic parameters.

[0069] The relative pose of the camera is decomposed from the homography matrix to estimate the initial pose of the camera.

[0070] Step S103: 3D sparse point cloud reconstruction.

[0071] In this embodiment, the actual position of key points in three-dimensional space is recovered using triangulation, that is, the spatial coordinates of three-dimensional points are calculated through geometric relationships, thereby generating a sparse three-dimensional structure of the scene.

[0072] like Figure 1 As shown, this embodiment of the invention provides a 3D Gaussian reconstruction method for fisheye images, including the following steps:

[0073] Step S200: Optimize the 3D Gaussian reconstruction algorithm based on the invertible residual network and octahedral projection model.

[0074] Specifically, in one implementation of this embodiment, step S200 includes the following steps:

[0075] Step S201: Construct a lens distortion model.

[0076] It should be noted that the lens distortion model is used to reflect the correspondence between pixel positions in the real image and the distorted image. By constructing the lens distortion model, the 3DGS technology is extended, enabling the 3D Gaussian reconstruction algorithm to adapt to fisheye lenses.

[0077] In this embodiment, the constructed lens distortion model is defined as a ray transformation from the distortion-free direction to the distortion direction, and its corresponding formula is as follows:

[0078] ;

[0079] in, Represents the pixel coordinates on a distortion-free, ideal image; Represents the distortion coordinates on the fisheye image; This represents all trainable camera parameters; It is a function representing the distortion transformation process, and this function is invertible, that is, it can be reversed through inverse mapping. Simulate the distortion correction transformation process.

[0080] Step S202: Use a reversible residual network to simulate the camera distortion parameters of the lens distortion model, and add the camera distortion parameters to the 3D Gaussian reconstruction algorithm.

[0081] In this embodiment, a reversible residual network is used to simulate the camera distortion parameters of the lens distortion model, and the camera distortion parameters are added to the 3D Gaussian reconstruction algorithm to achieve optimization of the 3D Gaussian reconstruction algorithm based on the reversible residual network.

[0082] In one implementation of this embodiment, step S202 includes the following steps:

[0083] Step S202a: Construct a reversible residual network.

[0084] In this embodiment, invertible residual networks (InvResNet) are introduced to construct the invertible residual network. It consists of four residual modules, each capable of backpropagation. For example... Figure 5 The diagram shown illustrates the camera parameter optimization process in this embodiment, which includes a reversible residual network. The principle of the residual block in the reversible residual network is: Output feature = Features learned by the main path + Input. Figure 6 The diagram shown is a flowchart illustrating the principle of the residual block in this embodiment. The features learned by the main path are obtained through the weight layers and activation functions commonly used in deep learning. The skip connection method of the residual block can retain the detailed information of the low-level input, enabling the deep neural network to learn features more comprehensively.

[0085] Step S202b: Use the reversible residual network to simulate the camera distortion parameters of the lens distortion model.

[0086] In this embodiment, a reversible residual network is used to adaptively simulate lens distortion. Camera internal parameters and external references The function mapping process integrates camera distortion parameters into the end-to-end differentiable optimization process, such as... Figure 6 As shown, while optimizing reconstruction parameters using neural networks, the camera's intrinsic and extrinsic parameters and distortion parameter magnitudes can be refined. More accurate fisheye distortion coordinates can be obtained through the iterative update process of the reversible residual network. .

[0087] Step S202c: The camera distortion parameters are added to the 3D Gaussian reconstruction algorithm. When the 3D Gaussian reconstruction algorithm optimizes the reconstruction parameters, the camera distortion parameters are refined.

[0088] In this embodiment, camera distortion parameters are added to the 3D Gaussian reconstruction algorithm. Utilizing the backpropagation characteristics of the reversible residual network, each residual module can perform backpropagation, thereby achieving end-to-end estimation and optimization of camera distortion parameters during the network training process of the 3D Gaussian reconstruction algorithm to optimize reconstruction parameters.

[0089] This embodiment introduces a reversible residual network module to perform end-to-end modeling and optimization of camera distortion parameters in fisheye images, thereby incorporating lens distortion parameters into the optimization process of the 3D Gaussian reconstruction algorithm. Compared with conventional pre-calibration methods, the 3D Gaussian reconstruction algorithm optimized by the reversible residual network in this embodiment can achieve smaller reconstruction errors, thereby minimizing reconstruction errors during the 3D reconstruction process.

[0090] Step S203: Replace the planar perspective model of the 3D Gaussian reconstruction algorithm with the octahedral projection model to obtain the optimized 3D Gaussian reconstruction algorithm.

[0091] It should be noted that in the process of projecting fisheye images, traditional equidistant cylindrical projection directly projects the sphere onto the plane without relying on intermediate projection geometry. Its intuitive linear transformation formula makes it easy to operate, but the extremely low projection complexity leads to a reduction in projection uniformity, and the pixel sampling density at the two poles of the sphere is greater than that at the equator.

[0092] Cube projection projects the spherical content onto a cube model in perspective, then unfolds and stitches the faces together to form a rectangular image. Compared to cylindrical projection, the uniformity of the cube model's projection is improved. However, since the perspective projection process does not change the size of the solid angles, this means that two points with equal solid angles on the sphere will still exhibit a higher density in the central area and a lower density at the edges when projected onto the cube.

[0093] In this embodiment, the 3DGS projection process is extended to fisheye images with a large field of view using an octahedral model. Octahedral projection is a projection method that uses an octahedron, where each face is an equilateral triangle, as the projection model. Because it has more faces than cube projection, the uniformity of the projection is improved. Therefore, by extending the original 3D Gaussian planar perspective projection model to an octahedral projection model, a relatively uniform pixel density is maintained from the center to the edge of the fisheye image, thus achieving adaptive Gaussian rendering for fisheye images.

[0094] In one implementation of this embodiment, step S203 includes the following steps:

[0095] Step S203a: Based on the octahedral projection model, find the corresponding coordinates of each rendered pixel on the unit octahedral texture surface.

[0096] In this embodiment, by resampling on the octahedral texture surface, distorted images with a field of view exceeding 180° can be rendered. For each rendered pixel, its corresponding position is found in the constructed octahedral texture, and resampling is performed in conjunction with the mixed distortion field to achieve distortion-aware rendering.

[0097] Step S203b: Calculate the mapping relationship between the coordinates of the rendered pixel and the corresponding coordinates.

[0098] In this embodiment, the coordinates of the rendered pixel are defined as follows: Considering lens distortion, the rendered pixels are modeled as Therefore, the distortion coordinates corresponding to the rendered pixel are The corresponding coordinates on the octahedral texture surface are The calculated mapping relationship is as follows:

[0099] ;

[0100] Wherein, the corresponding coordinates of the octahedral texture surface satisfy... ; For the sign function.

[0101] In this embodiment, the `sign` function is the sign function, which returns different signs depending on the input value. When the input value is greater than 0, the function returns 1; when the input value is equal to 0, the function returns 0; and when the input value is less than 0, the function returns -1.

[0102] Step S203c: Implement distortion rendering of the rendering pixel on the octahedral texture surface according to the mapping relationship.

[0103] In this embodiment, the distortion rendering of the rendering pixel on the octahedral texture surface is realized according to the mapping relationship; and the entire resampling process is completely differentiable and can be directly used as a plug-and-play module for rendering of 3D Gaussian reconstruction algorithm.

[0104] This embodiment introduces an octahedral projection model, which, compared to conventional isometric projection and cube projection models, can better achieve projection uniformity, thereby reducing stretching and image boundary distortion, and achieving a more ideal modeling effect.

[0105] like Figure 1 As shown, this embodiment of the invention provides a 3D Gaussian reconstruction method for fisheye images, including the following steps:

[0106] Step S300: Reconstruct the three-dimensional model of the fisheye image based on the optimized 3D Gaussian reconstruction algorithm and the sparse three-dimensional point cloud.

[0107] It should be noted that the 3D Gaussian reconstruction algorithm uses a projection function. Real-world three-dimensional points Projected onto two-dimensional pixel coordinates Thus, the projection relationship of the 3D Gaussian reconstruction algorithm can be obtained for model rendering.

[0108] Specifically, in one implementation of this embodiment, step S300 includes the following steps:

[0109] Step S301: Calculate the projection relationship from point coordinates to pixel coordinates in three-dimensional coordinates based on the optimized 3D Gaussian reconstruction algorithm.

[0110] In this embodiment, to address the issue that conventional 3D Gaussian reconstruction algorithms are only applicable to perspective projection models without lens distortion, a distortion model is introduced through step S200. Based on the 3D Gaussian reconstruction algorithm optimized by the invertible residual network and the octahedral projection model, the projection relationship between the point and pixel coordinates of the three-dimensional coordinates is further derived and calculated.

[0111] In this embodiment, the projection relationship from the point in the three-dimensional coordinate system to the pixel coordinate system, i.e., the 3D Gaussian position in the camera coordinate system, is... In the pixel coordinate system The projection relationship between them is given by the following formula:

[0112] ;

[0113] in, Represents the homography matrix; Indicates camera internal parameters; This represents the process of octahedral projection.

[0114] Step S302: Based on the projection relationship, perform model rendering on the sparse three-dimensional point cloud to reconstruct the three-dimensional model of the fisheye image.

[0115] In this embodiment, the projection relationship obtained in step S301 is used to perform model rendering on the sparse three-dimensional point cloud, generate the texture and color of the model, reconstruct the three-dimensional model of the fisheye image, and obtain the final high-fidelity real-scene three-dimensional model.

[0116] like Figure 7 The image shown is a schematic diagram illustrating the 3D reconstruction process of the original fisheye image in this embodiment. We selected... Figure 2 The original fisheye image 1 in the original fisheye image group is used for explanation. Figure 7 From left to right, the images represent the original fisheye image 1, the sparse 3D point cloud, the 3D Gaussian model, and the final reconstructed 3D model, respectively. The point coordinates of the sparse 3D point cloud are obtained using the structure-of-motion reconstruction algorithm. The coordinates of each ellipsoid in the 3D Gaussian model are determined according to the distortion model. The final 3D model is obtained through octahedral projection and rendering using a Gaussian model.

[0117] like Figure 8 As shown, this demonstrates the 3D Gaussian reconstruction method using fisheye images for... Figure 2 The diagram shows an optimized 3DGS 3D reconstruction model generated after 3D reconstruction of the original fisheye image set. Figure 8 From left to right, the images are: the optimized 3DGS 3D reconstruction model 1 corresponding to the original fisheye image 1, the optimized 3DGS 3D reconstruction model 2 corresponding to the original fisheye image 2, and the optimized 3DGS 3D reconstruction model 3 corresponding to the original fisheye image 3. As can be seen from the figure, the optimized 3D Gaussian reconstruction algorithm achieves higher realism in 3D reconstruction of the fisheye image, a special type of image, and achieves the ideal modeling effect.

[0118] This embodiment achieves the following technical effects through the above technical solution:

[0119] This embodiment acquires a fisheye image and generates a sparse 3D point cloud corresponding to the fisheye image based on the structure-of-motion (SOG) algorithm. It then optimizes the 3D Gaussian reconstruction algorithm based on a reversible residual network and an octahedral projection model. Based on the optimized 3D Gaussian reconstruction algorithm and the sparse 3D point cloud, it reconstructs a 3D model of the fisheye image. By simulating camera distortion parameters of a lens distortion model using a reversible residual network, it performs end-to-end modeling and optimization of the fisheye distortion parameters. Incorporating these camera distortion parameters into the 3D Gaussian reconstruction algorithm better reduces the 3D reconstruction error of the fisheye image. Furthermore, it replaces the planar perspective model of the 3D Gaussian reconstruction algorithm with an octahedral projection model. This model provides better projection uniformity compared to conventional isometric projection and cube projection models, thereby reducing stretching and image boundary distortion to achieve a more ideal modeling effect. Finally, it generates a 3D model of the fisheye image using the optimized 3D Gaussian reconstruction algorithm, improving the realism of the 3D model reconstructed from the fisheye image.

[0120] Exemplary device

[0121] Based on the above embodiments, the present invention also provides a 3D Gaussian reconstruction system for fisheye images, comprising:

[0122] The point cloud generation module is used to acquire fisheye images and generate sparse 3D point clouds corresponding to the fisheye images based on the structure-reconstruction-motion algorithm.

[0123] The algorithm optimization module optimizes the 3D Gaussian reconstruction algorithm based on the invertible residual network and the octahedral projection model.

[0124] The 3D reconstruction module is used to reconstruct the 3D model of the fisheye image based on the optimized 3D Gaussian reconstruction algorithm and the sparse 3D point cloud.

[0125] In one implementation of this embodiment, the algorithm optimization module further includes:

[0126] The distortion model building module is used to build lens distortion models;

[0127] The reversible residual network optimization module is used to simulate the camera distortion parameters of the lens distortion model using a reversible residual network, and to add the camera distortion parameters to the 3D Gaussian reconstruction algorithm.

[0128] The octahedral projection model optimization module is used to replace the planar perspective model of the 3D Gaussian reconstruction algorithm with the octahedral projection model to obtain the optimized 3D Gaussian reconstruction algorithm.

[0129] This embodiment achieves the following technical effects through the above technical solution:

[0130] This embodiment optimizes the 3D Gaussian reconstruction algorithm through an algorithm optimization module. It simulates camera distortion parameters of the lens distortion model using a reversible residual network, performing end-to-end modeling and optimization of fisheye distortion parameters. Incorporating these camera distortion parameters into the 3D Gaussian reconstruction algorithm better reduces the 3D reconstruction error of fisheye images. Furthermore, it replaces the planar perspective model of the 3D Gaussian reconstruction algorithm with an octahedral projection model. This model offers better projection uniformity compared to conventional isometric and cubic projection models, thereby reducing stretching and image boundary distortion for a more ideal modeling effect. Finally, the optimized 3D Gaussian reconstruction algorithm generates a 3D model of the fisheye image, improving the realism of the 3D model reconstructed from the fisheye image using the 3D Gaussian algorithm.

[0131] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 9 As shown.

[0132] The terminal includes: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor of the terminal provides computing and control capabilities; the memory of the terminal includes a computer-readable storage medium and internal memory; the computer-readable storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the computer-readable storage medium; the interface is used to connect to external devices; the display screen is used to display relevant information; and the communication module is used to communicate with a cloud server or other devices.

[0133] When executed by the processor, this computer program is used to implement the 3D Gaussian reconstruction method for fisheye images.

[0134] It will be understood by those skilled in the art that Figure 9The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0135] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a 3D Gaussian reconstruction program for a fisheye image, the 3D Gaussian reconstruction program for a fisheye image being executed by the processor to implement the operation of the 3D Gaussian reconstruction method for the fisheye image as described above.

[0136] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a 3D Gaussian reconstruction program for a fisheye image, which, when executed by a processor, is used to implement the operation of the 3D Gaussian reconstruction method for the fisheye image described above.

[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include both non-volatile and volatile memory.

[0138] In summary, this invention provides a method, system, terminal, and storage medium for 3D Gaussian reconstruction of fisheye images, comprising: acquiring and preprocessing a multimodal image dataset to obtain an initial image dataset; inputting the initial image dataset into corresponding autoencoders to obtain corresponding time-invariant features, and pre-training the corresponding modal autoencoders by feature reconstruction; classifying the obtained features using the geographic location information of the initial image data to generate positive and negative sample pairs, and adjusting the trained autoencoders using a contrastive loss function constrained by cosine similarity; based on the adjusted autoencoders, extracting features with spatiotemporal invariant information from the corresponding modal images, and performing feature fusion by integrating spatiotemporal invariant information that takes into account modal importance; and outputting the fused image features. This invention improves the fusion accuracy of multimodal remote sensing image features.

[0139] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A 3D Gaussian reconstruction method for fisheye images, characterized in that, include: Acquire a fisheye image and generate a sparse 3D point cloud corresponding to the fisheye image based on the structure-reconstruction-motion algorithm. Optimize the 3D Gaussian reconstruction algorithm based on invertible residual network and octahedral projection model; Based on the optimized 3D Gaussian reconstruction algorithm and the sparse 3D point cloud, the 3D model of the fisheye image is reconstructed. The optimized 3D Gaussian reconstruction algorithm based on invertible residual networks and octahedral projection models includes: Construct a lens distortion model; The camera distortion parameters of the lens distortion model are simulated using an invertible residual network, and the camera distortion parameters are then incorporated into the 3D Gaussian reconstruction algorithm. The octahedral projection model is used to replace the planar perspective model of the 3D Gaussian reconstruction algorithm to obtain the optimized 3D Gaussian reconstruction algorithm. The step of replacing the planar perspective model of the 3D Gaussian reconstruction algorithm with the octahedral projection model includes: Based on the octahedral projection model, find the corresponding coordinates of each rendered pixel on the unit octahedral texture surface; Calculate the mapping relationship between the coordinates of the rendered pixel and the corresponding coordinates; The distortion rendering of the rendered pixels on the octahedral texture surface is realized according to the mapping relationship; The calculation of the mapping relationship between the coordinates of the rendered pixel and the corresponding coordinates includes: Define the coordinates of the rendered pixel as follows: The corresponding distorted coordinates are The corresponding coordinates on the octahedral texture surface are ; Calculate the mapping relationship based on the defined coordinates: ； Wherein, the corresponding coordinates of the octahedral texture surface satisfy... ; For the sign function.

2. The 3D Gaussian reconstruction method for fisheye images according to claim 1, characterized in that, The construction of the lens distortion model includes: A lens distortion model is constructed, which is defined as a ray transformation from the undistorted direction to the distorted direction.

3. The 3D Gaussian reconstruction method for fisheye images according to claim 1, characterized in that, The step of simulating camera distortion parameters of the lens distortion model using a reversible residual network and incorporating these camera distortion parameters into the 3D Gaussian reconstruction algorithm includes: Construct a reversible residual network; The camera distortion parameters of the lens distortion model are simulated using the reversible residual network. The camera distortion parameters are added to the 3D Gaussian reconstruction algorithm, and the camera distortion parameters are refined when the 3D Gaussian reconstruction algorithm optimizes the reconstruction parameters.

4. The 3D Gaussian reconstruction method for fisheye images according to claim 1, characterized in that, The step of reconstructing the 3D model of the fisheye image based on the optimized 3D Gaussian reconstruction algorithm and the sparse 3D point cloud includes: Based on the optimized 3D Gaussian reconstruction algorithm, the projection relationship from point coordinates to pixel coordinates in three-dimensional coordinates is calculated; Based on the projection relationship, the sparse 3D point cloud is modeled to reconstruct the 3D model of the fisheye image.

5. A 3D Gaussian reconstruction system for fisheye images, used to implement the 3D Gaussian reconstruction method for fisheye images as described in any one of claims 1-4, characterized in that, include: The point cloud generation module is used to acquire fisheye images and generate sparse 3D point clouds corresponding to the fisheye images based on the structure-reconstruction-motion algorithm. The algorithm optimization module optimizes the 3D Gaussian reconstruction algorithm based on the invertible residual network and the octahedral projection model. The 3D reconstruction module is used to reconstruct the 3D model of the fisheye image based on the optimized 3D Gaussian reconstruction algorithm and the sparse 3D point cloud.

6. A terminal, characterized in that, include: The processor and memory, the memory storing a 3D Gaussian reconstruction program for a fisheye image, the 3D Gaussian reconstruction program for a fisheye image being executed by the processor to implement the operation of the 3D Gaussian reconstruction method for a fisheye image as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a 3D Gaussian reconstruction program for a fisheye image, which, when executed by a processor, is used to implement the operation of the 3D Gaussian reconstruction method for a fisheye image as described in any one of claims 1-4.

Images