Underwater three-dimensional reconstruction method and system and storage medium

Abstract

The invention discloses an underwater three-dimensional reconstruction method and system and a storage medium, and the method comprises the steps: estimating a camera posture from a given image sequence through employing an SFM, and constructing a 3D sparse point cloud; circularly rendering underwater scenes of different camera postures by using 3DGS, initializing 3D sparse point clouds to obtain 3D Gaussian point clouds, and representing a rendered image corresponding to each camera posture through a group of 3D Gaussian point clouds; when a rendered picture is output through 3DGS cyclic rendering every time, calculating a residual error between the rendered picture and an input image in the image sequence, performing Fourier transform on the residual error, removing a high-frequency signal by using a low-pass filter, performing inverse Fourier transform on a reserved part, and optimizing all parameters of each Gaussian point by using parameters after inverse Fourier transform, so as to obtain a rendered picture; and inputting the 3D Gaussian point cloud after optimization of each Gaussian point back to circulating rendering of the 3DGS. A low-pass filtering technology is used in a 3DGS iteration process to construct a three-dimensional scene representation with consistent illumination, and ghost shadows are removed from the three-dimensional scene representation so as to recover a scene with uniform illumination.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an underwater three-dimensional reconstruction method, system and storage medium. Background Art

[0002] With the continuous progress of marine science and technology, underwater detection and 3D reconstruction technology are playing an increasingly important role in the development and utilization of marine resources. As a coastal country, my country has abundant marine resources, and the development of underwater detection technology is of great strategic significance to the country's economy and security. Therefore, the marine economy is placed in an important position.

[0003] As an important underwater detection vehicle, underwater robots play a key role in scanning and surveying underwater structures. Computer vision technology has made rapid progress in recent years, and 3D reconstruction technology has been widely used on land. However, due to the particularity of the underwater environment, traditional 3D reconstruction technology is difficult to directly apply in underwater scenes.

[0004] For example, 3DGS (3D Gaussian Splatting), a technology that has attracted widespread attention in the field of 3D reconstruction in recent years, is a point cloud reconstruction method based on Gaussian functions. However, this method is currently mainly used for scenes on land and cannot be directly applied to underwater environments. This is because in images taken underwater, the propagation of light is affected by the absorption and scattering of water, resulting in a decrease in image quality and the inability of light to accurately reach the target object or underwater structure, thus affecting the precision and accuracy of 3D reconstruction.

[0005] In the field of 3D reconstruction, SFM (Structure from Motion) is a basic method used to recover the camera pose and sparse 3D structure of the scene from a series of continuous images. However, the output of SFM is usually a sparse point cloud. Therefore, recent progress in the field of 3D reconstruction, such as NeRF (Neural Radiance Field) and 3DGS, is based on the results of SFM, and then achieves a new perspective rendering of the scene with real perception.

[0006] However, underwater light scattering is not conducive to building realistic perceptual representations based on SFM, while NeRF and 3DGS both further learn three-dimensional scenes by optimizing photometric loss rather than reprojection error. Therefore, these models are severely affected by inconsistent lighting. Existing studies are either based on 2D image filtering or supervised deep learning. However, methods based on 2D image filtering can play a certain role when the scene has small motion, but their performance may degrade for three-dimensional scenes with large motion. Methods based on deep learning require a large number of labeled datasets for training, while the labeling of underwater light scattering datasets requires a lot of manpower and has limited generalization performance. Therefore, deep learning methods have poor generalization performance when facing new observations. Summary of the invention

[0007] In order to solve the problem of afterimage caused by inconsistent lighting and light scattering in the existing three-dimensional construction of underwater environment, the present invention provides an underwater three-dimensional reconstruction method, system and storage medium to solve the above-mentioned technical defects.

[0008] In a first aspect, the present invention provides an underwater three-dimensional reconstruction method, the method comprising:

[0009] Use SFM to estimate the camera pose from a given image sequence and construct a 3D sparse point cloud;

[0010] 3DGS is used to perform loop rendering of underwater scenes with different camera postures. A 3D Gaussian point cloud is initialized from a 3D sparse point cloud. The rendered image corresponding to each camera posture is represented by a set of 3D Gaussian point clouds. Each Gaussian point in the 3D Gaussian point cloud is represented by a mean, a semi-positive covariance matrix, opacity, and a view-related color parameterized by a spherical harmonic function. When each 3DGS loop renders and outputs a rendered image, the residual between the rendered image and the input image in the image sequence is calculated, the residual is Fourier transformed, a low-pass filter is used to remove high-frequency signals, the retained part is Fourier inverse transformed, and all parameters of each Gaussian point are optimized using the parameters after the inverse Fourier transform. The optimized 3D Gaussian point cloud of each Gaussian point is input back into the loop rendering of 3DGS. Low-pass filtering technology is used in the 3DGS iteration process to construct a three-dimensional scene representation with consistent lighting, and residual images are removed from it to restore a scene with uniform lighting.

[0011] Preferably, using 3DGS to perform cyclic rendering of underwater scenes with different camera postures includes:

[0012] Initialize the 3D Gaussian point cloud from the 3D sparse point cloud. The rendered image corresponding to each camera pose is represented by a set of 3D Gaussian point clouds. Each Gaussian point in the 3D Gaussian point cloud is represented by the mean, semi-positive covariance matrix, opacity, and view-dependent color parameterized by spherical harmonics.

[0013] Project the 3D Gaussian point cloud into the 2D image to generate 2D Gaussian points; the two-dimensional covariance matrix Σ' of the 2D Gaussian points i It is expressed as:

[0014] Σ' i =(JWΣ i W T J T ) 1:2,1:2

[0015] Among them, J is the Jacobian matrix of the projective affine approximation, W is the weight matrix, and W T and J T is the transposed matrix of J and W, Σ i is the semi-positive definite covariance matrix of the corresponding Gaussian point in the 3D Gaussian point cloud, 1:2, 1:2 are the first two rows and the first two columns of the matrix respectively; the mean of the corresponding Gaussian point in the 3D Gaussian point cloud is projected onto the 2D image to obtain the mean of the 2D Gaussian point;

[0016] Calculate the color of each pixel. Project the 3D Gaussian point cloud into a 2D image to facilitate subsequent pixel processing.

[0017] Preferably, calculating the color of each pixel includes:

[0018] For each pixel, according to the distance from the 2D Gaussian point to the 2D image plane, traverse the Gaussian list from front to back to synthesize the color related to the view, where r represents the ray direction corresponding to the pixel;

[0019] Obtain pixel color using view-dependent color, 2D covariance matrix, and mean calculation of 2D Gaussian points It is expressed as:

[0020]

[0021] Among them, α i is the mixing weight, is the color associated with the view, ∑′ i is the two-dimensional covariance matrix, μ′ i is the mean of the 2D Gaussian points, where the mixing weights are learned from a set of images with a known projection matrix.

[0022] Preferably, the blending weights are learned from a set of images with known projection matrices, including computing the loss between predicted and true colors using DSSIM The mixed weight is obtained by adjustment, expressed as:

[0023]

[0024] in, is the predicted color and C is the true color.

[0025] Preferably, the residual R between the rendered image and the input image in the image sequence is calculated, which is expressed as:

[0026]

[0027] Where I is the input image, To render the image.

[0028] Preferably, the residual is subjected to Fourier transformation, a low-pass filter is used to remove high-frequency signals, and the retained part is subjected to inverse Fourier transformation, which is expressed as:

[0029] S = ifft(fft(R)[0k])

[0030] Among them, S represents caustics, R represents residual, fft and ifft represent Fourier transform and inverse Fourier transform respectively, and [0:k] means retaining the k frequencies with the smallest absolute value in the frequency space.

[0031] Preferably, all parameters of each Gaussian point are optimized using the parameters after inverse Fourier transformation, which is expressed as:

[0032]

[0033] Where G is the 3D Gaussian point cloud, I is the input image, To render the image, optimize G to optimize all parameters of each Gaussian point, dist() is the distance metric, S is the caustic, and has the same format as the training image, with RGB three channels. In the rendering process, low-pass filtering technology is used to remove the residual image caused by light scattering and reduce the high-frequency noise in the image, so as to achieve a three-dimensional reconstruction result with uniform illumination.

[0034] According to a second aspect of the present invention, an underwater three-dimensional system is provided, the system comprising:

[0035] A building module, configured to estimate the camera pose from a given image sequence using SFM and construct a 3D sparse point cloud;

[0036] The rendering module is configured to use 3DGS to perform cyclic rendering of underwater scenes with different camera postures; wherein, a 3D Gaussian point cloud is initialized from a 3D sparse point cloud, and a rendering image corresponding to each camera posture is represented by a set of 3D Gaussian point clouds, and each Gaussian point in the 3D Gaussian point cloud is represented by a mean, a semi-positive definite covariance matrix, an opacity, and a view-related color parameterized by a spherical harmonic function; when each 3DGS cyclic rendering outputs a rendered image, a residual between the rendered image and an input image in an image sequence is calculated, the residual is Fourier transformed, a low-pass filter is used to remove high-frequency signals, the retained part is Fourier inverse transformed, all parameters of each Gaussian point are optimized using the parameters after the Fourier inverse transformation, and the optimized 3D Gaussian point cloud of each Gaussian point is input back into the cyclic rendering of the 3DGS.

[0037] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, any of the above methods is implemented.

[0038] Compared with the prior art, the beneficial results of the present invention are:

[0039] The present invention relates to an underwater three-dimensional reconstruction method, system and storage medium, which aims to solve the afterimage problem caused by inconsistent illumination and light scattering in the three-dimensional construction of underwater environment in the prior art. The SFM is used to estimate the camera posture from a given image sequence to construct a 3D sparse point cloud; the underwater scene with different camera postures is cyclically rendered using 3DGS, and a 3D Gaussian point cloud is initialized from the 3D sparse point cloud. The rendered image corresponding to each camera posture is represented by a group of 3D Gaussian point clouds; when the 3DGS cyclic rendering outputs a rendered image each time, the residual between the rendered image and the input image in the image sequence is calculated, the residual is Fourier transformed, the high-frequency signal is removed by a low-pass filter, the retained part is Fourier inverse transformed, and all parameters of each Gaussian point are optimized by the parameters after the Fourier inverse transformation, and the 3D Gaussian point cloud after the optimization of each Gaussian point is input back into the cyclic rendering of 3DGS. By using low-pass filtering technology in the 3DGS iteration process, a three-dimensional scene representation with consistent illumination is constructed, and afterimages are removed therefrom to restore a scene with uniform illumination. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present invention. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0041] Figure 1 is a schematic flow chart of an underwater three-dimensional reconstruction method according to an embodiment of the present invention;

[0042] Figure 2 is a schematic structural diagram of an underwater three-dimensional reconstruction system according to an embodiment of the present invention;

[0043] Figure 3 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0044] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0045] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] In order to solve the afterimage problem caused by inconsistent lighting and light scattering in the three-dimensional construction of underwater environment in the prior art, the present application proposes an underwater three-dimensional reconstruction method, system and storage medium.

[0047] In a first aspect, this embodiment discloses an underwater 3D reconstruction method, such as Figure 1 As shown, the method includes:

[0048] Use SFM to estimate the camera pose from a given image sequence and construct a 3D sparse point cloud;

[0049] Specifically, SFM (Structure from Motion) is used as a basic method to recover the camera pose and the sparse 3D structure of the scene from a series of continuous images.

[0050] 3DGS is used to perform cyclic rendering of underwater scenes with different camera postures. A 3D Gaussian point cloud is initialized from a 3D sparse point cloud. The rendered image corresponding to each camera posture is represented by a set of 3D Gaussian point clouds. Each Gaussian point in the 3D Gaussian point cloud is represented by a mean μ i , the semi-positive definite covariance matrix Σ i, opacity and view-dependent colors parameterized by spherical harmonics; when each 3DGS loop renders and outputs a rendered image, the residual between the rendered image and the input image in the image sequence is calculated, the residual is Fourier transformed, a low-pass filter is used to remove high-frequency signals, the retained part is inversely Fourier transformed, all parameters of each Gaussian point are optimized using the parameters after the inverse Fourier transform, and the optimized 3D Gaussian point cloud of each Gaussian point is input back into the loop rendering of 3DGS. Low-pass filtering technology is used in the 3DGS iteration process to construct a three-dimensional scene representation with consistent lighting, and afterimages are removed from it to restore a scene with uniform lighting.

[0051] Specifically, 3DGS (3D Gaussian Splatting) is a point cloud reconstruction method based on Gaussian function.

[0052] Specifically, using 3DGS to perform cyclic rendering of underwater scenes with different camera postures includes:

[0053] Initialize the 3D Gaussian point cloud from the 3D sparse point cloud. The rendered image corresponding to each camera pose is represented by a set of 3D Gaussian point clouds. Each Gaussian point in the 3D Gaussian point cloud is represented by the mean, semi-positive covariance matrix, opacity, and view-dependent color parameterized by spherical harmonics.

[0054] Project the 3D Gaussian point cloud into the 2D image to generate 2D Gaussian points; the two-dimensional covariance matrix Σ' of the 2D Gaussian points i It is expressed as:

[0055]

[0056] Among them, J is the Jacobian matrix of the projective affine approximation, W is the weight matrix, and W T and J T is the transposed matrix of J and W, Σ i is the semi-positive definite covariance matrix of the corresponding Gaussian points in the 3D Gaussian point cloud, 1:2, 1:2 are the first two rows and the first two columns of the matrix respectively; the mean of the corresponding Gaussian points in the 3D Gaussian point cloud is projected into the 2D image to obtain the mean of the 2D Gaussian points; the 3D Gaussian point cloud is projected into the 2D image to facilitate the subsequent pixel processing.

[0057] Calculate the color of each pixel.

[0058] Furthermore, calculating the color of each pixel includes:

[0059] For each pixel, according to the distance from the 2D Gaussian point to the 2D image plane, traverse the Gaussian list from front to back to synthesize the color related to the view, where r represents the ray direction corresponding to the pixel;

[0060] Obtain pixel color using view-dependent color, 2D covariance matrix, and mean calculation of 2D Gaussian points It is expressed as:

[0061]

[0062] Among them, α i is the mixing weight, x is the position of the pixel in space, x-μ' i is the deviation between the pixel coordinates and the Gaussian mean, is the color associated with the view, Σ' i is the two-dimensional covariance matrix, μ' i is the mean of the 2D Gaussian points, where the mixing weights are learned from a set of images with a known projection matrix.

[0063] Furthermore, the blending weights are learned from a set of images with known projection matrices, including computing the loss between predicted and true colors using DSSIM. The mixed weight is obtained by adjustment, expressed as:

[0064]

[0065] in, is the predicted color and C is the true color.

[0066] It should be noted that 3DGS further defines an adaptive optimization process in which α i Gaussians with lower values ​​will be deleted, and 3D Gaussians with larger sizes will be split into multiple new Gaussian points to better represent the three-dimensional scene. In order to better solve the problem of natural light entering the water being scattered at different angles due to the undulations of the wave surface, this application adds a new optimization process on this basis. When each 3DGS loop rendering outputs a rendered image, the residual (residual image) between the rendered image and the input image in the image sequence is calculated, the residual is Fourier transformed, and a low-pass filter is used to remove high-frequency signals. The retained part is Fourier inverse transformed, and all parameters of each Gaussian point are optimized using the parameters after the inverse Fourier transform, and the optimized 3D Gaussian point cloud of each Gaussian point is input back into the loop rendering of 3DGS.

[0067] Specifically, the residual is Fourier transformed, a low-pass filter is used to remove high-frequency signals, and the retained part is inversely Fourier transformed, which is expressed as:

[0068] S = ifft(fft(R) [0:k] )

[0069] Among them, S represents caustics, R represents residual, fft and ifft represent Fourier transform and inverse Fourier transform respectively, and [0:k] represents retaining the k frequencies with the smallest absolute value in the frequency space.

[0070] Specifically, all parameters of each Gaussian point are optimized using the parameters after inverse Fourier transformation, which can be expressed as:

[0071]

[0072] Where G is the 3D Gaussian point cloud, I is the input image, To render the image, optimize G to optimize all parameters of each Gaussian point, dist() is the distance metric, S is the caustic, and has the same format as the training image, with RGB three channels. In the rendering process, low-pass filtering technology is used to remove the residual image caused by light scattering and reduce the high-frequency noise in the image, so as to achieve a three-dimensional reconstruction result with uniform illumination.

[0073] Figure 2 is an underwater 3D reconstruction system according to an embodiment of the present application, such as Figure 2 As shown, the system includes a construction module 201, which is configured to estimate the camera pose from a given image sequence using SFM and construct a 3D sparse point cloud;

[0074] The rendering module 202 is configured to use 3DGS to perform cyclic rendering on underwater scenes with different camera postures; wherein, a 3D Gaussian point cloud is initialized from a 3D sparse point cloud, and a rendered image corresponding to each camera posture is represented by a set of 3D Gaussian point clouds, and each Gaussian point in the 3D Gaussian point cloud is represented by a mean, a semi-positive definite covariance matrix, an opacity, and a view-related color parameterized by a spherical harmonic function; when each 3DGS cyclic rendering outputs a rendered image, a residual between the rendered image and an input image in the image sequence is calculated, the residual is Fourier transformed, a low-pass filter is used to remove high-frequency signals, the retained part is Fourier inverse transformed, all parameters of each Gaussian point are optimized using the parameters after the inverse Fourier transformation, and the optimized 3D Gaussian point cloud of each Gaussian point is input back into the cyclic rendering of the 3DGS.

[0075] Reference below Figure 3 , which shows a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. Figure 3 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0076] like Figure 3As shown, the computer system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage part 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the system 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0077] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0078] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0079] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0080] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0081] The modules involved in the embodiments of the present application may be implemented by software or by hardware.

[0082] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; using SFM to estimate the camera posture from a given image sequence and construct a 3D sparse point cloud; using 3DGS to perform cyclic rendering on underwater scenes with different camera postures; wherein, a 3D Gaussian point cloud is initialized from the 3D sparse point cloud, and the rendered image corresponding to each camera posture is represented by a set of 3D Gaussian point clouds, and each Gaussian point in the 3D Gaussian point cloud is represented by a mean, a semi-positive definite covariance matrix, an opacity, and a view-related color parameterized by a spherical harmonic function; when each 3DGS cyclic rendering outputs a rendered image, the residual between the rendered image and the input image in the image sequence is calculated, the residual is Fourier transformed, a low-pass filter is used to remove high-frequency signals, the retained part is Fourier inverse transformed, and the parameters after the Fourier inverse transformation are used to optimize all parameters of each Gaussian point, and the optimized 3D Gaussian point cloud of each Gaussian point is input back into the cyclic rendering of 3DGS. In the 3DGS iteration process, a low-pass filtering technique is used to construct a 3D scene representation with consistent lighting, and the residual image is removed from it to restore a scene with uniform lighting.

[0083] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

Claims

1. An underwater three-dimensional reconstruction method, characterized in that: include: Use SFM to estimate the camera pose from a given image sequence and construct a 3D sparse point cloud; 3DGS is used to perform cyclic rendering of underwater scenes with different camera postures; wherein a 3D Gaussian point cloud is initialized from the 3D sparse point cloud, and a rendered image corresponding to each camera posture is represented by a set of 3D Gaussian point clouds, and each Gaussian point in the 3D Gaussian point cloud is represented by a mean, a semi-positive covariance matrix, an opacity, and a view-dependent color parameterized by a spherical harmonic function; When each 3DGS rendering cycle outputs a rendered image, the residual between the rendered image and the input image in the image sequence is calculated, the residual is Fourier transformed, a low-pass filter is used to remove high-frequency signals, the retained part is inverse Fourier transformed, all parameters of each Gaussian point are optimized using the parameters after the inverse Fourier transform, and the 3D Gaussian point cloud after the optimization of each Gaussian point is input back into the 3DGS rendering cycle.

2. The method according to claim 1, characterized in that The cyclic rendering of underwater scenes with different camera postures by using 3DGS includes: Initializing a 3D Gaussian point cloud from the 3D sparse point cloud, wherein the rendered image corresponding to each camera pose is represented by a set of 3D Gaussian point clouds, wherein each Gaussian point in the 3D Gaussian point cloud is represented by a mean, a semi-positive definite covariance matrix, an opacity, and a view-dependent color parameterized by a spherical harmonic function; The 3D Gaussian point cloud is projected into a 2D image to generate 2D Gaussian points; wherein the two-dimensional covariance matrix Σ' of the 2D Gaussian points i It is expressed as: In i =(JWΣ i W T J T ) 1:2,1:2 Among them, J is the Jacobian matrix of the projective affine approximation, W is the weight matrix, and W T and J T is the transposed matrix of J and W, Σ i is the semi-positive definite covariance matrix of the corresponding Gaussian point in the 3D Gaussian point cloud, 1:2, 1:2 are the first two rows and the first two columns of the matrix respectively; the mean of the corresponding Gaussian point in the 3D Gaussian point cloud is projected onto the 2D image to obtain the mean of the 2D Gaussian point; Calculate the color of each pixel.

3. The method according to claim 2, characterized in that Calculating the color of each pixel includes: For each pixel, according to the distance from the 2D Gaussian point to the 2D image plane, the Gaussian list is traversed from front to back to synthesize the color associated with the view, where r represents the ray direction corresponding to the pixel; Obtain pixel color using the view-related color, the two-dimensional covariance matrix, and the mean of the 2D Gaussian points It is expressed as: Among them, x is the position of the pixel in space, α i is the mixing weight, is the view-related color, Σ' i is the two-dimensional covariance matrix, μ' i is the mean of the 2D Gaussian points, wherein the mixing weights are learned from a set of images with a known projection matrix.

4. The method according to claim 3, characterized in that The blending weights are learned from a set of images with known projection matrices, including computing the loss between predicted and true colors using DSSIM The mixed weight is obtained by adjustment, which is expressed as: in, is the predicted color and C is the true color.

5. The method according to claim 1, characterized in that The residual R of the rendered image and the input image in the image sequence is calculated as: Wherein, I is the input image, for said rendered image.

6. The method according to claim 1, characterized in that The residual is subjected to Fourier transformation, a low-pass filter is used to remove high-frequency signals, and the retained part is subjected to inverse Fourier transformation, which is expressed as: S=ifft(fft(R) [0:k] ) Among them, S represents caustics, R represents residual, fft and ifft represent Fourier transform and inverse Fourier transform respectively, and [0:k] represents retaining the k frequencies with the smallest absolute value in the frequency space.

7. The method according to claim 1, characterized in that The parameters after inverse Fourier transformation are used to optimize all parameters of each Gaussian point, which is expressed as: Where G is the 3D Gaussian point cloud, I is the input image, For the rendered image, optimizing G is to optimize all parameters of each Gaussian point, dist() is the distance metric, S is the caustic, and has the same format as the training image, with RGB three channels.

8. An underwater three-dimensional reconstruction system, characterized in that: include: A building module, configured to estimate the camera pose from a given image sequence using SFM and construct a 3D sparse point cloud; A rendering module is configured to perform cyclic rendering of underwater scenes with different camera postures using 3DGS; wherein a 3D Gaussian point cloud is initialized from the 3D sparse point cloud, and a rendering image corresponding to each camera posture is represented by a set of 3D Gaussian point clouds, and each Gaussian point in the 3D Gaussian point cloud is represented by a mean, a semi-positive definite covariance matrix, an opacity, and a view-related color parameterized by a spherical harmonic function; When each 3DGS rendering cycle outputs a rendered image, the residual between the rendered image and the input image in the image sequence is calculated, the residual is Fourier transformed, a low-pass filter is used to remove high-frequency signals, the retained part is inverse Fourier transformed, all parameters of each Gaussian point are optimized using the parameters after the inverse Fourier transform, and the 3D Gaussian point cloud after the optimization of each Gaussian point is input back into the 3DGS rendering cycle.

9. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that: When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 7 is implemented.

Images