Underwater scene three-dimensional reconstruction method and system based on fusion of sonar and optical images

Abstract

This invention discloses a method and system for underwater scene 3D reconstruction based on sonar and optical image fusion, to obtain more accurate underwater scene 3D reconstruction results. The method includes: performing time synchronization, spatial calibration, and coordinate system unification on a camera and sonar mounted on a carrier, and using the carrier's pose matrix in the world coordinate system as a differentiable parameter; initializing 3D Gaussian primitives embedded with acoustic reflectivity attributes using sparse point clouds; optimizing the 3D Gaussian primitives by using sonar echo intensity fitting to generate radial centripetal force, anchoring the center of the 3D Gaussian primitives on a sampling arc with the sonar position as the center and the sonar measured distance as the radius; driving the 3D Gaussian primitives to slide along the sampling arc using arc constraint loss and photometric loss gradients; and outputting the underwater scene 3D reconstruction rendering result based on the optimized 3D Gaussian primitives.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional reconstruction technology, specifically, it relates to underwater scene three-dimensional reconstruction technology, and more specifically, it relates to an underwater scene three-dimensional reconstruction method and system based on sonar and optical image fusion. Background Technology

[0002] Underwater scene 3D reconstruction technology is a core foundation for marine scientific research, deep-sea resource exploration, underwater cultural heritage protection, and autonomous navigation of underwater robots. However, the extreme complexity of the underwater environment presents a far greater technological challenge to this task than that of the terrestrial environment.

[0003] From a physical optics perspective, light propagation in water is subject to severe absorption and scattering. Red light attenuates rapidly in shallow water, while blue-green light has strong penetrating power. This results in a significant tone shift in underwater images, exhibiting a distinct blue-green hue. Before reaching the camera sensor, light undergoes a triple evolution: direct component, forward scattering, and backscattering. Backscattering, caused by suspended particles in the water, forms a thick "white fog" on the image, greatly reducing scene contrast and visibility. This makes traditional feature-point matching-based computer vision algorithms highly susceptible to failure in turbid waters.

[0004] In situations where visual perception is limited, forward-looking sonar, as an active acoustic detection method, exhibits extremely high tolerance to turbidity and can achieve long-range scene perception. However, acoustic imaging has its inherent geometric limitations: forward-looking sonar records echoes using a polar coordinate system, but due to the physical width of its beam in the vertical dimension, elevation angle information is lost. This means that one sonar pixel corresponds to an arc in three-dimensional space, making it impossible to reconstruct an accurate three-dimensional shape using only a single sonar device.

[0005] In recent years, 3D Gaussian Splatting (3DGS), as a strong competitor to Neural Radiation Field (NeRF), has attracted widespread attention due to its explicit point cloud representation and extremely high rendering speed. However, the application of 3DGS in underwater scenes is still in its early stages and has obvious limitations: due to the lack of effective geometric constraints, 3DGS exhibits extreme geometric instability in the reconstruction of underwater seabeds with scarce textures, easily generating a large number of floating artifacts. At the same time, 3DGS cannot solve the problem of entangled underwater reconstruction parameters caused by the coupling of depth and attenuation parameters in underwater optical imaging. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing underwater scene 3D reconstruction technologies and to provide a method and system for underwater scene 3D reconstruction based on sonar and optical image fusion and 3D Gaussian splashing.

[0007] To achieve the above-mentioned objectives, the underwater scene 3D reconstruction method provided by this invention employs the following technical solution:

[0008] A method for 3D reconstruction of underwater scenes based on sonar and optical image fusion includes:

[0009] Time synchronization and spatial calibration of the camera and forward-looking sonar mounted on the carrier;

[0010] Using the carrier coordinate system as the pivot, and based on the carrier's pose matrix in the world coordinate system, camera observations and sonar observations are unified to the world coordinate system; at the same time, the carrier's pose matrix in the world coordinate system is used as a differentiable parameter.

[0011] A sparse point cloud of the scene is obtained, and a three-dimensional Gaussian primitive is initialized using the sparse point cloud; the three-dimensional Gaussian primitive is embedded with an acoustic reflectivity attribute.

[0012] Optimize the three-dimensional Gaussian primitives, and output the three-dimensional reconstruction rendering result of the underwater scene based on the optimized three-dimensional Gaussian primitives;

[0013] Optimizing the three-dimensional Gaussian elements includes:

[0014] A physical law rendering branch for sonar imaging is constructed, acoustic loss is calculated, and radial centripetal force is generated by fitting the sonar echo intensity. The center of the three-dimensional Gaussian element is anchored on a sampling arc with the sonar position as the center and the actual sonar distance as the radius.

[0015] The Euclidean distance from the center of the 3D Gaussian primitive to the sampling arc is calculated to obtain the arc constraint loss. The arc constraint loss is used to restrict the center of the 3D Gaussian primitive to slide along the sampling arc. At the same time, the photometric loss of the image obtained by the camera is used to generate a photometric loss gradient as a tangential driving force to drive the 3D Gaussian primitive to slide along the sampling arc, so that the 3D Gaussian primitive converges to the actual 3D coordinates of the object.

[0016] In some embodiments of this application, optimizing the three-dimensional Gaussian elements further includes:

[0017] A joint loss function is constructed, comprising the photometric loss, the acoustic loss, the arc constraint loss, and the regularization loss, to jointly optimize the properties of the three-dimensional Gaussian elements and the pose matrix of the carrier in the world coordinate system.

[0018] In some embodiments of this application, optimizing the three-dimensional Gaussian elements further includes:

[0019] The system calculates in real time the standard deviation of chromaticity, mean saturation, and contrast of the current frame image obtained by the camera to obtain the current value of the quality evaluation index of the color image.

[0020] When the current value of the quality evaluation index is less than the index threshold and the contrast of the image is less than the contrast threshold, the image enters the optical degradation region, increases the weight of the acoustic loss and the weight of the arc constraint loss, and decreases the weight of the photometric loss; when the current value of the quality evaluation index is not less than the index threshold, the weight of the photometric loss is increased.

[0021] In some embodiments of this application, the rendering results of underwater scene 3D reconstruction based on optimized 3D Gaussian primitives are further included:

[0022] A spatial octree index structure is established for the set of three-dimensional Gaussian elements, and each node stores a subset of the three-dimensional Gaussian elements in the local space corresponding to the node.

[0023] For non-leaf nodes, feature coarsening is performed based on the opacity and scaling of the three-dimensional Gaussian primitives to generate a multi-level hierarchical structure.

[0024] In some embodiments of this application, the rendering results of underwater scene 3D reconstruction based on optimized 3D Gaussian primitives are included:

[0025] The system monitors the current rendering frame rate of the 3D Gaussian primitives in real time. When the current rendering frame rate is less than the target frame rate threshold, it increases the clipping threshold of the opacity of the 3D Gaussian primitives. It also degrades the octree tiles in non-critical view areas according to the current set pose of the camera, and preloads the medium-level tiles in front of the path and dynamically releases the high-frequency texture blocks far away from the view area behind the carrier's motion vector.

[0026] In some embodiments of this application, obtaining a sparse point cloud of a scene and initializing a 3D Gaussian primitive using the sparse point cloud includes:

[0027] Pixels whose energy intensity exceeds the intensity threshold in the sonar echo are obtained, and the sparse point cloud of the scene is generated by combining the pose matrix of the carrier in the world coordinate system and the static transformation matrix of the sonar coordinate system relative to the carrier coordinate system.

[0028] The sparse point cloud of the scene is used as the initial position of the three-dimensional Gaussian primitive.

[0029] In some embodiments of this application, the camera and forward-looking sonar mounted on the carrier are spatially calibrated in the following manner:

[0030] Obtain the camera's intrinsic parameters;

[0031] Obtain a quasi-pinhole model with underwater refraction correction;

[0032] Obtain the static transformation matrix of the camera coordinate system relative to the carrier coordinate system, the static transformation matrix of the sonar coordinate system relative to the carrier coordinate system, and the relative transformation matrix between the camera coordinate system and the sonar coordinate system.

[0033] In some embodiments of this application, time synchronization is performed between the camera mounted on the carrier and the forward-looking sonar, including:

[0034] The main control chip sends a synchronization pulse signal to achieve hard-triggered synchronization between camera exposure time and sonar pulse time.

[0035] To achieve the aforementioned objectives, the underwater scene 3D reconstruction system provided by this invention employs the following technical solution:

[0036] A three-dimensional underwater scene reconstruction system based on sonar and optical image fusion includes:

[0037] The time synchronization module is used to synchronize the time between the camera and the forward-looking sonar mounted on the carrier.

[0038] The spatial calibration module is used to perform spatial calibration on the camera and forward-looking sonar mounted on the carrier.

[0039] The coordinate system transformation module is used to unify camera observations and sonar observations to the world coordinate system, using the carrier coordinate system as the pivot and the carrier's pose matrix in the world coordinate system; at the same time, the carrier's pose matrix in the world coordinate system is used as a differentiable parameter.

[0040] The sparse point cloud acquisition module is used to obtain sparse point clouds of the scene;

[0041] A 3D Gaussian primitive initialization module is used to initialize 3D Gaussian primitives using the sparse point cloud; the 3D Gaussian primitives are embedded with acoustic reflectivity properties.

[0042] A three-dimensional Gaussian element optimization module is used to optimize the three-dimensional Gaussian element;

[0043] The reconstruction rendering result output module is used to output the 3D reconstruction rendering result of the underwater scene based on the optimized 3D Gaussian primitives.

[0044] The 3D Gaussian element optimization module optimizes the 3D Gaussian element, including:

[0045] A physical law rendering branch for sonar imaging is constructed, acoustic loss is calculated, and radial centripetal force is generated by fitting the sonar echo intensity. The center of the three-dimensional Gaussian element is anchored on a sampling arc with the sonar position as the center and the actual sonar distance as the radius.

[0046] The Euclidean distance from the center of the 3D Gaussian primitive to the sampling arc is calculated to obtain the arc constraint loss. The arc constraint loss is used to restrict the center of the 3D Gaussian primitive to slide along the sampling arc. At the same time, the photometric loss of the image obtained by the camera is used to generate a photometric loss gradient as a tangential driving force to drive the 3D Gaussian primitive to slide along the sampling arc, so that the 3D Gaussian primitive converges to the actual 3D coordinates of the object.

[0047] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the above-described method for three-dimensional reconstruction of underwater scenes based on sonar and optical image fusion is implemented.

[0048] Compared with the prior art, the advantages and positive effects of the present invention are:

[0049] This invention provides a method and system for 3D underwater scene reconstruction based on sonar and optical image fusion. It employs 3D Gaussian splashing for underwater scene reconstruction. When optimizing the 3D Gaussian primitives, a rendering branch based on the physical laws of sonar imaging is constructed. Radial centripetal force is generated by fitting the sonar echo intensity, anchoring the center of the 3D Gaussian primitive on a sampling arc with the sonar position as the center and the measured sonar distance as the radius. This achieves the anchoring of the geometric depth of the 3D Gaussian primitive using precise sonar measurements, effectively solving the geometric instability problem existing in 3D Gaussian splashing scene reconstruction. Furthermore, it achieves decoupling of depth and attenuation parameters in underwater optical imaging at the physical level, effectively addressing the issues that exist during reconstruction. The problem of multiple solutions caused by parameter entanglement is addressed. By calculating the Euclidean distance from the center of the 3D Gaussian primitive to the sampling arc, the arc constraint loss is obtained to further lock the physical dimension accuracy of the reconstruction result, prevent geometric derailment, and further improve the geometric stability of the reconstruction process. Furthermore, the photometric loss of the camera image is used to generate a tangential driving force, which drives the 3D Gaussian primitive to slide along the sampling arc and converge to the actual 3D coordinates of the object. The high angular resolution of the optical vision system is used to compensate for the perception defects of the sonar system in the vertical dimension, effectively solving the dimensional collapse problem caused by the loss of pitch angle information of the sonar system. Using the above-mentioned optimized 3D Gaussian primitive, more accurate 3D reconstruction results of underwater scenes can be obtained.

[0050] Other features and advantages of the present invention will become clearer after reading the detailed embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 Flowcharts of some embodiments of the underwater scene 3D reconstruction method based on sonar and optical image fusion provided by the present invention;

[0053] Figure 2 The diagram shows the structural block diagrams of some embodiments of the underwater scene 3D reconstruction system based on sonar and optical image fusion provided by the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0055] It should be noted that the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they can be implemented by those skilled in the art. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0056] To address the problems existing in current technologies that rely on 3D Gaussian splashing to reconstruct underwater scenes or solely on sonar equipment, this invention creatively proposes a method and system for 3D underwater scene reconstruction based on the fusion of sonar and optical images. This method uses acoustic images acquired by sonar and optical images acquired by a camera, along with 3D Gaussian splashing, to perform 3D underwater scene reconstruction, achieving accurate reconstruction results.

[0057] Figure 1 The flowchart shown is a flowchart of some embodiments of the underwater scene 3D reconstruction method based on sonar and optical image fusion provided by the present invention. Specifically, it is a flowchart of some embodiments of underwater scene 3D reconstruction using 3D Gaussian splashing.

[0058] like Figure 1 As shown, this embodiment uses the following process to reconstruct an underwater three-dimensional scene.

[0059] Step S11: Perform time synchronization and spatial calibration of the camera and forward-looking sonar mounted on the carrier.

[0060] The purpose of this step is to synchronize the carrier, the camera mounted on the carrier, and the forward-looking sonar in both time and space. This step can be implemented using existing technologies.

[0061] In some embodiments, time synchronization is achieved using hardware-level synchronization. Specifically, the main control chip sends a synchronization pulse signal to achieve hard-triggered synchronization between the camera exposure time and the sonar pulse time.

[0062] In some embodiments, spatial calibration is performed using the following hierarchical calibration method:

[0063] The camera's intrinsic parameters are obtained to achieve land-based intrinsic calibration. In some embodiments, the camera's intrinsic parameter matrix K (focal length, principal point) and lens distortion coefficients are obtained using a standard checkerboard calibration board in an air environment, thereby locking in the camera's inherent properties.

[0064] Obtain a quasi-pinhole model for underwater refraction correction. To address the refraction effects in underwater imaging, in some embodiments, the camera is embedded within a waterproof housing and submerged underwater. A multi-layer interface projection model based on Snell's law is established, and the normal vector and offset distance of the waterproof housing relative to the camera are calculated. Through pre-calculation, the mapping from pixel points to 3D rays is corrected into a quasi-pinhole model to eliminate systematic errors caused by distance-dependent reconstructed scale variations due to medium refraction, and to ensure that the underlying geometric calculations can directly adapt to the 3D Gaussian splashing rasterization operator.

[0065] Obtain the static transformation matrix of the camera coordinate system relative to the carrier coordinate system, the static transformation matrix of the sonar coordinate system relative to the carrier coordinate system, and the relative transformation matrix between the camera coordinate system and the sonar coordinate system. In some embodiments, in a controlled water body, the static transformation matrix of the camera coordinate system relative to the carrier coordinate system is solved by using acoustic-optical joint calibration of the target. and the static transformation matrix of the sonar coordinate system relative to the carrier coordinate system. Based on static transformation matrix and static transformation matrix The relative transformation matrix between the camera coordinate system and the sonar coordinate system can be derived. .

[0066] Step S12: Using the carrier coordinate system as the pivot, unify camera observations and sonar observations to the world coordinate system based on the carrier's pose matrix in the world coordinate system.

[0067] The purpose of this step is to unify the coordinate systems of the carrier, camera, and sonar in space, aligning their respective observations to the same coordinate system for easier processing.

[0068] Specifically, in this embodiment, after determining the transformation matrix between the carrier, camera, and sonar in step S11, the carrier coordinate system is used as a pivot to uniformly align both the camera observation and the sonar observation, which are local observations, to the global world coordinate system.

[0069] The carrier coordinate system has its origin at the carrier's center of mass. By combining the navigation system with a velocimeter, inertial measurement unit, and pressure gauge, the carrier's pose matrix in the world coordinate system can be obtained in real time. .

[0070] For 3D points in the world coordinate system Its coordinates in the camera coordinate system and coordinates in sonar coordinates They can be expressed by the following formulas respectively:

[0071] ; .

[0072] Based on the various transformation matrices, it is also possible to convert the coordinates of points in the camera coordinate system or the sonar coordinate system into coordinates in the world coordinate system.

[0073] Meanwhile, in this embodiment, the pose matrix of the carrier in the world coordinate system is... As a differentiable parameter, it can be incorporated into the optimization process of differentiable rendering and backpropagation, and automatically adjusted through the loss function during the model training loop.

[0074] This embodiment uses the pose matrix of the carrier in the world coordinate system as a differentiable parameter and the carrier coordinate system as a pivot for coordinate unification, which can effectively absorb the dynamic pose error caused by the cumulative drift of the velocimeter or the interference of the magnetometer.

[0075] Step S13: Obtain the sparse point cloud of the scene and initialize the 3D Gaussian primitives using the sparse point cloud.

[0076] In this embodiment, the three-dimensional Gaussian element, in addition to having position... Rotation matrix Scaling vector Opacity spherical harmonic coefficients In addition to standard attributes, acoustic reflectivity is also embedded. The initial value of acoustic reflectivity can be set according to the original echo intensity (pixel brightness) of the sonar image; the greater the echo intensity, the greater the initial value of acoustic reflectivity. By embedding acoustic reflectivity attributes into the 3D Gaussian primitives, the 3D Gaussian primitives are able to carry sound wave energy during the initialization phase, providing physical weights for their subsequent gradient-driven spatial migration.

[0077] The acquisition method of sparse point cloud of scene and the specific method of initializing 3D Gaussian primitives based on sparse point cloud of scene can adopt existing technologies, and this embodiment does not limit them.

[0078] In some embodiments, obtaining a sparse point cloud of the scene and initializing a 3D Gaussian primitive using the sparse point cloud includes:

[0079] Identify pixels in the sonar echo whose energy intensity exceeds an intensity threshold, and combine this with the pose matrix of the carrier in the world coordinate system. The static transformation matrix of the sonar coordinate system relative to the carrier coordinate system Generate sparse point clouds of the scene.

[0080] Specifically, in the sonar polar coordinate system, pixels whose sonar echo energy intensity exceeds an intensity threshold are acquired, and their polar coordinates, including radial distance and azimuth angle, are recorded. Adhering to the assumption of a flat seabed (assuming the sonar is perpendicular to the ground and the target is directly below the sonar), the pitch angle is set to 0°. In the sonar coordinate system, the polar coordinates of pixels exceeding the intensity threshold are converted to Cartesian coordinates to obtain local points in the sonar coordinate system. Then, the pose matrix of the carrier in the world coordinate system, acquired in real time through the navigation system combined with a velocimeter, inertial measurement unit, and pressure gauge, is used. and the static transformation matrix of the sonar coordinate system relative to the carrier coordinate system This transforms local points in the sonar coordinate system into sparse point clouds of the scene in the world coordinate system.

[0081] Then, the sparse point cloud of the scene is used as the initial position of the 3D Gaussian primitive.

[0082] By initializing 3D Gaussian primitives using sparse point clouds of the scene generated by the sonar coordinate system, the precise radial distance of the sonar can be used to provide an absolute scale reference for 3D reconstruction, thus avoiding scale collapse.

[0083] Step S14: Optimize the 3D Gaussian primitives and output the 3D reconstruction rendering results of the underwater scene based on the optimized 3D Gaussian primitives.

[0084] To improve the accuracy of the reconstruction results, a method including the following steps is used to optimize the three-dimensional Gaussian elements:

[0085] A rendering branch based on the physical laws of sonar imaging is constructed, acoustic loss is calculated, and radial centripetal force is generated by fitting the sonar echo intensity. Using the radial centripetal force, the center of the 3D Gaussian element is anchored on a sampling arc with the sonar position as the center and the measured sonar distance as the radius.

[0086] Sonar echo intensity is primarily affected by detection distance and reflectivity, and is almost entirely unaffected by the optical turbidity of the water. Based on this, the functional expression for acoustic loss is defined as follows:

[0087] .

[0088] in, Acoustic loss; For sonar at the measured distance of The measured azimuth angle is The intensity of the sonar echo detected at the actual target point can be obtained by reading the pixel values ​​of the sonar image, and is a known value; The sequence number of the three-dimensional Gaussian unit; The number of three-dimensional Gaussian elements is determined by the sonar hardware structure and can be determined based on the sonar's beamwidth and scanning angle range. For the first The acoustic reflectivity property of each three-dimensional Gaussian element is a differentiable parameter. Its initial value can be set according to the original echo intensity (pixel brightness) of the sonar image and is automatically optimized and adjusted during the optimization process. For the first The opacity property of each 3D Gaussian element is a differentiable parameter. Its initial value can be assigned as needed or empirically, and it will be automatically optimized and adjusted during the optimization process. ), ( ) are the measured target point and the first Radial distance and angular deviation of the center point of each three-dimensional Gaussian element; This is the point spread function, used to simulate the detection blurring effect of sonar. Its function expression is determined by the sonar hardware structure and can be obtained from the sonar parameter manual.

[0089] During the training iterations of optimizing 3D Gaussian primitives, the gradient flow generated by the acoustic loss function manifests as an acoustic radial centripetal force in polar coordinates. When there is a deviation between the center position of the 3D Gaussian primitive and the measured sonar distance, this radial centripetal force drives the 3D Gaussian primitive to shift towards the measured distance, thereby anchoring the center of the 3D Gaussian primitive on a sampling arc with the sonar position as the center and the measured sonar distance as the radius. This achieves the anchoring of the geometric depth of the 3D Gaussian primitive using sonar measurements with precise dimensions, effectively solving the geometric instability problem in 3D Gaussian splash reconstruction scenes.

[0090] According to the underwater optical physical imaging model theory, the observed pixel intensity is related to the depth of optical imaging and the water attenuation coefficient. During training and optimization, to reduce reprojection errors, either the depth or the water attenuation coefficient can be reduced to fit the darkened image. This compensatory fitting method often results in the reconstructed model losing its true physical proportions. In this embodiment, the geometric depth of the three-dimensional Gaussian elements is anchored using sonar measured distances. This eliminates the possibility of modifying the imaging depth to absorb residuals in the optical branch. Therefore, during training and optimization, the only way to fit the image's color cast and degradation is to find the true distribution of the water attenuation coefficient. This achieves decoupling of depth and attenuation parameters in underwater optical imaging at the physical level, effectively solving the problem of multiple solutions caused by parameter entanglement during reconstruction and improving the accuracy of the reconstruction results.

[0091] Considering the limitation of forward-looking sonar, which can measure object distance but not height, this study, in addition to optimizing the 3D Gaussian basis using radial centripetal force constraints, also utilizes the high angular resolution of the optical vision system to compensate for the sonar system's perception deficiency in the vertical dimension. Specifically, the Euclidean distance from the center of the 3D Gaussian primitive to the sampling arc is calculated to obtain the arc constraint loss, which is used to restrict the sliding of the 3D Gaussian primitive's center along the sampling arc. Simultaneously, a photometric loss gradient is generated using the photometric loss of the image obtained from the camera, serving as a tangential driving force to drive the 3D Gaussian primitive to slide along the sampling arc, causing it to converge to the actual 3D coordinates of the object.

[0092] The expression for the arc constraint loss is as follows: .

[0093] in, For arc constraint loss; For sampling arcs; The coordinates of the center of the three-dimensional Gaussian element located on the sampling arc; For the current optimization iteration The position of a 3D Gaussian element represents the attribute of the 3D Gaussian element to be optimized.

[0094] The arc constraint loss serves as the geometric guide for the 3D Gaussian primitive (GRP). By calculating the Euclidean distance from the center of the GRP to the sampling arc, it penalizes any displacement behavior that deviates from the sampling arc track during the optimization process. This ensures that when the GRP is subsequently driven by the tangential force of the gradient, its center point can only slide on a track with a radius equal to the sonar measured distance. This locks in the physical dimension accuracy of the reconstruction result, prevents geometric derailment, and further improves the geometric stability of the reconstruction process.

[0095] In optimizing the 3D Gaussian primitives, a virtual camera image is rendered based on the current 3D Gaussian primitives. This virtual image is then compared pixel-by-pixel with the real camera image to obtain the photometric loss. The photometric loss is differentiated to generate a photometric loss gradient. This gradient serves as the tangential driving force guiding the movement of the 3D Gaussian primitives, causing them to slide along a sampling arc to find the angle position that best matches the rendered image with the actual image captured by the camera. This allows the 3D Gaussian primitives to converge to the actual 3D coordinates of the object.

[0096] By employing a combination of geometric guide rail positioning and tangential driving force generated by an optical vision system, the high angular resolution of the optical vision system is used to compensate for the perception defects of the sonar system in the vertical dimension, effectively solving the problem of dimensional collapse caused by the loss of pitch angle information of the sonar system.

[0097] By anchoring the geometric depth of the 3D Gaussian primitives using sonar measurements to lock the distance, and combining the tangential driving force generated by the optical vision system to lock the height, the 3D Gaussian primitives eventually converge to the true 3D coordinates of the object. This achieves a precise conversion from a spatial arc to a physical point, effectively solving the problem of height ambiguity in the reconstruction of large underwater scenes.

[0098] In some other embodiments, the optimization of the three-dimensional Gaussian elements includes the following optimization process:

[0099] A joint loss function, comprising photometric loss, acoustic loss, arc constraint loss, and regularization loss, is constructed to jointly optimize the attributes of the 3D Gaussian primitives and the pose matrix of the carrier in the world coordinate system. The determination methods for photometric loss, acoustic loss, and arc constraint loss are described in the above embodiments. The determination method for regularization loss adopts existing technology, aiming to address the noise problem in underwater environments.

[0100] When optimizing the 3D Gaussian primitives, not only are the attributes of the 3D Gaussian-based primitives optimized using the joint loss function, but the pose matrix of the vehicle in the world coordinate system is also optimized to achieve real-time fine-tuning of the vehicle's navigation drift and ensure the consistency and alignment between the reconstructed underwater scene and the vehicle's motion trajectory.

[0101] In the constructed joint loss function, which includes photometric loss, acoustic loss, arc constraint loss, and regularization loss, each loss has a weight. The joint loss is obtained by summing the products of each loss and its corresponding weight. In some embodiments, to improve the robustness of the 3D reconstruction system under different water turbidity levels, at least some weights are dynamically and adaptively adjusted based on the quality evaluation index values ​​of the color image. The specific implementation process includes:

[0102] The system calculates the chromaticity standard deviation, saturation mean, and contrast of the current frame image obtained from the camera in real time to obtain the current value of the quality evaluation index of the color image.

[0103] The current value of the quality evaluation metric is compared with the metric threshold, and the contrast of the current frame image is compared with the contrast threshold. Both the metric threshold and the contrast threshold are preset values. When the current value of the quality evaluation metric is less than the metric threshold and the contrast of the current frame image is less than the contrast threshold, the system enters the optical degradation region. The weights of acoustic loss and arc constraint loss are increased, while the weight of photometric loss is decreased. The purpose is to force entry into acoustic guidance mode when optical information is unreliable, utilizing the sonar's penetration to determine the geometric framework of the scene. Conversely, if the current value of the quality evaluation metric is not less than the metric threshold, the weight of photometric loss is increased to refine the albedo details of objects in the scene.

[0104] When rendering underwater scene 3D reconstruction results based on optimized 3D Gaussian primitives, the reconstruction results are typically visualized. To reduce the hardware computing power pressure during rendering visualization, in some embodiments, the rendering results of underwater scene 3D reconstruction based on optimized 3D Gaussian primitives include:

[0105] A spatial octree index structure is established for the set of three-dimensional Gaussian elements, and each node stores a subset of the three-dimensional Gaussian elements in the local space corresponding to the node.

[0106] For non-leaf nodes, feature coarsening is performed based on the opacity and scaling properties of the 3D Gaussian primitives to generate a multi-level hierarchical structure.

[0107] By establishing a spatial octree index structure, the retrieval efficiency of 3D Gaussian primitives during reconstruction can be improved. Furthermore, by employing a multi-level hierarchical structure, only a very small number of representative 3D Gaussian primitives need to be loaded when rendering distant areas. This reduces the computational burden on hardware during rendering and visualization.

[0108] In other embodiments, the rendering results of the underwater scene 3D reconstruction based on the optimized 3D Gaussian primitives also include:

[0109] The system monitors the current rendering frame rate of 3D Gaussian primitives in real time. When the current frame rate is lower than the target frame rate threshold, it increases the clipping threshold of the 3D Gaussian primitive opacity. It also downgrades octree tiles in non-critical view areas based on the camera's current pose, preloads mid-level tiles ahead of the path based on the carrier's motion vector, and dynamically releases high-frequency texture blocks far from the view area. The target frame rate threshold is a known set value. By increasing the clipping threshold of the 3D Gaussian primitive opacity, it filters out small 3D Gaussian primitives with minimal visual contribution in real time, significantly reducing the number of primitives entering the rasterization rendering operator. This significantly reduces the floating-point operation load on the image processor and improves the rendering frame rate. Combined with the downgrading of octree tiles, preloading of mid-level tiles, and dynamic release of high-frequency texture blocks, it further reduces the pressure on hardware computing power during rendering and visualization, enabling smooth interaction and reconstruction of large-scale underwater scenes on hardware devices.

[0110] Figure 2 The diagram shows a structural block diagram of some embodiments of the underwater scene 3D reconstruction system based on sonar and optical image fusion provided by the present invention. Specifically, it shows a structural block diagram of some embodiments of underwater scene 3D reconstruction using 3D Gaussian splashing.

[0111] like Figure 2 As shown, the system of this embodiment includes structural units, the functions of the structural units, and the relationships between them, as detailed below:

[0112] The system includes:

[0113] The time synchronization module 21 is used to synchronize the time of the camera and forward-looking sonar mounted on the carrier.

[0114] The spatial calibration module 22 is used to perform spatial calibration on the camera and forward-looking sonar mounted on the carrier.

[0115] The coordinate system transformation module 23 is used to unify camera observation and sonar observation to the world coordinate system, with the carrier coordinate system as the pivot and the carrier's pose matrix in the world coordinate system as the world coordinate system; at the same time, the carrier's pose matrix in the world coordinate system is used as a differentiable parameter.

[0116] The sparse point cloud acquisition module 24 is used to acquire sparse point clouds of the scene.

[0117] The 3D Gaussian primitive initialization module 25 is used to initialize the 3D Gaussian primitive using the sparse point cloud obtained by the sparse point cloud acquisition module 24; wherein the 3D Gaussian primitive is embedded with an acoustic reflectivity attribute.

[0118] The 3D Gaussian primitive optimization module 26 is used to optimize the 3D Gaussian primitives initialized by the 3D Gaussian primitive initialization module 25.

[0119] The reconstruction rendering result output module 27 is used to output the three-dimensional reconstruction rendering result of the underwater scene based on the three-dimensional Gaussian primitives optimized by the three-dimensional Gaussian primitive optimization module 26.

[0120] The system with the above structure runs the corresponding software program, performs the corresponding functions, and follows the instructions. Figure 1 The underwater scene 3D reconstruction method and other embodiments perform underwater scene 3D reconstruction to achieve the same result as... Figure 1 The corresponding technical effects of the embodiments and other embodiments.

[0121] Other embodiments of the present invention also provide a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements... Figure 1 The embodiments and other embodiments describe methods for three-dimensional reconstruction of underwater scenes, and achieve the technical effects of the corresponding embodiments.

[0122] The aforementioned computer storage media can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The computer storage media can be any available storage medium accessible to general-purpose or special-purpose computers.

[0123] In some embodiments, a computer storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in application-specific integrated circuits (ASICs). Of course, the processor and storage medium can also exist as discrete components in the device.

[0124] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0125] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions claimed by the present invention.

Claims

1. A method for three-dimensional reconstruction of underwater scenes based on sonar and optical image fusion, characterized in that, The method includes: Time synchronization and spatial calibration of the camera and forward-looking sonar mounted on the carrier; Using the carrier coordinate system as the pivot, and based on the carrier's pose matrix in the world coordinate system, camera observations and sonar observations are unified to the world coordinate system; at the same time, the carrier's pose matrix in the world coordinate system is used as a differentiable parameter. A sparse point cloud of the scene is obtained, and a three-dimensional Gaussian primitive is initialized using the sparse point cloud; the three-dimensional Gaussian primitive is embedded with an acoustic reflectivity attribute. Optimize the three-dimensional Gaussian primitives, and output the three-dimensional reconstruction rendering result of the underwater scene based on the optimized three-dimensional Gaussian primitives; Optimizing the three-dimensional Gaussian elements includes: A physical law rendering branch for sonar imaging is constructed, acoustic loss is calculated, and radial centripetal force is generated by fitting the sonar echo intensity. The center of the three-dimensional Gaussian element is anchored on a sampling arc with the sonar position as the center and the actual sonar distance as the radius. The Euclidean distance from the center of the 3D Gaussian primitive to the sampling arc is calculated to obtain the arc constraint loss. The arc constraint loss is used to restrict the center of the 3D Gaussian primitive to slide along the sampling arc. At the same time, the photometric loss of the image obtained by the camera is used to generate a photometric loss gradient as a tangential driving force to drive the 3D Gaussian primitive to slide along the sampling arc, so that the 3D Gaussian primitive converges to the actual 3D coordinates of the object.

2. The underwater scene 3D reconstruction method based on sonar and optical image fusion according to claim 1, characterized in that, Optimizing the three-dimensional Gaussian elements also includes: A joint loss function is constructed, comprising the photometric loss, the acoustic loss, the arc constraint loss, and the regularization loss, to jointly optimize the properties of the three-dimensional Gaussian elements and the pose matrix of the carrier in the world coordinate system.

3. The underwater scene 3D reconstruction method based on sonar and optical image fusion according to claim 2, characterized in that, Optimizing the three-dimensional Gaussian elements also includes: The system calculates in real time the standard deviation of chromaticity, mean saturation, and contrast of the current frame image obtained by the camera to obtain the current value of the quality evaluation index of the color image. When the current value of the quality evaluation index is less than the index threshold and the contrast of the image is less than the contrast threshold, the image enters the optical degradation region, increases the weight of the acoustic loss and the weight of the arc constraint loss, and decreases the weight of the photometric loss; when the current value of the quality evaluation index is not less than the index threshold, the weight of the photometric loss is increased.

4. The underwater scene 3D reconstruction method based on sonar and optical image fusion according to claim 1, characterized in that, The rendering results of underwater scene 3D reconstruction based on optimized 3D Gaussian primitives include: A spatial octree index structure is established for the set of three-dimensional Gaussian elements, and each node stores a subset of the three-dimensional Gaussian elements in the local space corresponding to the node. For non-leaf nodes, feature coarsening is performed based on the opacity and scaling of the three-dimensional Gaussian primitives to generate a multi-level hierarchical structure.

5. The underwater scene 3D reconstruction method based on sonar and optical image fusion according to claim 4, characterized in that, The rendering results based on the optimized 3D Gaussian primitive output for 3D reconstruction of underwater scenes also include: The system monitors the current rendering frame rate of the 3D Gaussian primitives in real time. When the current rendering frame rate is less than the target frame rate threshold, it increases the clipping threshold of the opacity of the 3D Gaussian primitives. It also degrades the octree tiles in non-critical view areas according to the current set pose of the camera, and preloads the medium-level tiles in front of the path and dynamically releases the high-frequency texture blocks far away from the view area behind the carrier's motion vector.

6. The underwater scene 3D reconstruction method based on sonar and optical image fusion according to any one of claims 1 to 5, characterized in that, Obtaining a sparse point cloud of the scene, and initializing 3D Gaussian primitives using the sparse point cloud, includes: Pixels whose energy intensity exceeds the intensity threshold in the sonar echo are obtained, and the sparse point cloud of the scene is generated by combining the pose matrix of the carrier in the world coordinate system and the static transformation matrix of the sonar coordinate system relative to the carrier coordinate system. The sparse point cloud of the scene is used as the initial position of the three-dimensional Gaussian primitive.

7. The method for three-dimensional reconstruction of underwater scenes based on sonar and optical image fusion according to any one of claims 1 to 5, characterized in that, The camera and forward-looking sonar mounted on the carrier were subjected to the following layered spatial calibration: Obtain the camera's intrinsic parameters; Obtain a quasi-pinhole model with underwater refraction correction; Obtain the static transformation matrix of the camera coordinate system relative to the carrier coordinate system, the static transformation matrix of the sonar coordinate system relative to the carrier coordinate system, and the relative transformation matrix between the camera coordinate system and the sonar coordinate system.

8. The method for three-dimensional reconstruction of underwater scenes based on sonar and optical image fusion according to any one of claims 1 to 5, characterized in that, Time synchronization between the camera and forward-looking sonar mounted on the carrier includes: The main control chip sends a synchronization pulse signal to achieve hard-triggered synchronization between camera exposure time and sonar pulse time.

9. A three-dimensional reconstruction system for underwater scenes based on sonar and optical image fusion, characterized in that, The system includes: The time synchronization module is used to synchronize the time between the camera and the forward-looking sonar mounted on the carrier. The spatial calibration module is used to perform spatial calibration on the camera and forward-looking sonar mounted on the carrier. The coordinate system transformation module is used to unify camera observations and sonar observations to the world coordinate system, using the carrier coordinate system as the pivot and the carrier's pose matrix in the world coordinate system; at the same time, the carrier's pose matrix in the world coordinate system is used as a differentiable parameter. The sparse point cloud acquisition module is used to obtain sparse point clouds of the scene; A 3D Gaussian primitive initialization module is used to initialize 3D Gaussian primitives using the sparse point cloud; the 3D Gaussian primitives are embedded with acoustic reflectivity properties. A three-dimensional Gaussian element optimization module is used to optimize the three-dimensional Gaussian element; The reconstruction rendering result output module is used to output the 3D reconstruction rendering result of the underwater scene based on the optimized 3D Gaussian primitives. The 3D Gaussian element optimization module optimizes the 3D Gaussian element, including: A physical law rendering branch for sonar imaging is constructed, acoustic loss is calculated, and radial centripetal force is generated by fitting the sonar echo intensity. The center of the three-dimensional Gaussian element is anchored on a sampling arc with the sonar position as the center and the actual sonar distance as the radius. The Euclidean distance from the center of the 3D Gaussian primitive to the sampling arc is calculated to obtain the arc constraint loss. The arc constraint loss is used to restrict the center of the 3D Gaussian primitive to slide along the sampling arc. At the same time, the photometric loss of the image obtained by the camera is used to generate a photometric loss gradient as a tangential driving force to drive the 3D Gaussian primitive to slide along the sampling arc, so that the 3D Gaussian primitive converges to the actual 3D coordinates of the object.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the underwater scene three-dimensional reconstruction method based on sonar and optical image fusion as described in any one of claims 1 to 8.

Images