A method and apparatus for adaptive 3D reconstruction and visualization of sonar images based on Gaussian splashing.

By using the Gaussian Splatting algorithm and an improved illumination model, high-quality 3D reconstruction of sonar images from 2D was achieved, solving the problems of low resolution and high noise in traditional sonar images and improving the accuracy of underwater target identification and analysis.

CN120339546BActive Publication Date: 2026-11-17ZHEJIANG UNIV OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510403203.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-04-01
Publication Date
2026-11-17
Estimated Expiration
2045-04-01

AI Technical Summary

Technical Problem

Traditional side-scan sonar images are affected by environmental noise, have low resolution, lack three-dimensional information, and are difficult to accurately identify underwater targets.

Method used

The Gaussian Splatting algorithm is used for 3D reconstruction of sonar images. High-quality 3D models are generated through Gaussian splashing technology, and the illumination model is improved by combining spherical harmonic functions to optimize image quality and detail representation.

Benefits of technology

It significantly improves the quality of 3D reconstruction and target recognition accuracy of sonar images, enables better analysis of the shape and distribution of underwater targets, reduces computational resource consumption, and is suitable for underwater detection and facility inspection in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120339546B_ABST
    Figure CN120339546B_ABST
Patent Text Reader

Abstract

A method and apparatus for adaptive 3D reconstruction and visualization of sonar images based on Gaussian splashing, comprising the following steps: S1, building a high-performance training server, accessing the network via 10 Gigabit Ethernet to receive raw sonar image data; S2, porting the sonar image denoising filter components required for the Gaussian splashing algorithm to the software stack to ensure efficient computation and optimization; S3, performing image preprocessing, including denoising and background segmentation, removing irrelevant background information, such as water column areas, to provide accurate data input for subsequent reconstruction; S4, using SfM technology to recover preliminary point cloud data; S5, training the point cloud data using the Gaussian splashing algorithm, combining the characteristics of sonar images, and using a coloring function based on an anisotropic sound source model and a micro-surface bidirectional reflection distribution function to generate a 3D reconstruction model; S6, visualizing the results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of sonar image processing, and in particular to a method and apparatus for adaptive 3D reconstruction and visualization of sonar images using Gaussian splatting technology, which can be widely applied to underwater detection, target recognition and environmental modeling. Background Technology

[0002] Side-scan sonar technology is widely used in underwater exploration, seabed topography mapping, and shipwreck retrieval. By emitting sound waves and receiving reflected signals, sonar systems can generate two-dimensional images of underwater targets. However, traditional side-scan sonar images are often affected by environmental noise, resulting in low image quality, limited resolution, and the presentation of only two-dimensional planar information. These limitations make it difficult to accurately identify and analyze targets, especially in complex underwater environments where images may contain significant noise and distortion, further reducing imaging effectiveness.

[0003] Existing sonar image processing methods primarily focus on optimizing and analyzing two-dimensional data, but lack the ability to reconstruct depth information from images, thus failing to fully reflect the spatial structure of underwater objects. Therefore, improving the quality of sonar images, restoring missing three-dimensional information, and enhancing the accuracy of target identification have become current technological challenges.

[0004] Gaussian Splatting is a technique based on Gaussian kernel function point clouds. By fitting image data with a locally continuous and densely sampled spatial Gaussian kernel function, it can achieve efficient and reasonable interpolation and reconstruction. Gaussian Splatting has a natural advantage in processing sparse images or complex materials. This invention proposes a method for 3D reconstruction and visualization of sonar images based on the Gaussian Splatting algorithm. This method can transform traditional 2D sonar images into high-quality 3D reconstructed images, supplementing missing spatial information with relatively low computational cost, and enabling real-time visualization of the 3D results. This method can process low-resolution sonar data, generate high-quality 3D models, and can be used in applications such as target recognition, surveying, and environmental monitoring. Summary of the Invention

[0005] High-noise side-scan sonar images suffer from poor image quality due to low resolution and high noise levels, and are limited to a two-dimensional plane, making accurate target identification difficult. This invention overcomes these shortcomings by providing a method and apparatus for adaptive 3D reconstruction and visualization of sonar images based on Gaussian splatting. The Gaussian splatting algorithm is used to reconstruct the 3D structure of sonar images, supplementing lost 3D information and effectively improving image quality. The reconstructed images significantly improve the accuracy of subsequent analysis and enable better target identification.

[0006] The first aspect of this invention relates to an adaptive 3D reconstruction and visualization method for sonar images based on Gaussian splashing, comprising the following steps:

[0007] S1. Set up a training server. The server connects to the network via 10 Gigabit Ethernet to receive raw data.

[0008] S2. Port the CUDA components in the algorithm module required for Gaussian splashing to the software stack;

[0009] S3. Image preprocessing: A series of scanned sonar images are processed and denoised; then, the target region is extracted through background segmentation, and irrelevant background information is removed.

[0010] S4. Based on Structure from Motion (SfM) technology, point cloud data is generated by extracting feature points, perspective transformation is optimized, and covariance matrix, spherical harmonic coefficients and opacity are assigned.

[0011] S5. Train Gaussian point clouds. Based on the generated initial point clouds, perform differentiable rendering using a rendering model. Apply the Gaussian splashing algorithm to generate Gaussian point clouds and improve the color and lighting shading model according to the characteristics of sonar images.

[0012] S6. Results visualization: The optimized Gaussian point cloud is rendered and visualized in real time to generate a high-quality 3D model.

[0013] The CUDA components mentioned in step S2 include: a sonar image denoising filter component, a SIFT feature extraction component in the Structure for Motion Restoration (SfM) technique, a Random Sample Consensus (RANSAC) fitting component, a differentiable Gaussian rasterization component, and a spherical harmonic function calculation component.

[0014] The differentiable Gaussian rasterization component includes the following construction steps:

[0015] First, all Gaussian points are projected onto the screen space according to the camera parameters. Then, radix sort is used to sort all Gaussian points by depth. Next, the screen space bounding box of each Gaussian point is calculated by approximating it with orthogonal projection and using a 2.35 standard deviation cutoff radius.

[0016] The screen is divided into 16*16 blocks. Each Gaussian point is added to the block it covers according to the screen space bounding box, generating a list of Gaussian points inside each block. The GPU part and algorithm are used to perform a one-time continuous video memory allocation on the list of Gaussian points of all blocks to maintain the depth order of the Gaussian points.

[0017] For each small screen block, a GPU thread block is allocated, and each thread processes 1 pixel. All Gaussian points in the list of that small block are traversed in depth order from far to near. The pixel color is calculated according to the subsequent formula, and forward rendering is performed.

[0018] I0(d)=0 (1)

[0019] I j (d)=(1-A j G j (J -1 (d)))I j-1 (d)+A j G j (J -1 (d))C(p j (2)

[0020] After the pixel color gradient calculation is completed, a GPU thread block is created for each small screen block, and each thread processes one pixel. The Gaussian points in the list of all small blocks are traversed in depth order from near to far. The pixel color gradient is converted into the Gaussian point color C using the Jacobi matrix of formula (2). j gradient and Gaussian function G j The gradient of (d) is accumulated and temporarily stored in the shared memory within the thread block; after accumulating the gradient of every 16 Gaussian points, it is accumulated into the final gradient tensor in the global video memory using atomic operations.

[0021] Step S5 includes:

[0022] S51. Calculate the Gaussian function G. j The core idea of ​​Gaussian Splatting is to represent each point in a point cloud using a 3D Gaussian distribution, distributing energy in space through its probability density function (PDF), thereby achieving smooth and continuous rendering. In the 3D Gaussian distribution model, each point is represented by a 3D Gaussian function.

[0023]

[0024] p is the center of Gauss.

[0025] Σ is the covariance matrix, which controls the shape and size of the Gaussian distribution.

[0026] S52. Improve the illumination model in traditional Gaussian splashing to suit the characteristics of sonar images, and use it to calculate the echo intensity C(p) of each Gaussian. j );

[0027] S53. Render and reconstruct image I based on Gaussian parameters;

[0028] S54. Train the Gaussian parameters.

[0029] Among them, the lighting model described in step S52 uses the function C(p) j The following is the specific content:

[0030] Based on the formula for spherical harmonic functions:

[0031]

[0032] in It is a Legendre polynomial, where l and m are order and degree scalars, respectively;

[0033] Within the framework of traditional Gaussian splashing, the color of each Gaussian point is defined by the spherical harmonic function series in its line of sight direction;

[0034]

[0035] in These are the coefficients of the spherical harmonic function, t i p is the viewpoint translation vector of the i-th image. j It is the location of the j-th Gaussian center; p j -t i This is the result of the viewpoint transformation of the Gaussian center position, that is, the direction of the line of sight from this viewpoint;

[0036] The colorized model is refined, the sound intensity and reflection intensity are calculated, and multiplied to obtain the final reflected sound intensity. The originally coupled sound source is discussed separately from the bidirectional reflection distribution function (BRDF):

[0037] First, let's consider the sound source. Sonar sound sources often exhibit anisotropic characteristics, being stronger directly in front of the line of sight and weaker around the edges, such as... Figure 2 As shown. This effect is fitted using the spherical harmonic function of the sonar local coordinate system:

[0038]

[0039] Among them, S(p j ) is p j The sound intensity at the point, These are the coefficients of the corresponding spherical harmonic function of the sound source.

[0040] Secondly, there is the bidirectional reflection distribution function at the Gaussian point. A microsurface reflection model is used, such as... Figure 3As shown, the surface of an object is considered to be composed of countless micro-surfaces with different orientations, all of which are specular reflective surfaces. Given the incident and reflection directions, the reflection intensity is proportional to the probability density of the corresponding specular reflection produced by the micro-surface. The orientation of such a micro-surface is called the half-angle direction. The final reflection intensity also includes a geometric term expressing the mutual occlusion of the micro-surfaces, and a Fresnel term calculating the reflection and refraction ratio at the interface of the two media based on different propagation speeds.

[0041] Since the sound source and viewpoint are usually close, the angle between the incident and reflected directions of the sound wave is extremely small. Therefore, it can be approximated that the incident and reflected directions are collinear and opposite in direction. This allows us to use the same vector with different signs to represent the incident, reflected, and half-angle directions, thus approximating the probability density, geometric, and Fresnel terms in the microsurface reflection model as functions of this vector. A unified set of spherical harmonic function series is then used for fitting.

[0042] Although the probability density of a microsurface is generally defined in the local coordinate system of the reflection point, it can be simplified by considering the closure of the series of definite-order spherical harmonic functions under rotational transformations. The reflection distribution function can be simply fitted using the spherical harmonic function of the global coordinate system.

[0043]

[0044] Multiplying the sound intensity (Equation 6) by the reflection intensity (Equation 7) yields:

[0045]

[0046] Where C(p) j ) represents the final reflected sound intensity at Gaussian point j.

[0047] The specific details of the model used to render the reconstructed image I in step S53 are as follows:

[0048] Projecting the center points of all Gaussians onto the image space and sorting them by depth from farthest to nearest, we discuss the intensity I(d) of a pixel d without loss of generality; I(d) is composed of the echo intensity C of each Gaussian. j It is formed by mixing sequentially; initially, all pixel intensities are assumed to be 0.

[0049] I0(d)=0 (9)

[0050] Where the subscript j corresponds to the order from farthest to nearest; the intensity after mixing with the j-th Gaussian is:

[0051] I j (d)=(1-A j G j (J-1 (d)))I j-1 (d)+A j G j (J -1 (d))C(p j (10)

[0052] G j (d) represents the contribution of the j-th Gaussian point to pixel d, and the final pixel echo intensity is the mixture result of all Gaussian points I. n (d); J -1 (d) is the inverse function of perspective transformation, transforming pixel d to Gaussian G. j The depth plane containing the center point.

[0053] Specifically, step S54 includes:

[0054] S541, Using camera parameters R i and t i Perform differentiable rasterization to plot the reconstructed image I;

[0055] S542. Compare the reconstructed image with the original captured image and calculate the loss function L; minimize L using the gradient descent algorithm, and backpropagate the obtained gradient back to the Gaussian center point p through the steps described in S53, S52, and S51. j Covariance matrix Σ j spherical harmonic coefficients and Opacity A j The loss L is a first-order loss, a weighted average of the loss function D-SSIM, where the weights are parameters set by the user in advance.

[0056] S543. During the optimization process, the density of the Gaussian gas is dynamically adjusted; for p j A Gaussian gradient with magnitude > τ is replicated to increase density; opacity A is removed. j <α Gaussian; where τ and α are constants specified by the user in advance;

[0057] S544. Repeat the above steps until the specified number of times is reached.

[0058] Step S6 includes:

[0059] S61. Save the training results as a PLY format point cloud file on the server, and open the HTTPS port on the server to provide a web-based 3D visualization application.

[0060] S62. Optimize and port the open-source Splatapult framework to the web page.

[0061] S63. Develop a visual user interface component based on WebAssembly.

[0062] Step S62 includes:

[0063] S621, Construct a WebGPU deep sorting component;

[0064] S622, Construct WebGL drawing components.

[0065] The WebAssembly-based visual user interface component mentioned in step S63 includes:

[0066] Object mode: Supports Xbox controller operation. The right controller rotates the object, the left controller moves the viewpoint up, down, left, and right, and the trigger button zooms in and out by adjusting the field of view (FOV). It also provides an opacity filter function. You can use the left and right shoulder buttons to adjust an opacity threshold. Only Gaussian points with an opacity higher than the threshold will be displayed on the screen.

[0067] Scene mode: Supports Xbox controller operation, with the right controller rotating the view, the left controller moving the viewpoint forward, backward, left, and right, and the trigger moving the viewpoint up and down; provides an opacity filter function, using the left and right shoulder buttons to adjust an opacity threshold, only Gaussian points with an opacity higher than the threshold will be displayed in the interface.

[0068] A second aspect of the present invention relates to an adaptive 3D reconstruction and visualization apparatus for sonar images based on Gaussian splashing, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the adaptive 3D reconstruction and visualization method for sonar images based on Gaussian splashing of the present invention.

[0069] The innovation of this invention is:

[0070] 1. Gaussian splashing was applied to sonar images.

[0071] 2. Utilizing the characteristics of sonar images, the spherical harmonic function series C used for coloring is... j The parameters are shifted from global space to the local space of the camera to adapt to the dynamic sound sources commonly found in sonar images, achieving better reconstruction results.

[0072] The present invention has the following beneficial effects:

[0073] 1. Supplementing 3D Information. Using a Gaussian splashing algorithm, low-resolution, noisy 2D sonar images are converted into high-quality 3D models. This method accurately captures the geometric structure of the target, compensating for the lack of depth information provided by 2D images, resulting in more complete and accurate reconstructions. For example, in underwater facility inspection, it allows for a more intuitive analysis of the target's shape and distribution, effectively improving the accuracy of underwater operations.

[0074] 2. Highly Efficient Reconstruction Capability. This technology offers an order of magnitude improvement in efficiency compared to traditional 3D reconstruction methods (such as NeRF). This is thanks to the Gaussian splashing algorithm's optimized point cloud representation, which can efficiently process large-scale data and reduce computational resource consumption. Even when processing complex sonar data, it can quickly generate 3D results. This efficiency makes this method particularly suitable for engineering monitoring and modeling applications of large underwater facilities, significantly reducing workload and time.

[0075] 3. High-Quality Reconstruction Results. The reconstructed model not only boasts highly reliable visual effects but also shows significant improvements in detail, particularly in edge sharpness and surface continuity. Combined with an improved lighting model, the reconstruction results more realistically reflect the physical characteristics of the target, making it suitable for sophisticated tasks such as dam crack detection and equipment condition assessment of nuclear power plant cooling pools. These high-quality reconstruction results provide a reliable basis for further data analysis, risk assessment, and scientific research. Attached Figure Description

[0076] Figure 1 This is a flowchart of the adaptive 3D reconstruction based on Gaussian splashing of the present invention.

[0077] Figure 2 This is a schematic diagram of the anisotropic point sound source of the present invention.

[0078] Figure 3 This is the micro-surface reflection model of the present invention and the approximate simulation diagram in the same direction.

[0079] Figure 4 This is a flowchart of the deployment process of the Gaussian splashing algorithm 3D reconstruction system of the present invention. Detailed Implementation

[0080] To better understand the above technical solution, the following will elaborate on the technical solution in conjunction with specific implementation methods.

[0081] Example 1

[0082] like Figure 1 This embodiment provides an adaptive 3D reconstruction and visualization method for sonar images based on Gaussian splashing. In the following text, lowercase letters represent 3D vectors (e.g., p), uppercase letters represent matrices or functions (e.g., V), and bold lowercase letters represent images (e.g., ...). The method of the present invention includes the following steps:

[0083] S1. Set up the training server system.

[0084] First, a training server system was set up, with each server equipped with four domestically produced Muxi C-series GPGPU processors. Connected to the network via 10 Gigabit Ethernet, it receives raw sonar image data acquired from underwater detection equipment. The servers utilize efficient data transmission protocols and hardware resources to process large-scale image data, ensuring rapid data transmission and processing.

[0085] S2. Port the CUDA components to the MXMACA software stack.

[0086] To improve the algorithm's execution efficiency, the CUDA components required for the Gaussian splash algorithm were ported to MXMACA software stack independently developed by Muxi. This software stack supports parallel computing, can efficiently handle the computational tasks of each point cloud, and optimizes the data processing process, making the training process more efficient and scalable.

[0087] The process for steps S1 and S2 is as follows: Figure 4 As shown.

[0088] The CUDA components mentioned in step S2 include: a sonar image denoising filter component, a SIFT feature extraction component in the Structure for Motion Restoration (SfM) technique, a Random Sample Consensus (RANSAC) fitting component, a differentiable Gaussian rasterization component, and a spherical harmonic function calculation component.

[0089] Differentiable Gaussian rasterization components include the following construction steps:

[0090] First, all Gaussian points are projected onto the screen space according to the camera parameters. Then, radix sort is used to sort all Gaussian points by depth. Next, the screen space bounding box of each Gaussian point is calculated by approximating it with orthogonal projection and using a 2.35 standard deviation cutoff radius.

[0091] The screen is divided into 16*16 blocks. Each Gaussian point is added to the block it covers according to the screen space bounding box, generating a list of Gaussian points inside each block. The GPU part and algorithm are used to perform a one-time continuous video memory allocation on the list of Gaussian points of all blocks to maintain the depth order of the Gaussian points.

[0092] For each small screen block, a GPU thread block is allocated, and each thread processes 1 pixel. All Gaussian points in the list of that small block are traversed in depth order from far to near. The pixel color is calculated according to the subsequent formula, and forward rendering is performed.

[0093] I0(d)=0 (1)

[0094] I j (d)=(1-A j G j (J -1 (d)))I j-1 (d)+A j G j (J -1 (d))C j (2)

[0095] After the pixel color gradient calculation is completed, a GPU thread block is created for each small screen block, and each thread processes one pixel. The Gaussian points in the list of all small blocks are traversed in depth order from near to far. The pixel color gradient is converted into the Gaussian point color C using the Jacobi matrix of formula (2). j gradient and Gaussian function G j The gradient of (d) is accumulated and temporarily stored in the shared memory within the thread block; after accumulating the gradient of every 16 Gaussian points, it is accumulated into the final gradient tensor in the global video memory using atomic operations.

[0096] S3. Image preprocessing.

[0097] S31. Collect and organize a series of sonar images. i The image is then denoised to reduce the impact of environmental noise on image quality.

[0098] S32. Extract the target region through background segmentation (mask generation), remove irrelevant background information, and obtain the mask image. This lays the foundation for subsequent reconstruction.

[0099] S4. Run the Structure from Motion (SfM) software.

[0100] S41. Run standard Structure of Motion (SfM) software to extract and match two-dimensional feature points from the sonar image, generating initial point cloud data p. j The rotation matrix R corresponding to the shooting position of each image i Translation vector t i , and the perspective transformation J(p) of the acoustic camera.

[0101] S42. Process the results of the motion reconstruction structure. For each point P in the point cloud... j Generate random covariance matrix Σ j This transforms each point into a Gaussian, and then randomly generates the coefficients of the spherical harmonic function used subsequently. and Opacity A j(Opacity).

[0102] S5. Training Gaussian Point Clouds. Based on the generated initial point cloud, a Gaussian splashing algorithm is applied to generate Gaussian point clouds, and the shading model (illumination model) is improved to suit the characteristics of sonar images, thereby enhancing the realism and detail preservation of the reconstruction results.

[0103] S51. Calculate the Gaussian function G. j

[0104]

[0105] S52. Calculate the echo intensity C(p) of each Gaussian. j ):

[0106] The lighting model described in step S52 uses the function C(p) j The following is the specific content:

[0107] Based on the formula for spherical harmonic functions:

[0108]

[0109] in It is a Legendre polynomial, where l and m are order and degree scalars, respectively;

[0110] Within the framework of traditional Gaussian splashing, the color of each Gaussian point is defined by the spherical harmonic function series in its line of sight direction;

[0111]

[0112] in These are the coefficients of the spherical harmonic function, t i p is the viewpoint translation vector of the i-th image. j It is the location of the j-th Gaussian center; p j -t i This is the result of the viewpoint transformation of the Gaussian center position, that is, the direction of the line of sight from this viewpoint;

[0113] The shading model is refined, the sound intensity and reflection intensity are calculated, and then multiplied. The final reflected sound intensity is:

[0114]

[0115] Where C(p) j ) represents the final reflected sound intensity at Gaussian point j.

[0116] in, These are the coefficients of the spherical harmonic function of the reflection function. These are the coefficients of the spherical harmonic function of the sound source, t i R is the viewpoint translation vector of the i-th image.i p is the viewpoint rotation matrix for the i-th image. j It is the j-th Gaussian center position. j -t i This is the result of the viewpoint transformation of the Gaussian center position, that is, the direction of the line of sight from that viewpoint.

[0117] S53. Based on the various parameters of Gaussian, reconstruct the image I. The reconstruction process uses a differentiable rasterization algorithm. This rasterization algorithm uses the camera parameters R corresponding to the original image i. i and t i And the perspective transformation J(p) of the acoustic camera. The pixel intensity formula of the rasterization algorithm is:

[0118] I0(d)=0 (6)

[0119] I j (D)=(1-A j G j (J -1 (d)))I j-1 (d)+A j G j (J -1 (d))C j (7)

[0120] This formula is an iterative formula that projects the center points of all Gaussians onto the image space and sorts them from farthest to nearest depth, where I(d) is the intensity of each pixel d, and is determined by the echo intensity C of each Gaussian. j The elements are mixed sequentially. Initially, according to formula (1), all pixel intensities are assumed to be 0. Then, each Gaussian j is sequentially mixed in according to formula (2). Where G... j From (9), we can deduce that C j This is derived from (10).

[0121] S54. Train the Gaussian parameters.

[0122] S541. Calculate the loss function L based on the reconstructed image I and the corresponding original image. The loss L is a first-order loss, a weighted average of the D-SSIM loss function, where the weights λ are parameters preset by the user.

[0123] L=(1-λ)L1+λL D-SSIM (9)

[0124] S542. Call the gradient descent solver to minimize L, and backpropagate the resulting gradient to the Gaussian center point p. j Covariance matrix Σ j spherical harmonic coefficients and Opacity A j This allows for adjustments to the values ​​of these parameters, thereby improving the reconstruction results.

[0125] S543. During the optimization process of S35, the density of Gaussians is dynamically adjusted. For p j If the gradient magnitude is greater than τ, a Gaussian is considered to have insufficient density in the vicinity and is replicated to increase the density. For opacity A... j Gaussians with a value less than α are considered to have made too little contribution and are therefore removed. Here, τ and α are constants specified by the user beforehand.

[0126] S544. Repeat the above steps until the specified number of times is reached.

[0127] S6, Results Visualization

[0128] S61. Save the results, including the Gaussian center point p. j Covariance matrix Σ j spherical harmonic coefficients and Opacity A j Save as a local file for visualization.

[0129] S62. Repeat steps S51 to S53 and visualize the results.

[0130] S621. In the rendering process corresponding to S53, the depth sorting originally implemented using MXMACA was replaced with the WebGPU version.

[0131] S622. In the rendering process corresponding to S51, the differentiable MXMACA rasterization algorithm is replaced with the non-differentiable WebGL rasterization algorithm, thereby reducing computing power costs to meet visualization requirements.

[0132] S63. Develop a visual interface based on the VTK framework and compile it into WebAssembly for integration into a webpage.

[0133] The interface provides an object mode. Xbox controller operation is supported: the right controller rotates objects, the left controller moves the viewpoint up, down, left, and right, and the trigger buttons zoom in and out by adjusting the field of view (FOV). An opacity filter is provided; an opacity threshold can be adjusted using the left and right shoulder buttons, and only Gaussian points with an opacity higher than that threshold will be displayed in the interface.

[0134] Scene modes are provided in the interface. Xbox controller operation is supported: the right controller rotates the view, the left controller moves the viewpoint forward, backward, left, and right, and the triggers move the viewpoint up and down. An opacity filter function is provided; an opacity threshold can be adjusted using the left and right shoulder buttons, and only Gaussian points with an opacity higher than the threshold will be displayed in the interface.

[0135] Example 2

[0136] This embodiment relates to an adaptive 3D reconstruction and visualization device for sonar images based on Gaussian splatting, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the adaptive 3D reconstruction and visualization method for sonar images based on Gaussian splatting according to Embodiment 1.

[0137] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0141] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0142] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0144] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0146] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0147] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. An adaptive 3D reconstruction and visualization method for sonar images based on Gaussian splashing, comprising the following steps: S1. Set up a training server. The server connects to the network via 10 Gigabit Ethernet to receive raw data. S2. Port the CUDA components in the algorithm module required by Gaussian splashing to the software stack; S3. Image preprocessing: A series of scanned sonar images are processed and denoised; then, the target region is extracted through background segmentation, and irrelevant background information is removed. S4. Based on the motion restoration structure technology SfM, point cloud data is generated by extracting feature points, the perspective transformation is optimized, and the covariance matrix, spherical harmonic coefficients and opacity are assigned. S5. Train Gaussian point clouds. Based on the generated initial point clouds, perform differentiable rendering using a rendering model. Apply the Gaussian splashing algorithm to generate Gaussian point clouds and improve the color and lighting shading model according to the characteristics of sonar images. S6. Results visualization: The optimized Gaussian point cloud is rendered and visualized in real time to generate a high-quality 3D model.

2. The adaptive 3D reconstruction and visualization system for sonar images based on Gaussian speckle of claim 1, wherein: The CUDA components mentioned in step S2 include: a sonar image denoising filter component, a SIFT feature extraction component in the Structure for Motion Restoration (SfM) technique, a Random Sample Consensus (RANSAC) fitting component, a differentiable Gaussian rasterization component, and a spherical harmonic function calculation component.

3. The adaptive 3D reconstruction and visualization method for sonar images based on Gaussian splashing as described in claim 2, wherein the differentiable Gaussian rasterization component includes the following construction steps: First, all Gaussian points are projected onto the screen space according to the camera parameters. Then, radix sort is used to sort all Gaussian points by depth. Next, the screen space bounding box of each Gaussian point is calculated by approximating it with orthogonal projection and using a 2.35 standard deviation cutoff radius. The screen is divided into 16*16 blocks. Each Gaussian point is added to the block it covers according to the screen space bounding box, generating a list of Gaussian points inside each block. The GPU part and algorithm are used to perform a one-time continuous video memory allocation on the list of Gaussian points of all blocks to maintain the depth order of the Gaussian points. For each small screen block, a GPU thread block is allocated, and each thread processes 1 pixel. All Gaussian points in the list of that small block are traversed in depth order from far to near. The pixel color is calculated according to the subsequent formula, and forward rendering is performed. in, represents opacity, is the inverse function of the perspective transformation, represents the reflected sound intensity of Gaussian point j; After the pixel color gradient is calculated, a GPU thread block is opened again for each screen block, and each thread processes 1 pixel. All the Gauss points in the list of the block are traversed in depth order from near to far, and the Jacobi matrix of formula (2) is used to convert the pixel color gradient into the gradient of the Gauss point color of the Gauss function , and the gradient is accumulated in the shared memory in the thread block for temporary storage; after 16 Gauss point gradients are accumulated, the atomic operation is used to accumulate the final gradient tensor in the global memory.

4. The adaptive 3D reconstruction and visualization system for sonar images based on Gaussian splashing as described in claim 1, step S5 includes: S51, calculating a Gaussian function ; S52, the illumination model in traditional Gaussian splash is improved for sonar image characteristics, and it is used to calculate the reflected sound intensity of each Gaussian point ; S53. Render the reconstructed image I based on the Gaussian parameters; S54. Train the Gaussian parameters.

5. The adaptive 3D reconstruction and visualization system for sonar images based on Gaussian splashing as described in claim 4, wherein the illumination model described in step S52 uses a function The specific details are as follows: Based on the formula for spherical harmonic functions: wherein are associated Legendre polynomials, where and are order and degree scalars; Within the framework of traditional Gaussian splashing, the color of each Gaussian point is defined by the spherical harmonic function series in its line of sight direction; in These are the coefficients of the spherical harmonic function. It is the viewpoint translation vector of the i-th image. It is the location of the j-th Gaussian center; This is the result of the viewpoint transformation of the Gaussian center position, that is, the direction of the line of sight from this viewpoint; The shading model is refined, the sound intensity and reflection intensity are calculated, and then multiplied. The final reflected sound intensity is: in This represents the final reflected sound intensity at Gaussian point j; These are the coefficients of the spherical harmonic function of the sound source. It is the first Viewpoint rotation matrix of the image.

6. In the adaptive 3D reconstruction and visualization method for sonar images based on Gaussian splashing as described in claim 4, the specific content of the model used to render the reconstructed image I in step S53 is as follows: Project the center points of all Gaussians onto the image space and sort them by depth from farthest to nearest. Without loss of generality, discuss a pixel. intensity ; The echo intensity of each Gauss It is formed by mixing sequentially; initially, all pixel intensities are assumed to be 0. in, Subscript Corresponding to the order from farthest to nearest; a mixture of the first... The strength after Gauss is: It is the first Gaussian points to pixels The contribution of this factor results in the final pixel echo intensity being a mixture of all Gaussians. ; It is the inverse function of perspective transformation, which transforms pixels... Transform to Gaussian The depth plane containing the center point.

7. The adaptive 3D reconstruction system for sonar images based on Gaussian splashing as described in claim 4, step S54 specifically includes: S541, Using Camera Parameters and Perform differentiable rasterization to plot the reconstructed image I; S542. Compare the reconstructed image with the original captured image, and calculate the loss function L; minimize L using the gradient descent algorithm, and backpropagate the obtained gradient back to the Gaussian center point through the steps described in S53, S52, and S51. covariance matrix spherical harmonic coefficients and opacity The loss L is a first-order loss, a weighted average of the loss function D-SSIM, where the weights are parameters set by the user in advance. S543. During the optimization process, the density of Gaussians is dynamically adjusted; for gradient magnitude > The Gaussian shape is copied to increase density; opacity is removed. Gauss; among which, and These are all constants specified by the user in advance; S544. Repeat the above steps until the specified number of times is reached.

8. The adaptive 3D reconstruction and visualization method for sonar images based on Gaussian splashing as described in claim 1, characterized in that: Step S6 includes: S61. Save the training results as a PLY format point cloud file on the server, and open the HTTPS port on the server to provide a web-based 3D visualization application. S62. Optimize and port the open-source Splatapult framework to the web page; S63. Develop a visual user interface component based on WebAssembly; Step S62 includes: S621, Construct a WebGPU deep sorting component; S622, Construct WebGL drawing components.

9. The adaptive 3D reconstruction and visualization method for sonar images based on Gaussian splashing as described in claim 8, characterized in that: The WebAssembly-based visual user interface component mentioned in step S63 includes: Object mode: Supports Xbox controller operation. The right controller rotates the object, the left controller moves the viewpoint up, down, left, and right, and the trigger button zooms in and out by adjusting the field of view (FOV). It also provides an opacity filter function. You can use the left and right shoulder buttons to adjust an opacity threshold. Only Gaussian points with an opacity higher than the threshold will be displayed on the screen. Scene mode: Supports Xbox controller operation, with the right controller rotating the view, the left controller moving the viewpoint forward, backward, left, and right, and the trigger moving the viewpoint up and down; provides an opacity filter function, using the left and right shoulder buttons to adjust an opacity threshold, only Gaussian points with an opacity higher than the threshold will be displayed in the interface.

10. A device for adaptive 3D reconstruction and visualization of sonar images based on Gaussian splashing, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the adaptive 3D reconstruction and visualization method for sonar images based on Gaussian splashing as described in any one of claims 1-9.

Citation Information

Patent Citations

  • Novel view angle synthesis method based on Gaussian splash and fusing learnable basis function

    CN118505541A

  • Grid equipment three-dimensional reconstruction method based on Gaussian splashing

    CN119359955A