Sonar image adaptive three-dimensional reconstruction and visualization method and device based on Gaussian splashing

Abstract

The invention discloses a sonar image adaptive three-dimensional reconstruction and visualization method and device based on Gaussian splashing, and the method comprises the steps: S1, building a high-performance training server, accessing a network through a 10-gigabit Ethernet, and receiving original sonar image data; s2, transplanting a sonar image denoising filter assembly required by a Gaussian splash algorithm to a software stack to ensure efficient calculation and optimization; s3, image preprocessing is carried out, including denoising and background segmentation, irrelevant background information such as a water column region is removed, and accurate data input is provided for subsequent reconstruction; s4, recovering the initial point cloud data by using an SfM technology; s5, training point cloud data through a Gaussian splash algorithm, and generating a three-dimensional reconstruction model by combining sonar image characteristics and adopting a coloring function based on an anisotropic sound source model and a micro-surface bidirectional reflection distribution function; and S6, result visualization is carried out.

Description

Technical Field

[0001] The present invention relates to the field of sonar image processing, in particular to a method and device for adaptively three-dimensional reconstructing and visualizing sonar images using Gaussian Splatting technology, which can be widely applied to fields such as underwater detection, target recognition, and environmental modeling. Background Art

[0002] Side-scan sonar technology is widely used in fields such as underwater detection, seabed topography mapping, and sunken ship searching. By emitting sound waves and receiving reflected signals, the sonar system can generate two-dimensional images of underwater targets. However, traditional side-scan sonar images are often affected by environmental noise, with low image quality, limited resolution, and only presenting two-dimensional plane information. These limitations make it difficult to accurately identify and analyze targets. Especially in complex underwater environments, there may be more noise and distortion in the images, further reducing the imaging effect.

[0003] Existing sonar image processing methods mainly focus on the optimization and analysis of two-dimensional data, but lack the reconstruction of the depth information of the images and cannot fully reflect the spatial structure of underwater objects. Therefore, how to improve the quality of sonar images, restore the missing three-dimensional information, and improve the accuracy of target recognition has become a difficult point in current technology.

[0004] Gaussian Splatting is a technology based on Gaussian kernel function point clouds. By fitting image data with a spatially Gaussian kernel function that is locally continuous and densely sampled, efficient and reasonable interpolation and reconstruction can be achieved. Gaussian Splatting has natural advantages in dealing with sparse images or complex materials. The present invention proposes a method for three-dimensional reconstructing and visualizing sonar images based on the Gaussian Splatting algorithm. Through this method, traditional two-dimensional sonar images can be transformed into high-quality three-dimensional reconstructed images, supplementing the missing spatial information with a relatively low computing power cost, and enabling real-time visualization of the three-dimensional results. This method can process low-resolution sonar data, generate high-quality three-dimensional models, and can be used in application scenarios such as target recognition, mapping, and environmental monitoring. Summary of the Invention

[0005] Due to low resolution and a large amount of noise, high-noise side-scan sonar images have poor image quality and are limited to two-dimensional planes, making it difficult to accurately identify targets. The present invention aims to overcome the above-mentioned drawbacks of the prior art and provides a method and device for adaptively three-dimensional reconstructing and visualizing sonar images based on Gaussian Splatting. By using the Gaussian Splatting algorithm to three-dimensionally reconstruct sonar images, the missing three-dimensional information is supplemented, and the image quality is effectively improved. After reconstruction, the accuracy of subsequent analysis can be significantly improved, and target recognition can be better performed.

[0006] The first aspect of the present invention relates to a method for adaptive three-dimensional reconstruction and visualization of sonar images based on Gaussian splashing, comprising the following steps:

[0007] S1. Set up a training server. The server accesses the network through a 10 Gigabit Ethernet and receives raw data;

[0008] S2. Transplant the CUDA components in the algorithm modules required for Gaussian splashing onto the software stack;

[0009] S3. Image preprocessing. Organize a series of sonar images obtained and perform denoising processing on them; then extract the target area through background segmentation to eliminate irrelevant background information;

[0010] S4. Based on the Structure from Motion (SfM) technology, generate point cloud data by extracting feature points, optimize the perspective transformation, and assign covariance matrices, spherical harmonic function coefficients, and opacities;

[0011] S5. Train Gaussian point clouds. Based on the generated initial point clouds, perform differentiable rendering using a rendering model; apply the Gaussian splashing algorithm to generate Gaussian point clouds, and improve the color and lighting model (shading model) according to the characteristics of sonar images;

[0012] S6. Result visualization. Perform real-time rendering and visualization display on the optimized Gaussian point clouds to generate high-quality three-dimensional models.

[0013] Among them, the CUDA components described in step S2 include: a sonar image denoising filter component, a SIFT feature extraction component in the Structure from Motion (SfM) technology, a Random Sample Consensus (RANSAC) fitting component, a differentiable Gaussian rasterization component, and a spherical harmonic function calculation component.

[0014] Among them, the differentiable Gaussian rasterization component includes the following construction steps:

[0015] First, project all Gaussian points onto the screen space according to the camera parameters, and then sort all Gaussian points by depth using radix sort; then truncate the radius according to 2.35 times the standard deviation and approximate it by orthogonal projection to calculate the screen space bounding box of each Gaussian point;

[0016] Divide the screen into 16*16 small blocks, add each Gaussian point to the small blocks it covers according to the screen space bounding box to generate a list of Gaussian points inside each small block; use the GPU partial sum algorithm to perform a one-time continuous video memory allocation for the list of Gaussian points in all small blocks to maintain the depth order of Gaussian points;

[0017] For each small screen block, a GPU thread block is created, with each thread processing 1 pixel. Traverse all the Gaussian points in the list of this small block in depth order from far to near, calculate the pixel color according to the following formula, and perform forward rendering;

[0018] I0(d) = 0 (1)

[0019] I j (d) = (1 - A j G j (J -1 (d)))I j-1 (d) + A j G j (J -1 (d))C(p j ) (2)

[0020] After the pixel color gradient calculation is completed, again, for each small screen block, a GPU thread block is created, with each thread processing 1 pixel. Traverse all the Gaussian points in the list of this small block in depth order from near to far, and use the Jacobi matrix of formula (2) to convert the pixel color gradient into the gradient of the Gaussian point color C j and the gradient of the Gaussian function G j (d), and accumulate them into the shared memory in the thread block for temporary storage; after accumulating the gradients of 16 Gaussian points, use atomic operations to accumulate them into the final gradient tensor in the global video memory.

[0021] Among them, step S5 includes:

[0022] S51. Calculate the Gaussian function G j ; The core idea of Gaussian Splatting is to represent each point in the point cloud with a three-dimensional Gaussian distribution, and distribute energy in space through its probability density function (PDF), so as to achieve smooth and continuous rendering. Each point in the three-dimensional Gaussian distribution model is represented by a three-dimensional Gaussian function

[0023]

[0024] p is the Gaussian center position.

[0025] Σ is the covariance matrix, which controls the shape and size of the Gaussian distribution.

[0026] S52. Improve the lighting model in traditional Gaussian Splatting according to the characteristics of sonar images, and use it to calculate the echo intensity C(p j ) of each Gaussian;

[0027] S53. Render and reconstruct the image I according to the Gaussian parameters;

[0028] S54. Train the Gaussian parameters.

[0029] Among them, the illumination model described in step S52 is represented by the function C(p j ), and the specific content is as follows:

[0030] Based on the spherical harmonic function formula:

[0031]

[0032] Among them is the associated Legendre polynomial, where l and m are the order and degree scalars;

[0033] In the framework of traditional Gaussian splashing, the color of each Gaussian point is defined by the spherical harmonic function series in its line-of-sight direction;

[0034]

[0035] Among them is the spherical harmonic function coefficient, t i is the viewpoint translation vector of the i-th picture, p j is the position of the j-th Gaussian center; p j -t i is the result of the Gaussian center position after viewpoint transformation, that is, the line-of-sight direction from this viewpoint;

[0036] Refine the shading model, calculate the sound intensity and reflection intensity, and multiply them to obtain the final reflected sound intensity. Discuss the originally coupled sound source and Bidirectional Reflectance Distribution Function (BRDF) separately:

[0037] First is the sound source part. Considering that the sound source of sonar often has certain anisotropic characteristics, it is stronger in the direct line-of-sight direction and weaker around, as Figure 2 shown. Use the spherical harmonic function of the sonar local coordinate system to fit this effect:

[0038]

[0039] Among them, S(p j ) is the sound intensity at point p j , is the corresponding sound source spherical harmonic function coefficient.

[0040] Secondly is the bidirectional reflectance distribution function at the Gaussian point. Adopt the microfacet reflection model, such as Figure 3As shown in the figure, it is considered that the object surface is composed of countless micro-surfaces with different orientations, and all the micro-surfaces are specular reflection surfaces. Given the incident direction and the reflection direction, the reflection intensity is proportional to the probability density of the micro-surfaces that produce the corresponding specular reflection. The orientation of such micro-surfaces is called the half-angle direction. The final reflection intensity also includes a geometric term that represents the mutual occlusion of the micro-surfaces, and a Fresnel term that calculates the reflection and refraction ratios at the interface of two media according to different propagation speeds.

[0041] Since the sound source and the viewing point are usually close, the angle between the incident direction and the reflection direction of the sound wave is extremely small. It can be approximately considered that the incident direction and the reflection direction are collinear and opposite in direction. In this way, the incident direction, the reflection direction, and the half-angle direction can be represented by the same vector with different signs. Therefore, the probability density term, the geometric term, and the Fresnel term in the micro-surface reflection model can all be approximated as functions of this vector. A set of spherical harmonic function series is uniformly used for fitting.

[0042] Although the micro-surface probability density is generally defined in the local coordinate system of the reflection point, considering the closure of the spherical harmonic function series of a certain order with respect to rotation transformation, it can be simplified. The reflection distribution function is simply fitted with the spherical harmonic function in the global coordinate system:

[0043]

[0044] Multiplying the sound intensity (Equation 6) by the reflection intensity (Equation 7), we can obtain:

[0045]

[0046] where C(p j ) represents the final reflected sound intensity of the Gaussian point j.

[0047] Among them, the specific content of the model used to render the reconstructed image I in step S53 is as follows:

[0048] Project the center points of all Gaussians into the image space and sort them from far to near according to the depth. Without loss of generality, discuss the intensity I(d) of a pixel d; I(d) is composed of the echo intensities C of each Gaussian j mixed in sequence; initially, it is considered that the intensity of all pixels is 0;

[0049] I0(d) = 0 (9)

[0050] where the subscript j corresponds to the order from far to near; the intensity after mixing the j-th Gaussian is:

[0051] I j (d) = (1 - A j G j (J-1 (d)))I j-1 (d)+A j G j (J -1 (d))C(p j ) (10)

[0052] G j (d) is the contribution of the j-th Gaussian point to pixel d, and the final pixel echo intensity is the mixed result I of all Gaussians n (d); J -1 (d) is the inverse function of the perspective transformation, which transforms pixel d to the depth plane where the center point of Gaussian G j is located.

[0053] Among them, step S54 specifically includes:

[0054] S541. Use the camera parameters R i and t i to perform differentiable rasterization and draw the reconstructed image I;

[0055] S542. Compare the reconstructed image with the result of the original shooting, calculate the loss function L; minimize L using the gradient descent algorithm, and backpropagate the obtained gradient through the steps described in S53, S52, and S51 to the Gaussian center point p j , covariance matrix Σ j , spherical harmonic function coefficients and opacity A j ; the loss L is the first-order loss, which is the weighted average of the D-SSIM loss function, where the weight is a parameter set by the user in advance;

[0056] S543. During the optimization process, dynamically adjust the density of the Gaussian; for the Gaussian with the gradient magnitude of p j > τ, make a copy to increase the density; delete the Gaussian with opacity A j < α; where τ and α are both constants specified by the user in advance;

[0057] S544. Repeat the above steps until the specified number of times is reached.

[0058] Among them, step S6 includes:

[0059] S61. Save the training result as a PLY format point cloud file on the server, and open the HTTPS port on the server to provide a web-based 3D visualization application on this port;

[0060] S62. Optimize and transplant the open-source splatapult framework to the web side;

[0061] S63. Develop visualization user interface components based on WebAssembly.

[0062] Among them, step S62 includes:

[0063] S621. Build a WebGPU depth sorting component;

[0064] S622. Build a WebGL rendering component.

[0065] Among them, the visualization user interface components based on WebAssembly described in step S63 include:

[0066] Object mode: Support XBox controller operation. Rotate the object with the right controller, move the viewpoint up, down, left, and right with the left controller, and use the trigger keys to zoom in and out by adjusting the field of view angle FOV; Provide a transparency filtering function. Use the left and right shoulder buttons to adjust an opacity threshold. Only the Gaussian points with an opacity higher than this threshold will be displayed in the interface.

[0067] Scene mode: Support XBox controller operation. Rotate the view with the right controller, move the viewpoint forward, backward, left, and right with the left controller, and move the viewpoint up and down with the trigger; Provide a transparency filtering function. Use the left and right shoulder buttons to adjust an opacity threshold. Only the Gaussian points with an opacity higher than this threshold will be displayed in the interface.

[0068] The second aspect of the present invention relates to a sonar image adaptive three-dimensional reconstruction and visualization device based on Gaussian splash, including a memory and one or more processors. An executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the sonar image adaptive three-dimensional reconstruction and visualization method of the present invention.

[0069] The innovation points of the present invention are:

[0070] 1. Apply Gaussian splash to sonar images.

[0071] 2. Utilize the characteristics of sonar images to transfer the parameters of the spherical harmonic function series C for coloring from the global space to the local space of the camera to adapt to the common dynamic sound sources in sonar images and achieve a better reconstruction effect. j

[0072] The present invention has the following beneficial effects:

[0073] 1. Complement three-dimensional information. Through the Gaussian splashing algorithm, convert low-resolution and high-noise two-dimensional sonar images into high-quality three-dimensional models. This method can accurately capture the geometric structure of the target, make up for the defect that two-dimensional images cannot provide depth information, and make the reconstruction result more complete and accurate. For example, in underwater facility detection, the shape and distribution of the target can be analyzed more intuitively, effectively improving the accuracy of underwater operations.

[0074] 2. High-efficiency reconstruction ability. This technology improves the efficiency by an order of magnitude compared with traditional three-dimensional reconstruction methods (such as NeRF). This benefits from the point cloud optimized representation method of the Gaussian splashing algorithm, which can efficiently process large-scale data and reduce the consumption of computing resources. Even when dealing with complex sonar data, three-dimensional results can be generated quickly. This high efficiency makes this method particularly suitable for engineering monitoring and modeling applications of large underwater facilities, significantly reducing the working cost and time.

[0075] 3. High-quality reconstruction results. The reconstructed model not only has a highly credible visual effect but also has a significant improvement in detail performance, especially in terms of edge sharpness and surface continuity. Combined with the improved lighting model, the reconstruction result can more realistically reflect the physical characteristics of the target and is suitable for refined tasks such as dam crack detection and equipment status assessment of nuclear power plant cooling ponds. The high-quality reconstruction results provide a reliable basis for further data analysis, risk assessment, and scientific research. Brief Description of the Drawings

[0076] Figure 1 is the flowchart of the adaptive three-dimensional reconstruction based on Gaussian splashing of the present invention.

[0077] Figure 2 is the schematic diagram of the anisotropic point sound source of the present invention.

[0078] Figure 3 is the approximate simulation diagram of the micro-surface reflection model of the present invention in the same direction.

[0079] Figure 4 is the flowchart of the three-dimensional reconstruction system deployment of the Gaussian splashing algorithm of the present invention. Detailed Embodiments

[0080] To better understand the above technical solutions, the technical solutions will be elaborated in detail below in combination with specific embodiments.

[0081] Embodiment 1

[0082] As Figure 1 , this embodiment provides an adaptive three-dimensional reconstruction and visualization method for sonar images based on Gaussian splashing. In the following, lowercase letters are used to represent three-dimensional vectors (such as p), uppercase letters are used to represent matrices or functions (such as V), and bold lowercase letters are used to represent images (such as ), the method of the present invention includes the following steps:

[0083] S1. Build a training server system.

[0084] First, build a set of training server systems. Each server is equipped with 4 domestic MX C series GPGPU processors, connected to the network through 10 Gigabit Ethernet, and receives the original sonar image data obtained from underwater detection devices. The server processes large-scale image data through an efficient data transmission protocol and hardware resources to ensure the rapid transmission and processing of data.

[0085] S2. Transplant the CUDA components to the MXMACA software stack.

[0086] To improve the execution efficiency of the algorithm, transplant the CUDA components required by the Gaussian splash algorithm to the MXMACA software stack independently developed by MX. This software stack supports parallel computing, can efficiently process the computing tasks of each point cloud, and optimizes the data processing process to make the training process more efficient and scalable.

[0087] The processes of step S1 and step S2 are as Figure 4 shown.

[0088] The CUDA components described in step S2 include: sonar image denoising filter components, SIFT feature extraction components in the Structure from Motion (SfM) technology for motion recovery, Random Sample Consensus (RANSAC) fitting components, differentiable Gaussian rasterization components, and spherical harmonic function calculation components.

[0089] The differentiable Gaussian rasterization components include the following construction steps:

[0090] First, project all Gaussian points onto the screen space according to the camera parameters, and then sort all Gaussian points by depth using radix sort; then truncate the radius according to 2.35 times the standard deviation and approximate it by orthogonal projection to calculate the screen space bounding box of each Gaussian point;

[0091] Divide the screen into 16*16 small blocks, add each Gaussian point to the small blocks it covers according to the screen space bounding box, and generate a list of Gaussian points inside each small block; use the GPU partial sum algorithm to perform a one-time continuous video memory allocation for the list of Gaussian points in all small blocks to maintain the depth order of Gaussian points;

[0092] Open a GPU thread block for each screen small block, each thread processes 1 pixel, traverse all Gaussian points in the list of this small block in depth order from far to near, calculate the pixel color according to the following formula, and perform forward rendering;

[0093] I0(d) = 0 (1)

[0094] I j (d) = (1 - A j G j (J -1 (d)))I j-1 (d) + A j G j (J -1 (d))C j (2)

[0095] After the pixel color gradient calculation is completed, a GPU thread block is opened for each screen block again, and each thread processes 1 pixel. Traverse all the Gaussian points in the list of this block in depth order from near to far, and use the Jacobi matrix of formula (2) to convert the pixel color gradient into the gradient of the Gaussian point color C j and the gradient of the Gaussian function G j (d), and accumulate it into the shared memory in the thread block for temporary storage; after accumulating the gradients of 16 Gaussian points, use atomic operations to accumulate them into the final gradient tensor in the global video memory.

[0096] S3. Image preprocessing.

[0097] S31. Collect and organize a series of scanning sonar images x i , and perform denoising processing on them to reduce the impact of environmental noise on the image quality.

[0098] S32. Extract the target area through background segmentation (mask generation), eliminate irrelevant background information, and obtain the mask image to lay the foundation for subsequent reconstruction.

[0099] S4. Run the Structure from Motion (SfM) software for structure recovery from motion.

[0100] S41. Run the standard Structure from Motion (SfM) software to extract and match two-dimensional feature points from the sonar images, and generate the initial point cloud data p j , the rotation matrix R i corresponding to the shooting position of each picture i and the translation vector t

[0101] S42. Process the running results of the structure recovery from motion. For each point P in the point cloud j generate a random covariance matrix Σ j , thus converting each point into a Gaussian. And randomly generate the spherical harmonic function coefficients for subsequent use and the opacity A j(Opacity).

[0102] S5. Train Gaussian point cloud. Based on the generated initial point cloud, apply the Gaussian splatting algorithm to generate the Gaussian point cloud, and improve the shading model according to the characteristics of the sonar image to enhance the authenticity and detail retention of the reconstruction result.

[0103] S51. Calculate the Gaussian function G j

[0104]

[0105] S52. Calculate the echo intensity C(p j ):

[0106] The shading model described in step S52 is represented by the function C(p j ), and the specific content is as follows:

[0107] Based on the spherical harmonic function formula:

[0108]

[0109] where is the associated Legendre polynomial, where l and m are the order and degree scalars;

[0110] In the framework of traditional Gaussian splatting, the color of each Gaussian point is defined by the spherical harmonic function series in its line-of-sight direction;

[0111]

[0112] where is the spherical harmonic function coefficient, t i is the viewpoint translation vector of the i-th image, p j is the j-th Gaussian center position; p j -t i is the result of the Gaussian center position after viewpoint transformation, that is, the line-of-sight direction from this viewpoint;

[0113] Refine the shading model, calculate the sound intensity and reflection intensity, and multiply them. The final reflected sound intensity is:

[0114]

[0115] where C(p j ) represents the final reflected sound intensity of Gaussian point j.

[0116] where, is the spherical harmonic function coefficient of the reflection function, is the spherical harmonic function coefficient of the sound source, t i is the viewpoint translation vector of the i-th image, Ri is the viewpoint rotation matrix of the i-th image, p j is the position of the j-th Gaussian center. p j -t i is the result of the viewpoint transformation of the Gaussian center position, that is, the line-of-sight direction from this viewpoint.

[0117] S53. Draw the reconstructed image I according to various parameters of the Gaussian. The drawing process uses a differentiable rasterization algorithm. The rasterization algorithm uses the camera parameters R i and t i corresponding to the original image i, and the perspective transformation J(p) of the acoustic camera. The pixel intensity formula of the rasterization algorithm is:

[0118] I0(d) = 0 (6)

[0119] I j (D) = (1 - A j G j (J -1 (d)))I j-1 (d) + A j G j (J -1 (d))C j (7)

[0120] This formula is an iterative formula that projects the center points of all Gaussians into the image space and sorts them from far to near by depth. Among them, I(d) is the intensity of each pixel d, which is successively mixed by the echo intensity C j of each Gaussian. Initially, it is considered that the intensity of all pixels is 0 according to formula (1). Then each Gaussian j is successively mixed in according to formula (2). Among them, G j is obtained from (9), and C j is obtained from (10).

[0121] S54. Train the Gaussian parameters.

[0122] S541. Calculate the loss function L according to the reconstructed image I and the corresponding original image. The loss L is a first-order loss, which is the weighted average of the D-SSIM loss function, where the weight λ is a parameter set in advance by the user.

[0123] L = (1 - λ)L1 + λL D-SSIM (9)

[0124] S542. Call the gradient descent solver to minimize L, and backpropagate the obtained gradient to the Gaussian center point p j , the covariance matrix Σ j , the spherical harmonic function coefficients and opacity A j , to improve the values of these parameters, thereby improving the reconstruction effect.

[0125] S543. During the optimization of S35, dynamically adjust the density of the Gaussian. For the Gaussian with the gradient magnitude of p j > τ, it is considered that the nearby density is insufficient, and it is copied to increase the density. For opacity A j < α Gaussian, its contribution is considered too low and it is deleted. Among them, both τ and α are constants specified by the user in advance.

[0126] S544. Repeat the above steps until the specified number of times is reached.

[0127] S6. Result visualization

[0128] S61. Save the results, and save the Gaussian center point p j , covariance matrix Σ j , spherical harmonic function coefficients and opacity A j locally for visualization.

[0129] S62. Repeat the process of S51 to S53 to visualize the results.

[0130] S621. During the drawing process corresponding to S53, the depth sorting originally implemented by MXMACA is changed to the WebGPU version.

[0131] S622. During the drawing process corresponding to S51, replace the differentiable MXMACA rasterization algorithm with a non-differentiable WebGL rasterization algorithm, thereby reducing the computing power cost to meet the requirements of visualization.

[0132] S63. Develop a visualization interface based on the VTK framework and compile it into WebAssembly to be integrated into the web page.

[0133] Provide an object mode in the interface. Support XBox controller operation. Rotate the object with the right controller, move the viewing point up, down, left, and right with the left controller, and use the trigger key to zoom in and out by adjusting the field of view (FOV). Provide a transparency filtering function. Use the left and right shoulder buttons to adjust an opacity threshold, and only the Gaussian points with an opacity higher than this threshold will be displayed in the interface.

[0134] Provide a scene mode in the interface. Support XBox controller operation. Rotate the view with the right controller, move the viewing point forward, backward, left, and right with the left controller, and move the viewing point up and down with the trigger. Provide a transparency filtering function. Use the left and right shoulder buttons to adjust an opacity threshold, and only the Gaussian points with an opacity higher than this threshold will be displayed in the interface.

[0135] Example 2

[0136] This embodiment relates to a sonar image adaptive three-dimensional reconstruction and visualization device based on Gaussian splashing, including a memory and one or more processors. Executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the sonar image adaptive three-dimensional reconstruction and visualization method based on Gaussian Splatting in Embodiment 1.

[0137] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, so that the instructions executed by the processors of the computer or other programmable data processing devices generate for implementation in the process Figure 1 one process or multiple processes and / or blocks Figure 1 a device for the functions specified in one block or multiple blocks.

[0138] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the process Figure 1 one process or multiple processes and / or blocks Figure 1 a device for the functions specified in one block or multiple blocks.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the process Figure 1 one process or multiple processes and / or blocks Figure 1 a device for the functions specified in one block or multiple blocks.

[0140] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0141] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0142] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0144] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0145] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present invention may also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

[0146] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts among the embodiments may be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts may be referred to the description of the method embodiment.

[0147] The above description is only for the embodiments of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and changes can be made to the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. An adaptive three-dimensional reconstruction and visualization method for sonar images based on Gaussian splashing, comprising the following steps: S1. Set up a training server. The server accesses the network through a 10 Gigabit Ethernet and receives raw data. S2. Transplant the CUDA components in the algorithm modules required for Gaussian splashing onto the software stack. S3. Image preprocessing. Organize a series of sonar images obtained and perform denoising processing on them. Then extract the target area through background segmentation to eliminate irrelevant background information. S4. Based on the Structure from Motion (SfM) technology, generate point cloud data by extracting feature points, optimize the perspective transformation, and assign covariance matrices, spherical harmonic function coefficients, and opacity. S5. Train Gaussian point clouds. Based on the generated initial point clouds, perform differentiable rendering using a rendering model. Apply the Gaussian splashing algorithm to generate Gaussian point clouds, and improve the color and lighting model (shading model) according to the characteristics of sonar images. S6. Result visualization. Perform real-time rendering and visual display on the optimized Gaussian point clouds to generate a high-quality three-dimensional model.

2. The sonar image adaptive three-dimensional reconstruction and visualization system based on Gaussian splash according to claim 1, wherein: The CUDA components described in step S2 include: a sonar image denoising filter component, a SIFT feature extraction component in the Structure from Motion (SfM) technology, a Random Sample Consensus (RANSAC) fitting component, a differentiable Gaussian rasterization component, and a spherical harmonic function calculation component.

3. For the adaptive three-dimensional reconstruction and visualization method for sonar images based on Gaussian splashing as described in claim 2, the differentiable Gaussian rasterization component includes the following construction steps: First, project all Gaussian points onto the screen space according to the camera parameters, and then sort all Gaussian points by depth using radix sort. Then truncate the radius according to 2.35 times the standard deviation and approximate it by orthogonal projection to calculate the screen space bounding box of each Gaussian point. Divide the screen into 16 * 16 small blocks, and add each Gaussian point to the small blocks it covers according to the screen space bounding box to generate a list of Gaussian points inside each small block. Use the GPU part and algorithm to perform a one-time continuous video memory allocation for the list of Gaussian points in all small blocks to maintain the depth order of Gaussian points. Open a GPU thread block for each screen small block. Each thread processes 1 pixel, traverse all Gaussian points in the list of this small block in depth order from far to near, calculate the pixel color according to the following formula, and perform forward rendering. I0(d) = 0 (1) I j (d) = (1 - A j G j (J -1 (d)))I j-1 (d) + A j G j (J -1 (d))C(p j ) (2) After the pixel color gradient calculation is completed, a GPU thread block is created for each screen patch again. Each thread processes 1 pixel, and the Gaussian points in all the patch lists are traversed in depth order from near to far. The pixel color gradient is converted into the Gaussian point color C using the Jacobi matrix in formula (2). j of the gradient and the Gaussian function G j (d), and accumulate it into the shared memory in the thread block for temporary storage; after accumulating the gradients of 16 Gaussian points, use atomic operations to accumulate them into the final gradient tensor in the global video memory.

4. For the adaptive three-dimensional reconstruction and visualization system for sonar images based on Gaussian splashing as described in claim 1, step S5 includes: S51. Calculate the Gaussian function G j ; S52. Improve the illumination model in traditional Gaussian splash according to the characteristics of sonar images, and use it to calculate the echo intensity C(p j ); S53. Render the reconstructed image I according to the Gaussian parameters. S54. Train the Gaussian parameters.

5. The sonar image adaptive three-dimensional reconstruction and visualization system based on Gaussian splash according to claim 4, the illumination model described in step S52 is represented by the function C(p j ) and the specific content is as follows: Based on the spherical harmonic function formula: wherein are associated Legendre polynomials, where l and m are order and degree scalars; Under the framework of traditional Gaussian splashing, the color of each Gaussian point is defined by the spherical harmonic function series in its line-of-sight direction. where are the spherical harmonic function coefficients, t i is the viewpoint translation vector of the i-th image, p j is the Gaussian center position of the j-th; p j -t i is the result of the Gaussian center position after the viewpoint transformation, that is, the line-of-sight direction from this viewpoint; Refine the shading model, calculate the sound intensity and reflection intensity, and multiply them. The final reflected sound intensity is: Among which C(p j ) represents the final reflected sound intensity at the Gaussian point j.

6. For the method for adaptive three-dimensional reconstruction and visualization of sonar images based on Gaussian splashing as described in claim 1, the specific content of the model used to render the reconstructed image I in step S53 is as follows: Project the centers of all Gaussians into the image space and sort them by depth from far to near. Without loss of generality, discuss the intensity I(d) of a pixel d; I(d) is composed of the echo intensities C of each Gaussian j which are mixed in turn; initially, assume that the intensity of all pixels is 0; I0(d) = 0 (6) Among them, The subscript j corresponds to the order from far to near; the intensity after mixing the j-th Gaussian is: I j (d) = (1 - A j G j (J -1 (d)))I j-1 (d) + A j G j (J -1 (d))C(p j ) (7) G j (d) is the contribution of the j-th Gaussian point to pixel d, and the final pixel echo intensity is the mixed result I n (d); J -1 (d) is the inverse function of the perspective transformation, which transforms pixel d to the depth plane where the center point of Gaussian G j is located.

7. For the system for adaptive three-dimensional reconstruction of sonar images based on Gaussian splashing as described in claim 1, step S54 specifically includes: S541. Use the camera parameters R i and t i to perform differentiable rasterization and draw the reconstructed image I; S542. Compare the reconstructed image with the result of the original capture, calculate the loss function L; minimize L using the gradient descent algorithm, and backpropagate the obtained gradient through the steps described in S53, S52, and S51 back to the Gaussian center point p j , the covariance matrix Σ j , the spherical harmonic function coefficients and the opacity A j ; the loss L is a first-order loss, a weighted average of the D-SSIM loss function, where the weight is a parameter pre-set by the user; S543. During the optimization process, dynamically adjust the density of the Gaussian; for the Gaussian with the gradient magnitude of p j > τ, make a copy to increase the density; delete the Gaussian with opacity A j < α; where τ and α are both constants specified by the user in advance. S544. Repeat the above steps until the specified number of times is reached.

8. The method for adaptive three-dimensional reconstruction and visualization of sonar images based on Gaussian splash according to claim 1, wherein: Step S6 includes: S61. Save the training result as a PLY format point cloud file on the server, and open the HTTPS port on the server. Provide a web-based 3D visualization application on this port. S62. Optimize and transplant the open-source splatapult framework to the web side. S63. Develop a visualization user interface component based on WebAssembly. Step S62 includes: S621. Build a WebGPU depth sorting component. S622. Build a WebGL drawing component.

9. The method for adaptive three-dimensional reconstruction and visualization of sonar images based on Gaussian splash according to claim 7, wherein: The visualization user interface component based on WebAssembly described in step S63 includes: Object mode: Supports XBox controller operation. The right handle rotates the object, the left handle moves the viewing point up, down, left, and right, and the trigger key zooms in and out by adjusting the field of view angle FOV; provides a transparency filtering function. Use the left and right shoulder buttons to adjust an opacity threshold. Only Gaussian points with an opacity higher than this threshold will be displayed in the interface. Scene mode: Supports XBox controller operation. The right handle rotates the viewing angle, the left handle moves the viewing point forward, backward, left, and right, and the trigger moves the viewing point up and down; provides a transparency filtering function. Use the left and right shoulder buttons to adjust an opacity threshold. Only Gaussian points with an opacity higher than this threshold will be displayed in the interface.

10. An adaptive three-dimensional reconstruction and visualization device for sonar images based on Gaussian splash, characterized in that, It includes a memory and one or more processors. Executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the method for adaptive three-dimensional reconstruction and visualization of sonar images based on Gaussian splashing as described in any one of claims 1-9.

Images