Underwater target three-dimensional reconstruction method and system based on three-dimensional imaging sonar image

Abstract

The invention provides an underwater target three-dimensional reconstruction method and system based on a three-dimensional imaging sonar image, and the method comprises the steps: collecting the three-dimensional imaging sonar data of an underwater target, segmenting the sonar point cloud data through employing an optimized point cloud segmentation model, and extracting a specific target point cloud; and performing sparse reconstruction and dense reconstruction on the target point cloud in sequence, and finally realizing high-precision three-dimensional reconstruction of the target. The system comprises a data acquisition module, a preprocessing module, a reconstruction algorithm module and a visualization module, and is suitable for underwater target surveying, mapping and monitoring. The invention aims to solve the problem of insufficient reconstruction precision and efficiency of a traditional method in a complex underwater environment.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction of underwater targets, and particularly to a method and system for three-dimensional reconstruction of underwater targets based on three-dimensional imaging sonar images. Background Art

[0002] The three-dimensional reconstruction technology of underwater targets has wide application value in the fields of marine exploration, military applications, underwater archaeology, environmental monitoring, etc. Due to the complexity of the underwater environment (such as problems like light attenuation, low visibility, turbidity, etc.), traditional optical sensors are severely restricted in underwater applications. Therefore, sonar technology, as a detection method relying on sound wave propagation, has become an important tool in the field of underwater detection due to its strong underwater penetration ability and stability.

[0003] Currently, underwater three-dimensional reconstruction technology mainly relies on the following two schemes: three-dimensional reconstruction of underwater targets based on optical cameras and three-dimensional reconstruction of underwater targets based on two-dimensional sonars. Although the optical method performs well in the air environment, its applicability is restricted in the underwater environment due to light scattering and attenuation. It only has high precision in clear waters and cannot detect targets in occluded areas. Two-dimensional sonar technology can generate point cloud data within a certain range through multiple scans and reconstruct the scene in combination with the motion trajectory. However, the data acquisition efficiency of two-dimensional sonar devices is low, and complex motion compensation algorithms are required, resulting in limited reconstruction accuracy and integrity. Three-dimensional imaging sonar (Forward-Looking 3D Sonar, FLS) is a new type of sonar device that can directly obtain the three-dimensional point cloud data of the underwater scene and the reflection intensity information of the target. Although three-dimensional imaging sonar technology has significant advantages in data acquisition, due to problems such as underwater noise interference, sparse point cloud data, and blurred target boundaries, its direct application to high-precision three-dimensional reconstruction still poses technical challenges: such as high difficulty in point cloud segmentation, insufficient accuracy in dense reconstruction, and complex underwater surface fitting. Therefore, it is very necessary to carry out research on a method for three-dimensional reconstruction of underwater targets that combines point cloud segmentation and three-dimensional reconstruction.

[0004] The ability to provide high-resolution point cloud data is an important tool for underwater target detection and reconstruction. In the prior art, there are problems such as low accuracy in point cloud segmentation and incomplete reconstruction results in three-dimensional reconstruction based on sonar point clouds. In addition, traditional reconstruction methods fail to fully combine the characteristics of underwater sonar data, resulting in limited applicability. Therefore, an innovative segmentation algorithm and an optimized reconstruction method are needed to improve the performance of the system. Summary of the Invention

[0005] The present invention aims to provide a method and system for three-dimensional reconstruction of underwater targets based on three-dimensional imaging sonar images, and realizes the efficient reconstruction of specific underwater targets through an innovative point cloud segmentation method and an improved reconstruction algorithm.

[0006] The technical solution for achieving the object of the present invention is as follows:

[0007] An underwater target three-dimensional reconstruction method based on three-dimensional imaging sonar images, comprising the steps of:

[0008] S1. Obtain sonar image data of the underwater area where the target is located through a three-dimensional imaging sonar device, and the sonar image includes point cloud data of the target area and its reflection intensity information;

[0009] S2. Segment the point cloud data by using a hybrid architecture based on a dynamic graph convolutional network and a Transformer, extract local geometric features through a dynamic graph structure, capture global context dependencies by combining a self-attention mechanism, and extract the point cloud of the target area;

[0010] S3. Apply the structure from motion method to the target point cloud data obtained in step S2 for sparse reconstruction of the point cloud to obtain a sparse three-dimensional structure;

[0011] S4. Use a patch-based multi-view dense reconstruction method to densify the sparse point cloud to generate dense point cloud data;

[0012] S5. Use the Poisson surface reconstruction method to perform surface fitting and optimization on the dense point cloud to generate a target three-dimensional model.

[0013] Further, the steps of S1 specifically include:

[0014] Use a three-dimensional imaging sonar device to perform sonar scanning in the underwater target area to obtain sonar image data of the area;

[0015] Extract the point cloud data of the target area from the obtained sonar image, and the point cloud data represents the spatial position of the underwater target;

[0016] Obtain the reflection intensity information of the target area, and the reflection intensity information reflects the distance and reflection characteristics between the underwater target and the sonar device. Combine the point cloud data with the reflection intensity information to form complete three-dimensional sonar image data for subsequent three-dimensional reconstruction of the underwater target.

[0017] Further, the underwater targets include: common regular-shaped targets and other irregular-shaped targets.

[0018] Further, the hybrid architecture of DGCNN and the Transformer specifically includes:

[0019] Dynamic graph construction layer: Construct a k-nearest neighbor graph for the input point cloud, dynamically update the adjacency relationship through a learnable edge weight function, and calculate local features:

[0020] F local= DGConv(X, N(x i ))

[0021] where DGConv is a dynamic graph convolution operation, and N(x i ) is a dynamically updated k-nearest neighbor set.

[0022] Transformer encoding layer: After serializing the global point cloud, it is input into the multi-head self-attention module to calculate the long-range dependencies of the features:

[0023] F global = MultiHead(Q, K, V)

[0024] where Q, K, and V are query, key, and value matrices generated through point cloud position encoding.

[0025] Feature fusion module: Fuses local features and global features through a gating mechanism:

[0026] F fused = σ(W g [F local ; F global ) ⊙ F local + (1 - σ(W g [F local ; F global )) ⊙ F global

[0027] where σ is the sigmoid function, W g is a learnable weight matrix, and ⊙ represents element-wise multiplication.

[0028] Output layer: Classifies the fused features through a fully connected network and outputs the point cloud segmentation result.

[0029] Furthermore, the specific steps of segmenting point cloud data using the hybrid architecture based on the dynamic graph convolution network and Transformer include:

[0030] Step S2-1, perform point cloud data definition and preprocessing. The input point cloud data format is defined as:

[0031] P = {(x i , y i , z i , r i , g i , b i , s i ) | i = 1, 2,..., N}

[0032] where (x i , y i , z i) are three-dimensional space coordinates, directly obtained through sonar imaging, (r i , g i , b i ) are RGB color values, obtained by registering and mapping an optical camera and sonar, s i is the sonar reflection intensity, characterizing the target material properties (metal / rock / living things, etc.).

[0033] Step S2-2, normalize the coordinates and modal data:

[0034]

[0035] where μ x , σ x are the coordinate mean and standard deviation, s min , s max are the intensity normalization boundaries.

[0036] Step S2-3, dynamic neighborhood construction (Top-K refinement), select the neighborhood based on the joint metric of geometric distance and feature similarity:

[0037] Distance calculation:

[0038] Geometric distance:

[0039] Feature distance:

[0040] Top-K selection:

[0041] Comprehensive distance: where α is a learnable parameter, optimized through backpropagation

[0042] Neighborhood set: N(i) = {j|d ij belongs to the top K minimum values}

[0043] Step S2-4, feature extraction and fusion:

[0044] Geometric feature extraction: map the coordinate information through an MLP network with shared weights:

[0045]

[0046] (MLP structure: input layer 32 → 64 → 128, ReLU activation);

[0047] Modal feature extraction: fuse color and sonar intensity information:

[0048]

[0049] (MLP structure: input layer 42 → 32 → 44, Sigmoid activation);

[0050] Cross-modal feature fusion: After splicing geometric and modal features, perform secondary encoding:

[0051]

[0052] (MLP structure: Input layer 192→256, LayerNorm normalization);

[0053] Step S2-5, Attention weight calculation and local aggregation

[0054] Attention weight calculation, calculate neighborhood weights through pairwise feature similarity:

[0055]

[0056] where a is a learnable vector, and || represents feature splicing;

[0057] Local feature aggregation, weighted sum of neighborhood features:

[0058]

[0059] Step S2-6, Global-local feature joint optimization:

[0060] Global context modeling, obtain global features through max pooling:

[0061] f global = MAX({h i | i = 1,…,N})

[0062] Feature enhancement, broadcast the global feature to each point and splice:

[0063]

[0064] Classification output, predict class probabilities through a fully connected layer:

[0065]

[0066] Step S2-7, Post-processing optimization:

[0067] Conditional random field (CRF) optimization, define the energy function:

[0068]

[0069] where: Unary potential is provided by the network predicted probability p i Binary potential: Parameter settings: λ = 3.0, θ = 0.1, optimized for underwater scenes. By combining local enhanced features with global features, the semantic expression ability of each point is further improved.

[0070] Finally, by classifying the final features of each point, the probability of the category to which the point belongs is output, and the category with the highest probability is selected as the final segmentation result for each point. In addition, a post-processing method based on point category consistency is used to optimize the segmentation results and eliminate noise points.

[0071] Furthermore, step S3 specifically includes:

[0072] Step 3-1: Extract key feature points from the target point cloud data, describe the feature points using feature descriptors based on geometric characteristics (such as normal vectors, curvature, etc.), and then use feature matching algorithms to match the feature points in the multi-view point cloud data to generate corresponding feature point pairs and form a preliminary matching relationship;

[0073] Step 3-2, calculate the relative pose and position information of the multi-view point cloud data, including the rotation matrix and translation vector, based on the feature point matching results:

[0074]

[0075] Where, P i and P′ i is the matching feature point pair, R is the rotation matrix, and t is the translation vector;

[0076] Step 3-3: Use the initial pose for global optimization and further optimize the camera parameters and 3D point coordinates through the bundle adjustment algorithm to minimize the reprojection error:

[0077]

[0078] Where π represents the projection function, x i,j is the observation point on the image plane, X i is a three-dimensional space point, K is the camera intrinsic parameter matrix;

[0079] In step 3-4, the multi-view point cloud data is jointly reconstructed based on the optimized posture and position information to generate a sparse 3D point cloud, and the mismatched feature points are eliminated to ensure the geometric accuracy and integrity of the sparse 3D point cloud.

[0080] Furthermore, step S4 uses a patch-based multi-view stereo (PMVS) method to densify the sparse point cloud, specifically including:

[0081] Step 4-1: Take the sparse point cloud data and multi-view information as input, including the three-dimensional spatial positions of the sparse points and the pose parameters of the multi-view point clouds. Then normalize the view parameters of the multi-view point clouds to ensure that the reconstruction calculations are carried out in a unified coordinate system;

[0082] Step 4-2: Generate an initial patch set according to the sparse point cloud. The initialization parameters of patch p = (X, N, V) include: the three-dimensional coordinates X of the patch center point, the normal vector of the patch calculated from the local geometric relationship of the sparse point cloud and the view set V indicating in which views the patch is visible;

[0083] Step 4-3: Starting from the initial patches, gradually expand to the neighborhood points to generate new candidate patches. By calculating the projection consistency between the new candidate patches and the multi-view images, select the patches that meet the reconstruction conditions:

[0084]

[0085] where C(p) is the patch consistency score, ω i is the weight, Sim(I i , π(p)) is the projection consistency metric, and π(p) represents the projection of the patch in view I i ;

[0086] Step 4-4: Perform global optimization on the generated candidate patch set. Through the joint constraints of multi-view geometry and intensity consistency, optimize the normal vector and position of each patch. By calculating the similarity of the overlapping regions of the patches, remove redundant patches:

[0087]

[0088] If the redundancy rate exceeds the threshold, remove the redundant patches. Eliminate redundant and incorrect patches to ensure the geometric accuracy and dense distribution of the patches. Through the generation and fusion of depth maps, further optimize the connectivity and boundary consistency between the patches;

[0089] Step 4-5: Extract the dense point cloud data from the optimized patch set, including the three-dimensional coordinates and normal vectors of each point, ensure the geometric structure integrity and accuracy of the dense point cloud, and take the final dense point cloud as the output for subsequent surface reconstruction steps.

[0090] Furthermore, step S5 uses the Poisson surface reconstruction method to perform surface fitting and optimization on the dense point cloud. The steps include:

[0091] Step 5-1: Perform normal vector estimation. Extract the normal vector of each point from the dense point cloud data, and use the estimation method based on the point cloud neighborhood to calculate the normal vector N i :

[0092]

[0093] In the formula, N i is the set of neighborhood points of point P i , and N i is the normal vector of the point. Standardize the direction of the normal vector and ensure the consistency of the normal vector through global optimization. N' i is the normal vector of the extracted point;

[0094] Step 5-2: Use the spatial position and normal vector of the point cloud to construct a Poisson equation. In the Poisson equation, the density function ρ(x, y, z) introduces the reflection intensity weight:

[0095] Δ·v = ρ

[0096] In the formula, ν is the vector field generated by the normal vector field; ρ is the distribution density function of the point cloud in three-dimensional space, generated by the position and normal vector weight of the point cloud. ρ(x, y, z) = ∑(i)ω i ·s i ·δ(x - x i , y - y i , z - z i ), where ω i is the point cloud weight, s i is the reflection intensity of the point cloud, used to suppress low-confidence regions, and δ is the three-dimensional Dirac function;

[0097] Step 5-3: Discretize the Poisson equation into a system of linear equations. Construct a multi-resolution grid by octree hierarchical structure, and use an iterative solution method (such as the conjugate gradient method) to numerically solve the system of equations to generate a preliminary reconstructed grid model;

[0098] Step 5-4: Optimize the generated preliminary grid model to eliminate noise and artifacts; through surface smoothing, in the surface optimization stage, adjust the smoothing weight according to the normal vector direction, retain sharp edges, and ensure geometric consistency and continuity between grid vertices:

[0099]

[0100] α′ ij = exp(-||N i - N j || 2 / σ

[0101] In the formula, v i is the position of vertex i, α′ ij is the smoothing weight parameter, N i , N j are the normal vectors of adjacent vertices; Output the optimized surface model as the final reconstruction result.

[0102] An underwater target three-dimensional reconstruction system based on three-dimensional imaging sonar images, comprising:

[0103] A data acquisition unit, which acquires sonar image data of the underwater area where the target is located through a three-dimensional imaging sonar device, and the sonar image contains point cloud data of the target area and its reflection intensity information;

[0104] A point cloud segmentation unit, which uses a hybrid architecture based on a dynamic graph convolutional network and a Transformer to segment the point cloud data, extracts local geometric features through a dynamic graph structure, combines a self-attention mechanism to capture global context dependencies, and extracts the point cloud of the target area;

[0105] A three-dimensional reconstruction module, which applies the structure from motion method to the target point cloud data for sparse reconstruction of the point cloud to obtain a sparse three-dimensional structure, uses a patch-based multi-view dense reconstruction method to densify the sparse point cloud, generates dense point cloud data, and uses the Poisson surface reconstruction method to perform surface fitting and optimization on the dense point cloud to generate the target three-dimensional model.

[0106] Compared with the prior art, the remarkable advantages of the present invention are as follows:

[0107] 1. The present invention uses a three-dimensional imaging sonar device to obtain three-dimensional point cloud data and reflection intensity information of an underwater target area in one scan, without complex motion compensation or multiple scans, significantly improving the data acquisition efficiency.

[0108] 2. The present invention uses a hybrid architecture based on a dynamic graph convolutional network (DGCNN) and a Transformer to segment the point cloud data, extracts local geometric features through a dynamic graph structure, combines a self-attention mechanism to capture global context dependencies, and extracts the point cloud of the target area, greatly improving the accuracy and robustness of the target point cloud segmentation, especially having significant advantages in complex underwater environments.

[0109] 3. Considering the characteristics of the point cloud data, the present invention combines the structure from motion (SfM) method and the patch-based multi-view dense reconstruction (PMVS) method, and introduces sonar characteristics. First, feature matching is performed on the segmented three-dimensional point cloud data to calculate the relative pose and position information of the multi-view point cloud data. Finally, three-dimensional reconstruction is performed on the point cloud data to obtain sparse point cloud data. Then, a dense three-dimensional structure of the target is reconstructed from the sparse point cloud data to ensure the fineness and integrity of the three-dimensional model.

[0110] 4. The present invention uses the Poisson surface reconstruction method to perform surface fitting on the dense point cloud, constructs a closed and continuous surface through normal vector estimation and global optimization, and at the same time uses multi-resolution grid optimization technology to eliminate noise and artifacts, generating a smooth and natural three-dimensional model.

[0111] 5. In view of problems such as noise, occlusion, and sparse point clouds in the underwater environment, the present invention designs a dedicated algorithm improvement and optimization strategy, which can still stably and accurately complete the three-dimensional reconstruction task under adverse conditions such as turbid waters and blurred target boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Figure 1 It is a flowchart of the underwater target three-dimensional reconstruction method based on three-dimensional imaging sonar images provided by the present invention;

[0113] Figure 2 It is a module diagram of the underwater target three-dimensional reconstruction method based on three-dimensional imaging sonar images provided by the present invention;

[0114] Figure 3 It is a flowchart of the point cloud segmentation described in the present invention;

[0115] Figure 4 It is a flowchart of the sparse reconstruction and dense reconstruction described in the present invention;

[0116] Figure 5 It is a flowchart of the surface reconstruction optimization described in the present invention;

[0117] Figure 6 It is a schematic diagram of the point cloud data described in the present invention;

[0118] Figure 7 It is a schematic diagram of the three-dimensional model obtained after dense reconstruction described in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0119] The present invention will be further described in detail below in conjunction with the drawings and specific embodiments.

[0120] The underwater environment is complex and the light propagation is limited, so the application range of traditional optical devices underwater is limited. The three-dimensional imaging sonar detects underwater targets through sound waves and can obtain their three-dimensional point cloud data, providing accurate information on the geometric shape, spatial position, and attitude of the targets. However, the point cloud data is only discrete sampling points, and a continuous geometric surface model can be generated through the three-dimensional reconstruction method, which is more suitable for intuitive analysis and subsequent operations.

[0121] Through the improvement of the algorithm and the optimization of the process, the present invention realizes the full-process integration from data acquisition to result output, not only improving the accuracy and efficiency of underwater target three-dimensional reconstruction, but also having high applicability and innovation, providing an efficient and reliable technical means for practical applications.

[0122] The present invention proposes a method for three-dimensional reconstruction of underwater targets based on three-dimensional imaging sonar images. Please refer to Figure 1 , specifically including:

[0123] S1. Obtain sonar image data of the underwater area where the target is located through a three-dimensional imaging sonar device. The sonar image contains the point cloud data of the target area and its reflection intensity information. Specifically, it includes:

[0124] Use a three-dimensional imaging sonar device to perform sonar scanning in the underwater target area to obtain sonar image data of this area.

[0125] Extract the point cloud data of the target area from the obtained sonar image. The point cloud data represents the spatial position of the underwater target.

[0126] Obtain the reflection intensity information of the target area. The reflection intensity information reflects the distance and reflection characteristics between the underwater target and the sonar device. Combine the point cloud data with the reflection intensity information to form complete three-dimensional sonar image data for subsequent three-dimensional reconstruction of the underwater target.

[0127] S2. Use a hybrid architecture based on the Dynamic Graph Convolutional Network (DGCNN) and Transformer to segment the point cloud data. Extract local geometric features through a dynamic graph structure, capture global context dependencies by combining with the self-attention mechanism, extract the point cloud of the target area, and combine Figure 3 , specifically including:

[0128] First, define and preprocess the point cloud data. The input point cloud data format is defined as:

[0129] P = {(x i , y i , z i , r i , g i , b i , s i ) | i = 1, 2,..., N}

[0130] where (x i , y i , z i ) are three-dimensional spatial coordinates directly obtained through sonar imaging, (r i , g i , b i ) are RGB color values obtained by registering and mapping an optical camera with the sonar, and s i is the sonar reflection intensity, representing the target material property (metal / rock / creature, etc.).

[0131] Second step, normalize the coordinate and modal data:

[0132]

[0133] where μ x , σ xis the coordinate mean and standard deviation, s min , s max is the intensity normalization boundary.

[0134] The third step is dynamic neighborhood construction (Top-K refinement), where the neighborhood is selected based on the joint metric of geometric distance and feature similarity:

[0135] Distance calculation:

[0136] Geometric distance:

[0137] Feature distance:

[0138] Top-K selection:

[0139] Comprehensive distance: where α is a learnable parameter optimized by backpropagation

[0140] Neighborhood set: N(i) = {j|d ij belongs to the top K smallest values}

[0141] The fourth step is feature extraction and fusion:

[0142] Geometric feature extraction: Map the coordinate information through an MLP network with shared weights:

[0143]

[0144] (MLP structure: input layer 32 → 64 → 128, ReLU activation);

[0145] Modal feature extraction: Fuse color and sonar intensity information:

[0146]

[0147] (MLP structure: input layer 42 → 32 → 44, Sigmoid activation);

[0148] Cross-modal feature fusion: Concatenate geometric and modal features and then perform secondary encoding:

[0149]

[0150] (MLP structure: input layer 192 → 256, LayerNorm normalization);

[0151] The fifth step is attention weight calculation and local aggregation

[0152] Attention weight calculation, calculate the neighborhood weights through point-to-point feature similarity:

[0153]

[0154] where a is a learnable vector, || represents feature concatenation, exp is the natural exponential function used to map the input to the positive domain and amplify the differences, and LeakyReLU is the activation function; in the above formula, LeakyReLU is applied to the vector inner product result a T [f i ||f k , and then the attention weights are calculated through the exponential function and normalization;

[0155] Local feature aggregation, weighted summation of neighborhood features:

[0156]

[0157] Sixth step, global-local feature joint optimization:

[0158] Global context modeling, obtaining global features through max pooling:

[0159] f global =MAX({h i |i = 1,…,N})

[0160] Feature enhancement, broadcasting the global features to each point and concatenating:

[0161]

[0162] Classification output, predicting class probabilities through a fully connected layer:

[0163]

[0164] The last step, post-processing optimization:

[0165] Conditional random field (CRF) optimization, defining the energy function:

[0166]

[0167] where: the unary potential is provided by the network predicted probability p i , and the pairwise potential: Parameter settings: λ = 3.0, θ = 0.1, optimized for underwater scenes. By combining the local enhanced features with the global features, the semantic expression ability of each point is further improved.

[0168] Finally, by classifying the final features of each point, the class probability of the point is output, and the class with the highest probability is selected as the final segmentation result of each point. In addition, a post-processing method based on point class consistency is used to optimize the segmentation result and eliminate noise points.

[0169] S3. Apply the Structure From Motion (SFM) method to the segmented target point cloud data to perform sparse point cloud reconstruction and obtain a sparse three-dimensional structure. Figure 4 , the specific steps include:

[0170] First, key feature points are extracted from the target point cloud data, and feature descriptors based on geometric characteristics (such as normal vectors, curvature, etc.) are used to describe the feature points. Then, a feature matching algorithm is used to match the feature points in the multi-view point cloud data to generate corresponding feature point pairs and form a preliminary matching relationship.

[0171] Next, the relative pose and position information of the multi-view point cloud data, including the rotation matrix and translation vector, are calculated based on the feature point matching results:

[0172]

[0173] Where, P i and P′ i is the matching feature point pair, R is the rotation matrix, and t is the translation vector;

[0174] In the third step, the initial pose is used for global optimization. The camera parameters and 3D point coordinates are further optimized through the bundle adjustment algorithm to minimize the reprojection error:

[0175]

[0176] Where π represents the projection function, x i,j is the observation point on the image plane, X i is a three-dimensional space point, K is the camera intrinsic parameter matrix;

[0177] Finally, based on the optimized posture and position information, the multi-view point cloud data are jointly reconstructed to generate a sparse 3D point cloud, and the mismatched feature points are eliminated to ensure the geometric accuracy and integrity of the sparse 3D point cloud.

[0178] S4. Use the patch-based multi-view stereo (PMVS) method to densify the sparse point cloud to generate dense point cloud data, including:

[0179] First, sparse point cloud data and multi-view information are taken as input, including the 3D spatial position of sparse points and the pose parameters of the multi-view point cloud. The view parameters of the multi-view point cloud are then normalized to ensure that the reconstruction calculation is performed in a unified coordinate system.

[0180] In the second step, an initial patch set is generated based on the sparse point cloud. The initialization parameters of patch p = (X, N, V) include: the three-dimensional coordinates X of the patch center point, the normal vector of the patch calculated from the local geometric relationship of the sparse point cloud and the view set V indicating in which views the patch is visible;

[0181] Next, starting from the initial patches, it gradually expands to neighborhood points to generate new candidate patches. By calculating the projection consistency between the new candidate patches and the multi-view images, the patches meeting the reconstruction conditions are screened out:

[0182]

[0183] In the formula, C(p) is the patch consistency score, ω i is the weight, Sim(I i , π(p)) is the projection consistency metric, and π(p) represents the projection of the patch in view I i ;

[0184] Then, global optimization is performed on the generated candidate patch set. Through the joint constraints of multi-view geometry and intensity consistency, the normal vector and position of each patch are optimized. By calculating the similarity of the overlapping regions of the patches, redundant patches are removed:

[0185]

[0186] If the redundancy rate exceeds the threshold, the redundant patches are removed. Redundant and incorrect patches are eliminated to ensure the geometric accuracy and dense distribution of the patches. Through the generation and fusion of depth maps, the connectivity and boundary consistency between the patches are further optimized;

[0187] Finally, dense point cloud data including the three-dimensional coordinates and normal vectors of each point are extracted from the optimized patch set to ensure the geometric structure integrity and accuracy of the dense point cloud. The final dense point cloud is used as the output for subsequent surface reconstruction steps.

[0188] S5. Use the Poisson surface reconstruction method to perform surface fitting and optimization on the dense point cloud, combined with Figure 5 , to generate a high-quality target three-dimensional model. The steps include:

[0189] First, normal vector estimation is performed. The normal vector of each point is extracted from the dense point cloud data, and an estimation method based on the point cloud neighborhood is used to calculate the normal vector N i of the point:

[0190]

[0191] In the formula, N(i) is the set of neighborhood points of point P i , and N iis the normal vector of the point. Standardize the direction of the normal vector and ensure the consistency of the normal vector through global optimization, N' i is the normal vector of the extracted point;

[0192] Next, using the spatial position and normal vector of the point cloud, construct a Poisson equation, and introduce the reflection intensity weight into the density function ρ(x, y, z) in the Poisson equation:

[0193] Δ·v = ρ

[0194] where ν is the vector field generated by the normal vector field; ρ is the distribution density function of the point cloud in three-dimensional space, generated by the position and normal vector weight of the point cloud, ρ(x, y, z) = ∑(i)ω i ·s i ·δ(x - x i , y - y i , z - z i ), where ω i is the point cloud weight, s i is the point cloud reflection intensity, used to suppress low-confidence regions, and δ is the three-dimensional Dirac function;

[0195] Then, discretize the Poisson equation into a linear system of equations, construct a multi-resolution grid through octree hierarchical construction, and use an iterative solution method (such as the conjugate gradient method) to numerically solve the system of equations to generate a preliminary reconstructed grid model;

[0196] Finally, optimize the generated preliminary grid model to eliminate noise and artifacts; through surface smoothing, in the surface optimization stage, adjust the smoothing weight according to the normal vector direction, retain sharp edges, and ensure geometric consistency and continuity between grid vertices:

[0197]

[0198] α′ ij = exp(-||N i - N j || 2 / σ

[0199] where v i is the position of vertex i, α′ ij is the smooth weight parameter, N i , N j are the normal vectors of adjacent vertices; output the optimized surface model as the final reconstruction result.

[0200] For example Figure 2 , this embodiment provides an underwater target three-dimensional reconstruction system based on a three-dimensional imaging sonar image, including:

[0201] The data acquisition unit uses a three-dimensional imaging sonar to collect underwater target data, obtains sonar image data of the area, extracts point cloud data and reflection intensity information of the target area from the obtained sonar image data, and combines the point cloud data with the reflection intensity information to form complete three-dimensional sonar image data;

[0202] The point cloud segmentation unit uses a hybrid architecture based on the Dynamic Graph Convolutional Network (DGCNN) and Transformer to segment the point cloud data, extracts local geometric features through a dynamic graph structure, captures global context dependencies by combining self-attention mechanisms, and extracts the point cloud of the target area;

[0203] The three-dimensional reconstruction module uses the SFM method for feature matching and sparse point cloud generation of the segmented point cloud, then uses the PMVS method to generate a high-density point cloud, combines noise suppression technology to optimize the reconstruction result, and finally uses Poisson surface reconstruction to optimize the three-dimensional model.

[0204] The system further includes a visualization module, an interactive interface developed based on Qt, which displays the three-dimensional reconstruction result in real time and supports data analysis and processing.

[0205] Combined with examples such as Figure 6 and Figure 7 show that Figure 6 is the actual target and the collected target point cloud data, Figure 7 is a schematic diagram of the three-dimensional model obtained after dense reconstruction using the PMVS method after point cloud segmentation and sparse point cloud generation. Through Figure 6 and Figure 7 it shows that the present invention realizes efficient and high-precision three-dimensional modeling of underwater targets.

[0206] The present invention proposes an innovative target segmentation and three-dimensional reconstruction method for three-dimensional imaging sonar point cloud data. By combining a hybrid architecture based on the Dynamic Graph Convolutional Network (DGCNN) and Transformer with SFM and PMVS technologies, it realizes efficient and high-precision three-dimensional modeling of underwater targets. The system provides a friendly user interaction function through the Qt interface and is applicable to various application scenarios such as underwater target mapping and monitoring.

[0207] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method for underwater targets based on three-dimensional imaging sonar images, characterized in that, Including the steps: S1. Obtain sonar image data of the underwater area where the target is located through a three-dimensional imaging sonar device. The sonar image includes point cloud data of the target area and its reflection intensity information. S2. Use a hybrid architecture based on a dynamic graph convolutional network and a Transformer to segment the point cloud data. Extract local geometric features through a dynamic graph structure, capture global context dependencies by combining the self-attention mechanism, and extract the point cloud of the target area. S3. Apply the structure from motion method to the target point cloud data obtained in step S2 for sparse reconstruction of the point cloud to obtain a sparse three-dimensional structure. S4. Use a patch-based multi-view dense reconstruction method to densify the sparse point cloud and generate dense point cloud data. S5. Use the Poisson surface reconstruction method to perform surface fitting and optimization on the dense point cloud to generate the target three-dimensional model.

2. The three-dimensional reconstruction method of an underwater target based on a three-dimensional imaging sonar image according to claim 1, wherein Step 1 specifically includes: Use a three-dimensional imaging sonar device to perform sonar scanning in the underwater target area to obtain sonar image data of this area. Extract the point cloud data of the target area from the obtained sonar image. The point cloud data represents the spatial position of the underwater target. Obtain the reflection intensity information of the target area. The reflection intensity information reflects the distance and reflection characteristics between the underwater target and the sonar device. Combine the point cloud data with the reflection intensity information to form complete three-dimensional sonar image data.

3. A three-dimensional reconstruction method for underwater targets based on three-dimensional imaging sonar images according to claim 1, characterized in that, The hybrid architecture based on the dynamic graph convolutional network and the Transformer specifically includes: A dynamic graph construction layer that constructs a k-nearest neighbor graph for the input point cloud, dynamically updates the adjacency relationship through a learnable edge weight function, and calculates local features: F local = DGConv(X, N(x i )) Among them, DGConv is a dynamic graph convolution operation, and N(x i ) is a dynamically updated set of k nearest neighbors; A Transformer encoding layer that serializes the global point cloud and inputs it into a multi-head self-attention module to calculate global features: F global = MultiHead(Q, K, V) Among them, Q, K, and V are query, key, and value matrices generated through point cloud position encoding; A feature fusion module that fuses local features and global features through a gating mechanism: F fused = σ(W g [F local ; F global ) ⊙ F local + (1 - σ(W g [F local ; F global )) ⊙ F global where σ is the sigmoid function, W g is the learnable weight matrix, and ⊙ represents element-wise multiplication; An output layer that classifies the fused features through a fully connected network and outputs the point cloud segmentation result.

4. A three-dimensional reconstruction method of an underwater target based on a three-dimensional imaging sonar image according to claim 1, characterized in that, In step S2, using the hybrid architecture based on the dynamic graph convolutional network and the Transformer to segment the point cloud data specifically includes: Step S2-1. Perform preprocessing on the point cloud data. Step S2-2. Normalize the preprocessed coordinate and sonar reflection intensity data. Step S2-3. Select neighborhoods based on the joint measurement of point cloud geometric distance and feature similarity, and construct a dynamic neighborhood set. Step S2-4. Extract and fuse geometric features and modal features of the point cloud data. Step S2-5. Based on the neighborhood set, calculate neighborhood weights for local feature aggregation, and perform weighted summation of neighborhood features. Step S2-6. Obtain global features through max pooling, and predict class probabilities through a fully connected layer. Step S2-7. Combine local enhanced features and global features through an energy function, perform conditional random field optimization, classify the final features of each point, output the class probability to which the point belongs, and select the class with the highest probability as the final segmentation result for each point.

5. A three-dimensional reconstruction method for underwater targets based on three-dimensional imaging sonar images according to claim 1, characterized in that, The dynamic neighborhood set constructed in step 2-3 is: N(i) = {j | d ij belongs to the first K smallest values} Among them, d ij represents the comprehensive distance, is the feature distance, is the geometric distance, α is a learnable parameter optimized by backpropagation, and K is a set value; The neighborhood weight in step S2-5 is: Among them, f i is the quadratic coding form of the geometric and modal fusion features, exp is the natural exponential function, and LeakyReLU is the activation function; The energy function in step 2-7 is: Among them, the unary potential energy is provided by the predicted probability p i The binary potential energy where λ, θ, and δ are set parameters.

6. The three-dimensional reconstruction method of underwater targets based on three-dimensional imaging sonar images according to claim 1, characterized in that, The step 3 specifically includes: Step 3-1: Extract key feature points from the target point cloud data, describe the feature points using feature descriptors based on geometric characteristics, and then use feature matching algorithms to match the feature points in the multi-view point cloud data to generate corresponding feature point pairs and form a preliminary matching relationship; Step 3-2: Based on the feature point matching results, calculate the relative pose and position information of the multi-view point cloud data, including the rotation matrix and translation vector; Step 3-3, use the initial pose for global optimization, further optimize the camera parameters and 3D point coordinates through the bundle adjustment algorithm to minimize the reprojection error; In step 3-4, the multi-view point cloud data is jointly reconstructed based on the optimized posture and position information to generate a sparse 3D point cloud and remove the mismatched feature points.

7. A three-dimensional reconstruction method for underwater targets based on three-dimensional imaging sonar images according to claim 1, characterized in that In step 4, the sparse point cloud is densified using a patch-based multi-view dense reconstruction method, which includes: Step 4-1, normalize the viewing angle parameters of the multi-view point cloud; Step 4-2: Generate an initial facet set based on the sparse point cloud. The facet initialization parameters include: the 3D coordinates of the facet center point, the facet normal vector calculated from the local geometric relationship of the sparse point cloud, and a view set indicating the views in which the facet is visible. Step 4-3: Starting from the initial patch, gradually expand to the neighboring points to generate new candidate patches. By calculating the projection consistency between the new candidate patches and the multi-view images, the patches that meet the reconstruction conditions are screened out. In step 4-4, the generated candidate facet set is globally optimized. The normal vector and position of each facet are optimized by jointly constraining the geometric and intensity consistency of multiple views. The redundant faces are eliminated by calculating the similarity of the overlapping areas of the faces. In steps 4-5, dense point cloud data is extracted from the optimized facet set, including the three-dimensional coordinates and normal vector of each point, as the final dense point cloud.

8. A three-dimensional reconstruction method for underwater targets based on three-dimensional imaging sonar images according to claim 1, characterized in that, The Poisson surface reconstruction method is used to perform surface fitting and optimization on dense point clouds, including: Step 5-1, extract the normal vector of each point from the dense point cloud data, and use the estimation method based on the point cloud neighborhood to estimate the normal vector of the point; Step 5-2: Use the spatial position and normal vector of the point cloud to construct the Poisson equation. In the Poisson equation, the density function introduces the reflection intensity weight; Step 5-3: discretize the Poisson equation into a system of linear equations, construct a multi-resolution grid through octree hierarchical construction, and numerically solve the system of equations using an iterative solution method to generate a preliminary reconstructed grid model; In step 5-4, the initially reconstructed mesh model is optimized by surface smoothing to eliminate noise and artifacts. In the surface optimization stage, the smoothing weight is adjusted according to the direction of the normal vector to retain sharp edges.

9. A three-dimensional reconstruction method for underwater targets based on three-dimensional imaging sonar images according to claim 1, characterized in that, The normal vector of the estimated point in step 5-1 is: In the formula, is the set of neighborhood points of point P i , N i is the estimated normal vector of the point, and N' i is the normal vector of the extracted point; The reflection intensity weight in step 5-2 is: Δ·v=ρ Wherein, v is a vector field generated by a normal vector field; ρ is a distribution density function of the point cloud in three-dimensional space, generated from the position of the point cloud and the normal vector weight, ρ(x, y, z) = ∑(i) ω i ·s i ·δ(x - x i , y - y i , z - z i ), where ω i is the point cloud weight, s i is the point cloud reflection intensity, and δ is the three-dimensional Dirac function.

10. An underwater target three-dimensional reconstruction system based on three-dimensional imaging sonar images for implementing the method according to any one of claims 1-9, characterized in that, include: A data acquisition unit, which acquires sonar image data of the underwater area where the target is located through a three-dimensional imaging sonar device, wherein the sonar image includes point cloud data of the target area and its reflection intensity information; Point cloud segmentation unit, which uses a hybrid architecture based on dynamic graph convolutional network and Transformer to segment point cloud data, extracts local geometric features through a dynamic graph structure, captures global context dependencies by combining self-attention mechanisms, and extracts the point cloud of the target area; 3D reconstruction module, which applies the structure from motion method to the target point cloud data for sparse reconstruction of the point cloud to obtain a sparse 3D structure, uses a patch-based multi-view dense reconstruction method to densify the sparse point cloud, generates dense point cloud data, and uses the Poisson surface reconstruction method to perform surface fitting and optimization on the dense point cloud to generate the target 3D model.

Images