Underwater target volume measurement method, device and equipment based on three-dimensional reconstruction, and medium

Abstract

The application discloses a kind of based on three-dimensional reconstruction underwater target volume measurement method, device, equipment and medium, belong to photogrammetry, underwater image processing and point cloud volume calculation cross technical field, comprising: by video acquisition equipment to underwater target video data acquisition, obtain the original image with picture format data;Underwater image enhancement method model based on depth estimation is carried out image enhancement to original image, obtain underwater enhanced image;Based on underwater enhanced image, reconstruct the real scene containing target;Based on three-dimensional reconstruction point cloud, segmentation obtains the target to be measured, obtains the volume parameter of underwater target.The application utilizes three-dimensional reconstruction technology, restores real target structure, improves the measurement accuracy of underwater target defect volume, provides reference standard for later repair, reduces maintenance cost, and guarantees the operation safety of hydraulic structure.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary technical field of photogrammetry, underwater image processing and point cloud volume calculation, and more specifically, it relates to a method, device, equipment and medium for measuring the volume of underwater targets based on three-dimensional reconstruction. Background Technology

[0002] The rapid development of underwater robots has greatly promoted the automation of inspection equipment and the intelligentization of inspection and identification, making defect detection in large hydraulic structures possible. In measurement tasks involving non-cooperative targets in space and underwater, close-range reconstruction and measurement of the target are key technologies. Because non-cooperative targets cannot provide effective cooperative information, and lack communication response mechanisms or other active sensors, other robots cannot identify or locate such targets through electronic interrogation or signal transmission. When the target's motion state and spatial structure are unknown, 3D-scale detection and reconstruction of the target using vision and other methods, along with precise measurement of its pose, provides the conditions for further operations.

[0003] However, most research still focuses on target detection and defect identification, lacking quantitative analysis of defect targets, especially in the area of ​​precise volume measurement. Current technologies mostly rely on two-dimensional image annotation or region recognition, failing to provide the true geometric parameters required for repair work, thus hindering the scientific rigor and efficiency of engineering decisions.

[0004] Existing underwater 3D reconstruction methods have two major problems: First, the image enhancement stage uses traditional histogram equalization and Retinex algorithms, which do not integrate depth information and cannot adapt to the differences in light attenuation in different underwater depth regions (such as the color shift and blurring differences between shallow and deep water), resulting in a point cloud distortion rate of more than 15%. Second, the volume measurement stage relies on manual segmentation of the target point cloud, which cannot automatically remove background planes (such as pool bottoms and pipe walls) and outlier noise points, resulting in low measurement efficiency (more than 30 minutes for single target measurement) and large errors (volume measurement error exceeds 8%), making it difficult to meet the rapid quantitative requirements for the repair of defects in hydraulic structures.

[0005] Therefore, how to effectively and accurately measure the volume of underwater targets is a problem that urgently needs to be solved. Summary of the Invention

[0006] In view of the shortcomings of the above or existing technologies, the present invention proposes a method, device, equipment and medium for measuring the volume of underwater targets based on three-dimensional reconstruction. The present invention can improve the measurement accuracy of the volume of defects in underwater targets, reduce maintenance costs and ensure the operational safety of hydraulic structures.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a method for measuring the volume of an underwater target based on three-dimensional reconstruction, comprising:

[0009] Video data is collected from underwater targets using video acquisition equipment to obtain raw images with image format data.

[0010] The underwater image enhancement method based on depth estimation enhances the original image to obtain an enhanced underwater image.

[0011] Based on underwater enhanced images, a realistic scene containing the target is reconstructed;

[0012] Based on the 3D reconstructed point cloud, the target to be measured is segmented and obtained, and the volume parameters of the underwater target are obtained.

[0013] As a further technical solution of the present invention, the underwater image enhancement method model based on depth estimation enhances the original image to obtain an enhanced underwater image; specifically including:

[0014] Introduce depth information based on the visual Transformer and U-shape Transformer;

[0015] While correcting the color cast of underwater images, local enhancement is performed on the details;

[0016] Depth information of underwater images is obtained through a monocular depth estimation algorithm without the need for an additional depth sensor, and the RGB three channels and the depth channel are fused at the input layer.

[0017] The convolution generator receives four-channel input and extracts multi-scale features containing depth information through successive convolution and downsampling operations.

[0018] As a further technical solution of the present invention, the fusion of the RGB three channels and the depth channel at the input layer specifically includes:

[0019] Channel-level multi-scale feature fusion receives output features from the convolutional generator, including linear projection, positional encoding, and channel multi-head attention modules. It learns the relationship between the RGB three channels and the depth channel through a channel self-attention mechanism, providing enhanced feature representations for the decoder.

[0020] Spatial-level global feature modeling is used to process the encoder's low-level features, including linear projection and positional encoding. It processes serialized features through the Transformer layer and uses a spatial self-attention mechanism to capture the global context and the interrelationships between deep regions.

[0021] The decoder receives the features processed by CMSFFT and SGFMT, reconstructs the enhanced image through upsampling and feature fusion, integrates multi-scale feature information using skip connections, and finally outputs the enhanced RGB image.

[0022] As a further technical solution of the present invention, the channel-level multi-scale feature fusion adopts an adversarial training mechanism, wherein the adversarial training mechanism is based on a generative adversarial network, the discriminator receives two sets of inputs, an enhanced image and a ground reference image, and trains to distinguish between real and fake images by comparing the two sets of samples, the enhanced image and the ground reference image.

[0023] As a further technical solution of the present invention, the reconstructing of a real scene containing the target based on underwater enhanced images specifically includes:

[0024] Sparse reconstruction is performed using incremental SFM (Structure from Motion).

[0025] Evaluation and verification are performed through multi-view geometric consistency verification and bidirectional depth consistency verification.

[0026] Dense point cloud files are generated by weighted calculation and fusion. The weighting factors in the weighted calculation include depth confidence weight, viewing angle weight, and resolution weight. The total weight is the product of the depth confidence weight, viewing angle weight, and resolution weight.

[0027] As a further technical solution of the present invention, the sparse reconstruction performed by incremental SFM specifically includes:

[0028] A pair of images is randomly selected for the initial reconstruction model. The fundamental matrix and camera intrinsic matrix are calculated by feature point matching, and the fundamental matrix is ​​converted into the intrinsic matrix.

[0029] The initial reconstruction model is optimized by using an adaptive threshold selection strategy based on the anti-contrast method.

[0030] The essential matrix is ​​decomposed into the camera's rotation matrix and translation vector, and the initial camera pose is recovered based on the rotation matrix and translation vector;

[0031] The initial camera pose is optimized by solving the PnP problem using the LM algorithm.

[0032] Secondly, the present invention provides a volume measurement device for underwater targets based on three-dimensional reconstruction, comprising:

[0033] The image acquisition module acquires video data of underwater targets through video acquisition equipment to obtain raw images with image format data.

[0034] The underwater image enhancement module uses a depth estimation-based underwater image enhancement method model to enhance the original image and obtain an enhanced underwater image.

[0035] The 3D reconstruction module reconstructs a realistic scene containing the target based on underwater enhanced images;

[0036] The volume measurement module, based on 3D reconstructed point cloud, segments and acquires the target to be measured, and obtains the volume parameters of the underwater target.

[0037] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0038] At least one processor, and a memory communicatively connected to said at least one processor;

[0039] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform a method for measuring the volume of an underwater target based on three-dimensional reconstruction.

[0040] Fourthly, the present invention provides a computer-readable storage medium storing a computer program for enabling a processor to implement a method for measuring the volume of an underwater target based on three-dimensional reconstruction.

[0041] The beneficial effects of this invention are as follows:

[0042] This invention provides data support for subsequent analysis and processing by acquiring raw degraded images of underwater targets; based on underwater degraded images, depth information can be estimated and incorporated into the enhancement process without additional depth sensors through channel-level multi-scale feature fusion and spatial-level global feature modeling, and adaptive processing is performed on the attenuation characteristics of different depth regions.

[0043] This invention is based on enhanced images and uses incremental SFM for sparse reconstruction. It designs a reconstruction initialization strategy based on the anti-contrast method and avoids manually setting a fixed threshold through adaptive threshold selection. At the same time, it optimizes the PnP problem solving process, combines the advantages of gradient descent and Gauss-Newton methods, and dynamically adjusts the damping factor to effectively reduce reprojection errors.

[0044] In the dense reconstruction stage, this invention improves the depth map calculation method by using a planar assumption model, multi-view frame expansion, and adaptive view weight selection strategy to enhance the accuracy of depth estimation. It also designs a geometric consistency verification mechanism based on bidirectional depth consistency check and multi-view spatial consistency check, and fuses the weights of depth confidence, viewing angle, and resolution to achieve high-precision point cloud generation.

[0045] This invention is based on 3D reconstructed point clouds. It uses the RANSAC plane fitting algorithm to segment point cloud files, iteratively filters out the interior points of the background plane, eliminates the background plane and retains the target point cloud, performs noise reduction processing on the segmented point cloud, simplifies the point cloud data by voxel downsampling, and performs triangulation to construct a triangular mesh surface, and achieves accurate volume calculation of the target object by accumulating the projected triangular prism volume of all triangular meshes.

[0046] This invention utilizes three-dimensional reconstruction technology to restore the true target structure, improve the measurement accuracy of underwater target defect volume, provide a reference standard for subsequent repair, reduce maintenance costs, and ensure the operational safety of hydraulic structures. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 The flowchart of the underwater target volume measurement method based on three-dimensional reconstruction provided by the present invention;

[0049] Figure 2 This invention provides a multi-scale feature network architecture diagram that includes depth information.

[0050] Figure 3 The flowchart for target reconstruction based on underwater enhanced image provided by this invention;

[0051] Figure 4 This invention provides a schematic diagram of solving PnP using the Levenberg-Marquardt algorithm.

[0052] Figure 5 The depth map fusion process provided by this invention uses point cloud representation to illustrate the fusion process;

[0053] Figure 6a This is a Delaunay triangulation diagram provided by the present invention;

[0054] Figure 6b The Voronoi diagram provided by this invention;

[0055] Figure 7 A three-dimensional spatial projection diagram is provided for an embodiment of the present invention;

[0056] Figure 8 These are raw underwater images collected in an embodiment of the present invention.

[0057] Figure 9This is a diagram illustrating the process of modeling raw underwater images acquired in an embodiment of the present invention.

[0058] Figure 10a This is a diagram illustrating the point cloud file loading and processing in an embodiment of the present invention.

[0059] Figure 10b This is a diagram showing the segmentation process of the target detection object in an embodiment of the present invention;

[0060] Figure 10c This is a diagram showing the removal of planes and outlier points in an embodiment of the present invention;

[0061] Figure 10d This is a diagram showing the triangulation measurement volume processing in an embodiment of the present invention;

[0062] Figure 11 In this embodiment of the invention, Matlab software was used to draw the point cloud heat map of the underwater depression target.

[0063] Figure 12a Image enhancement and reconstruction results for underwater depression targets;

[0064] Figure 12b Image showing the volume measurement results of the underwater depression target;

[0065] Figure 13 The structural diagram of the underwater target volume measurement device based on three-dimensional reconstruction provided by the present invention. Detailed Implementation

[0066] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0067] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0068] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "comprising" or "including," and similar terms used in this disclosure, mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, but do not exclude other elements or objects. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0069] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that mutually excludes other embodiments. It should be noted that the embodiments of the present invention can be applied to any applicable scenario.

[0070] Example 1

[0071] See Figure 1 This invention provides a method for measuring the volume of underwater targets based on three-dimensional reconstruction, comprising:

[0072] Step 101: Collect video data of the underwater target using a video acquisition device to obtain the original image with image format data;

[0073] Step 102: The underwater image enhancement method model based on depth estimation is used to enhance the original image to obtain an underwater enhanced image;

[0074] Step 103: Based on the underwater enhanced image, reconstruct the real scene containing the target;

[0075] Step 104: Based on the 3D reconstructed point cloud, segment and obtain the target to be measured, and obtain the volume parameters of the underwater target.

[0076] This embodiment is applicable to improving the accuracy of underwater target volume measurement. The method can be executed by an underwater robot equipped with an underwater camera. The underwater target volume measurement system can be implemented in software and can be configured on a server, computer, or mobile terminal, such as a mobile phone or tablet computer.

[0077] In this embodiment, the detected image is data in image format obtained by acquiring video data of an underwater target through a video acquisition device. The form of the detected image includes, but is not limited to, video data and still images. The video data may include multiple video frames, where each video frame can be a still image of the video constituting the target pipe. The detected image of the underwater target can be acquired by a video acquisition device, which includes, but is not limited to, a Closed Circuit Television (CCTV) pipe endoscope and a Quick View (QV) device.

[0078] In step 102, the underwater image enhancement method model based on depth estimation enhances the original image to obtain an enhanced underwater image; specifically, it includes:

[0079] Introduce depth information based on the visual Transformer and U-shape Transformer;

[0080] While correcting the color cast of underwater images, local enhancement is performed on the details;

[0081] Optimize multi-stage feature fusion in images to improve image clarity, contrast, and color restoration.

[0082] Depth information of underwater images is obtained through monocular depth estimation algorithms (such as ResNet-based depth prediction models) without the need for additional depth sensors (such as LiDAR or binocular cameras), and the RGB three channels are fused with the depth channel at the input layer.

[0083] The convolution generator receives four-channel input and extracts multi-scale features containing depth information through successive convolution and downsampling operations.

[0084] In this embodiment, the underwater image enhancement method model based on depth estimation introduces depth information on top of the U-shape Transformer and performs specific optimizations for the underwater image enhancement task. Through depth estimation, depth information is integrated into the enhancement process, and adaptive processing is performed to address the attenuation characteristics of different depth regions. Compared to the original model's three-channel input, this invention uses a four-channel input, directly fusing the RGB three channels and the depth channel at the input layer. The convolutional generator receives the four-channel input and extracts multi-scale features through continuous convolution and downsampling operations. All multi-scale features contain depth information. The specific network architecture is as follows... Figure 2 As shown.

[0085] In this embodiment of the invention, the RGB three channels and the depth channel are fused at the input layer; specifically, this includes:

[0086] The Channel-wise Multi-scale FeatureFusion Transformer (CMSFFT) receives output features from the convolution generator and includes linear projection, positional encoding, and channel multi-head attention modules. It learns the relationship between the RGB three channels and the depth channel through a channel self-attention mechanism, paying particular attention to severely degraded color channels to provide enhanced feature representations for the decoder.

[0087] Spatial-wise Global Feature Modeling Transformer (SGFMT) is used to process low-level features of the encoder, including linear projection and positional encoding. It processes serialized features through Transformer layers and uses spatial self-attention mechanism to capture global context and the interrelationships of deep regions.

[0088] The decoder receives the features processed by CMSFFT and SGFMT, reconstructs the enhanced image through upsampling and feature fusion, integrates multi-scale feature information using skip connections, and finally outputs the enhanced RGB image. Since the depth information has been processed at this stage, it does not appear in the final output.

[0089] In this embodiment of the invention, the channel-level multi-scale feature fusion adopts an adversarial training mechanism, which is based on a generative adversarial network. The discriminator receives two sets of inputs: an enhanced image and a ground truth reference image. It trains to distinguish between real and fake images by comparing the two sets of samples.

[0090] In designing the loss function, to fully utilize the color representation range of the LAB and LCH color spaces and accurately describe color saturation and brightness, a multi-color space loss function combining RGB, LAB, and LCH color spaces is adopted. Let the input image be x, the reference image be y, and the enhanced image be... The enhanced image is then represented in RGB space as: In LAB space, it is represented as In LCH space, it is represented as The same applies to the representation of the reference image y.

[0091] The loss functions for RGB, LAB, and LCH spaces are expressed as follows:

[0092] ;

[0093] ;

[0094] ;

[0095] Among them, L RGB L LAB It primarily measures the color difference between the A and B channels of an image, L LCH This mainly focuses on differences in brightness, contrast, and hue of the image.

[0096] In addition, a generative adversarial network loss function is introduced. This can further enhance the generator's learning ability. By playing a game between the generator G and the discriminator D, the generated images more closely resemble the distribution of real images, which can be represented as:

[0097] ;

[0098] To ensure that the enhanced image is consistent with the reference image in semantic content and structure, a perceptual loss function is introduced here. With structural similarity loss function The perceptual loss function can preserve the complex textures and structures of underwater scenes while repairing color distortion and avoiding over-sharpening and processing artifacts; the structural loss function can ensure that underwater objects such as fish, corals, and seabed topography maintain their structural integrity during the enhancement process.

[0099] The total loss function of the network can be expressed as:

[0100] ;

[0101] in, The hyperparameters were set to 100, 0.001, 1, 0.1, and 100 respectively in the experiment. The perceptual loss and structural similarity loss had the highest weights to ensure the image structure and semantic content during training.

[0102] See Figure 3 In step 103, based on the underwater enhanced image, a realistic scene containing the target is reconstructed; specifically including:

[0103] Step 131: Sparse reconstruction is performed using incremental SFM (Structure from Motion).

[0104] Step 132: Evaluation and verification are performed through multi-view geometric consistency verification and bidirectional depth consistency verification;

[0105] Step 133: Generate a dense point cloud file by weighted calculation and fusion.

[0106] In step 131, incremental SFM is used for sparse reconstruction; specifically, it includes:

[0107] A pair of images is randomly selected for the initial reconstruction model. The fundamental matrix and camera intrinsic matrix are calculated by feature point matching, and the fundamental matrix is ​​converted into the intrinsic matrix.

[0108] The initial reconstruction model is optimized by using an adaptive threshold selection strategy based on the anti-contrast method.

[0109] The essential matrix is ​​decomposed into the camera's rotation matrix and translation vector, and the initial camera pose is recovered based on the rotation matrix and translation vector;

[0110] The initial camera pose is optimized by solving the PnP problem using the LM algorithm.

[0111] In this embodiment of the invention, the 3D reconstruction of underwater targets can be divided into two main steps: sparse reconstruction and dense reconstruction. In the sparse reconstruction stage, incremental SFM is selected as the basic method for sparse reconstruction. The incremental SFM process can be divided into initialization and reconstruction stages. In the initialization stage, a pair of images is randomly selected for initial reconstruction. The fundamental matrix and camera intrinsic parameter matrix are calculated through feature point matching, and then the fundamental matrix is ​​converted into the essential matrix. In the reconstruction stage, the essential matrix is ​​further decomposed into the camera rotation matrix and translation vector, and finally the initial camera pose is restored based on the calculation results.

[0112] In the estimation of the fundamental matrix and homography matrix, the feature matching data usually contains noise (outside points). Generally, a fixed global threshold is used to distinguish between inside and outside points. However, the noise distribution of different datasets is significantly different. A fixed threshold may lead to a decrease in the estimation accuracy of the model. This invention adopts an adaptive threshold selection strategy based on anti-comparison discrimination to optimize the model accuracy.

[0113] The opposition-comparison method follows the Helmholtz principle, the core idea of ​​which is that when observed geometric biases significantly exceed the expected distribution of random noise, these biases are more likely to reflect the true geometric structure than accidental noise. During model estimation, the opposition-comparison method uses statistical inference to determine whether the geometric model is generated by noise, adaptively selects the optimal threshold to avoid manually fixing the threshold, and finally calculates the number of misjudgments (NMA) as a measure of model reliability. Under the opposition-comparison framework, NMA is calculated as follows:

[0114]

[0115] When the value of NMA(M) is less than or equal to 1, model M can be considered a reasonable matching model. The calculation of NMA(M,k) is as follows:

[0116]

[0117] Where M is the currently fitted geometric model, k is the number of assumed interior points, n is the total number of feature matching points, and the minimum number of sample points required to calculate the model. The fundamental matrix has 7 points, and the homography matrix has 4 points. The number of models that can be computed given a set of sample points. For all points, the k-th smallest error value to the model. d represents the probability that a randomly matched point falls within the error range, and d is the dimension of the error.

[0118] The core objective of the contrast-based discrimination method is to find the optimal model M. This process involves calculating the error at n corresponding points. The computational complexity of sorting is O(n log n). That is, each element needs to be processed. This comparison operation. When When the number of model combinations exceeds 2, the number of combinations grows exponentially, necessitating the use of the Random Sample Consensus (RANSAC) algorithm for random sampling. The contrast-based discriminant method is embedded in the initialization phase of the incremental SFM workflow: when calculating the fundamental and homography matrices, the contrast-based discriminant method further selects interior points on this subset based on the initial matching points, narrowing the parameter search space and thus improving the stability and accuracy of the estimation; during camera pose estimation, the contrast-based discriminant method provides a confidence estimate of the current view, based on which the reprojection error of new points during triangulation can be determined, ensuring the accuracy of pose estimation.

[0119] For pose optimization, the Levenberg-Marquardt (LM) algorithm is used to solve the PnP problem, as illustrated in the diagram below. Figure 4 As shown.

[0120] The core of the PnP problem lies in solving the camera pose, i.e., the rotation matrix R and the translation vector t, such that the 3D point... With two-dimensional observation points The reprojection error between the two solutions is minimized. The PnP method can only provide an initial solution and cannot guarantee the optimal solution. The reprojection error is defined based on R and t of the initial solution as:

[0121] ;

[0122] This invention minimizes the sum of squares of all errors, thus obtaining the optimal solution to the PnP problem. However, the sum of squares of all errors is a nonlinear function and cannot be solved directly. Therefore, this invention introduces LM optimization to further reduce the errors.

[0123] The LM algorithm is a nonlinear least squares optimization algorithm that combines the advantages of gradient descent and Gauss-Newton's method. To construct the LM iterative equations, the Jacobian matrix needs to be calculated. For the sake of simplified computation, this invention uses the Rodrigues formula to reduce the dimensionality of the rotation matrix R of the fundamental solution, transforming it from a nine-element matrix into a three-dimensional rotation vector. The elements of the Jacobian matrix are the error function with respect to the rotation vector. The partial derivatives of the translation vector t can be expressed as:

[0124] ;

[0125] Each point contributes one The submatrix of the Jacobian matrix is ​​obtained by concatenating all point errors into a complete Jacobian matrix. After obtaining the Jacobian matrix, the update equation for the LM method can be constructed:

[0126] ;

[0127] in, For approximate second-order information, Let I be the damping factor, and I be the identity matrix, controlling the behavior of the optimization process: when When it is large, Approaching the identity matrix makes the equation update direction resemble gradient descent; when When smaller, This dominates, making the equation update direction closer to the Gauss-Newton method. Pose increment It is a six-dimensional vector, consisting of rotation increments and translation increments, which can be obtained by Gaussian elimination.

[0128] The new rotation matrix R and the three-dimensional rotation vector are obtained through the update equation. Based on this, a new reprojection error can be obtained, expressed as follows:

[0129] ;

[0130] in, and This represents the camera rotation vector at the k-th and k+1-th iterations. and It represents the values ​​of the reprojection error function at different iteration steps; damping factor. Dynamically adjustable: When At that time, explain the updated camera pose. Reducing the reprojection error will decrease To accelerate the convergence speed; When this happens, it means the updated pose increases the error, which will improve... To avoid divergence, continuous iterative comparison is used. and The process continues until the error converges (the change in error between iterations is less than a threshold) or the maximum number of iterations is reached, ultimately yielding the desired result. .

[0131] In step 132, multi-view geometric consistency verification is a crucial step in the depth map fusion process. It improves the reliability of depth estimation by evaluating the consistency of depth estimates from different viewpoints, thereby filtering effective 3D points and removing noisy outliers. The geometric constraint verification mechanism provided by this invention comprises two levels: bidirectional depth consistency checking and multi-view spatial consistency checking.

[0132] Bidirectional depth consistency verification ensures consistent depth estimation for the same scene point across different viewpoints by constructing bidirectional projection constraints between views. This is achieved for each pixel p in the reference view r and its estimated depth. ,Will Back-projection into three-dimensional space yields point X:

[0133] ;

[0134] in, Let p be the homogeneous coordinates of pixel p. , and These are the intrinsic parameter matrix, rotation matrix, and translation vector of view r, respectively; projecting X onto the adjacent view i yields the pixel values. Retrieve its corresponding depth value If pixels If the image boundary is not met or there is no valid depth value, the view pair is skipped. The forward depth difference is the difference between the depth of point X projected onto view i and the actual depth map value, expressed as:

[0135] ;

[0136] in, The theoretical depth of point X projected onto view i; using depth values. Back projection can obtain points Calculate the reverse depth difference:

[0137] ;

[0138] in, yes Project the pixels back to view r. This corresponds to the theoretical depth. The difference between the current and reverse depths is less than the relative depth difference threshold. At that time, it can be determined that point X is in the view pair The depths between them are consistent.

[0139] The spatial consistency detection method provided by this invention is similar in core idea to bidirectional depth consistency, but spatial consistency detection directly compares data in three-dimensional space, allowing for a more comprehensive capture of reconstruction errors. The parameter calculations for spatial consistency mainly include the calculation of relative distance and normal vector consistency. Relative distance... Obtained by normalizing the spatial Euclidean distance, it is expressed as:

[0140] ;

[0141] in, Indicate point X and The Euclidean distance between them is normalized to ensure that the evaluation criterion is independent of the observation distance; for the consistency of the normal vector... , can be represented as:

[0142] ;

[0143] in, and Let r and i be the normal vectors estimated from views r and i, respectively; combining the relative distance and normal vector consistency by weights, we obtain the expression for the spatial consistency measure:

[0144] ;

[0145] Among them, the balance weight coefficient Set to 0.5, when Less than the geometric consistency threshold At that time, point X is considered to be consistent with view i.

[0146] In step 133, a dense point cloud file is generated by weighted calculation and fusion, specifically as follows:

[0147] After completing the geometric consistency verification, the dense fusion stage can begin. This stage generates a high-precision point cloud through weighted calculation and fusion. The basic mathematical form of dense fusion is weighted averaging.

[0148]

[0149] in, It is the 3D position estimation vector of point X in view i. The contribution weight for view i.

[0150] The key to dense fusion lies in the weighting factor The calculation, the weighted calculation of the weighting factors includes the depth confidence weight. Observation angle weight and resolution weight Total weight The depth confidence weight is calculated using the normalized cross-correlation coefficient; the observation angle weight is determined by the angle between the camera's line of sight and the target surface normal vector; and the resolution weight is calculated by the ratio of the camera's focal length to the target distance. Depth Confidence Derived from the multi-view matching process, reflecting the accuracy of depth estimation, ranging from [-1, 1], the higher the confidence level, the greater the influence of that view in the fusion process, calculated using normalized cross-correlation:

[0151]

[0152] Where W is the window with its center point as the origin. and These are the pixel positions of the reference view and the adjacent view, respectively; the depth confidence is converted into weights. The conversion result can be expressed as:

[0153] ;

[0154] Where c is the depth confidence level. The mapped version, b is a hard-coded constant set to 0.03 to ensure that the weights remain within a reasonable range in extreme cases, and d is the depth value.

[0155] Observation angle weight Prioritize views of the surface viewed directly. The weight is highest when the camera is directly facing the object's surface, i.e., when the angle between the line of sight and the surface normal is close to zero. The weight then decreases as the viewing angle increases, which can be expressed as:

[0156]

[0157] Where n is the surface normal vector at point X. Let X be the normalized vector pointing from point X to the center of camera i. Parameters for controlling the effect of angle on intensity.

[0158] Image resolution weight Essentially, it reflects the effective resolution of a point in an image, that is, the number of pixels corresponding to a unit distance in the image for a scene. Higher resolution results in higher weight. This weight can be expressed as:

[0159]

[0160] in, Let i be the focal length of camera i. Let be the angle between the line of sight and the surface normal vector. The closer the value is to 1, the more perpendicular the viewing angle is to the surface, and the higher the observation quality. The denominator is the distance from the point to the camera.

[0161] A depth map is a two-dimensional image, where each pixel value represents the distance from the camera to a point in the scene. A point cloud is a model formed by back-projecting the depth map onto three-dimensional space using camera parameters. In a multi-view stereo vision system, each viewpoint generates an independent depth map, each reflecting partial information about the scene observed from different angles. To intuitively illustrate the depth map fusion process, this invention uses point cloud representation, specifically as follows: Figure 5 As shown in the figure, the box on the left represents the point cloud model converted from various depth maps. Through weighted fusion and removal of outlier matching points, an accurate dense point cloud file is finally obtained.

[0162] In step 104, based on the 3D reconstructed point cloud, the target to be measured is segmented and obtained to obtain the volume parameters of the underwater target, specifically:

[0163] Based on the 3D reconstructed point cloud, the point cloud file is segmented using the RANSAC (Random Sample Consensus) plane fitting algorithm. The interior points of the background plane (such as the bottom of a pool or the wall of a tunnel) are selected through iterative filtering to eliminate the background plane and retain the target point cloud. The segmented point cloud is then denoised by voxel downsampling to simplify the point cloud data and triangulation is performed to construct a triangular mesh surface. The volume of the target object is accurately calculated by accumulating the projected triangular prism volumes of all triangular meshes.

[0164] After obtaining the downsampled target point cloud, volume calculation needs to be performed using Delaunay triangulation. The Delaunay diagram is derived from the Voronoi diagram, which segments a given space based on seed points. Assume there exists a point set P in... In space, It is a generated point. For each generated point The Voronoi element V can be represented as:

[0165] ;

[0166] in, Includes all points to the generation point The distance is less than to any generating point The distance x from the point Representing point x and The Euclidean distance between them.

[0167] like Figure 6a and Figure 6b As shown, the Delaunay triangulation and the Voronoi diagram form a strict duality. This duality is manifested in the following ways: the perpendicular bisector of each Voronoi edge corresponds to an edge of the Delaunay triangle; and the center of the circumcircle of each Delaunay triangle corresponds to a vertex of the Voronoi diagram. The core of the Delaunay triangulation is satisfying the empty circle property, that is, in the two-dimensional case, the circumcircle of any triangle does not contain any other sample points. This property can be directly extended to three-dimensional space, where it is required that the circumsphere of each tetrahedron does not contain any other sample points, thus ensuring the geometric optimality of the Delaunay triangulation.

[0168] Based on the above rules, the Delaunay triangulation process is as follows: First, an initial super-triangle containing all sampling points is constructed as the computational domain. Then, an incremental insertion strategy is used to locate each insertion point within the current triangular mesh after the containing triangle, and all adjacent triangles violating the empty circle criterion are deleted. Next, triangular fragments satisfying the Delaunay conditions are reconstructed through local edge flipping operations; this process uses local optimization for iterative verification. Finally, all elements related to the initial super-triangle are removed, outputting a Delaunay triangular mesh that conforms to the empty circle property.

[0169] In a three-dimensional coordinate system, assuming the positions of three points are known, let them be the vertices of a triangle. If the axis represents the vertical height, a triangular prism can be obtained by projecting this triangle onto the xy-plane. Using this principle, projecting all the triangular meshes formed by Delaunay triangulation onto the plane yields a large number of spatial triangular prisms. Superimposing the volumes of all these prisms gives the volume of the object being measured. The traditional method for calculating the volume of a triangular prism is the double integral method, which can be expressed as:

[0170] ;

[0171] in, Let be the projection region of the triangle onto the xy-plane. It is a function of the height of the upper surface. This is a function of the lower surface height.

[0172] The double integration method has high computational complexity. The following section introduces a method to reduce this complexity:

[0173] like Figure 7 As shown, given three points in three-dimensional space ,in The triangle formed by the three points on the base of the triangular prism Projection onto a plane.

[0174] First, define the line segment. With plane The intersection point is Obtained by linear interpolation and Coordinates. Then, calculate the coordinates of the two triangles. and Projected area on the z-constant plane and The volume v below the original triangle can be expressed as:

[0175] ;

[0176] Finally, by and Substituting v, we obtain the volume of a single spatial triangular prism. After obtaining the volume of a single triangular prism, the overall volume of the object to be measured can be expressed as:

[0177] ;

[0178] in, Let be the projected volume of the i-th spatial triangular prism. This algorithm reduces computational complexity from... Down to This significantly improves computational efficiency.

[0179] This invention utilizes three-dimensional reconstruction technology to restore the true target structure, improve the measurement accuracy of underwater target defect volume, provide a reference standard for subsequent repair, reduce maintenance costs, and ensure the operational safety of hydraulic structures.

[0180] Example 2

[0181] See Figure 8 and Figure 9 The image shown is the original underwater image acquired by this invention. This embodiment describes how the image is processed... Figure 7 and Figure 8 The original images were used to perform 3D reconstruction and volume measurement, and a full-process test was conducted. The underwater Q-shaped brick dataset collected in this invention was used, and the specific process is as follows:

[0182] like Figure 10a and Figure 10b As shown, the previously generated 3D point cloud file is loaded into the volume measurement module and displayed using a high-contrast gradient color scheme for easy differentiation during the demonstration. The blue-green portion represents the target measurement point cloud, while the gray portion represents the detected and separated normal planes.

[0183] After obtaining the segmented target point cloud, the outlier threshold k needs to be further adjusted. For large point clouds, the k value is usually larger, generally between 30 and 50, while for small point clouds, the k value can be smaller, generally between 10 and 20. Smaller k values ​​are more sensitive to noise but may identify normal fluctuations as outliers, while larger k values ​​are more robust but may miss some outliers, resulting in incorrect triangulation. Therefore, the k value needs to be adjusted from small to large, the results observed, and the optimal value selected. After deleting the background plane and outliers, triangulation can be performed, and the final volume is calculated by summing the results, as follows: Figure 10c and Figure 10d As shown, the underwater measured volume of the Q-shaped brick is 1380.21 cm³. 3 This completes the reconstruction measurement process for underwater protruding targets.

[0184] This invention is also applicable to the 3D reconstruction and measurement of underwater concave targets. Cracked tiles are placed in a water tank, and image data is acquired using a calibrated underwater camera. The volume measurement process for concave targets is consistent with the aforementioned method for handling underwater protruding targets, mainly by transforming the concave problem into a similar protruding problem through coordinate system flipping. To more intuitively represent the reconstruction effect, a point cloud heatmap of the underwater concave target is generated using Matlab software, as shown in the image. Figure 11 As shown in the figure. The yellow-green highlighted areas represent the raised portions of the target depression after it has been flipped. The image enhancement and reconstruction results of the underwater depression target are shown below. Figure 12a As shown, the volume measurement results are as follows: Figure 12b As shown, the measured volume of the crack defect is 4.81 cm³. 3 .

[0185] Example 3

[0186] See Figure 13 The present invention also provides a second aspect, namely, a volume measurement device for underwater targets based on three-dimensional reconstruction, comprising:

[0187] The image acquisition module 201 acquires video data of underwater targets through video acquisition equipment to obtain raw images with image format data.

[0188] The underwater image enhancement module 202 enhances the original image based on the underwater image enhancement method model of depth estimation to obtain an underwater enhanced image.

[0189] The 3D reconstruction module 203 reconstructs a realistic scene containing the target based on underwater enhanced images;

[0190] The volume measurement module 204, based on the 3D reconstructed point cloud, segments and acquires the target to be measured, and obtains the volume parameters of the underwater target.

[0191] This invention adopts a modular design architecture, with each module containing several sub-functions. It mainly includes four modules: image acquisition module, underwater image enhancement module, three-dimensional reconstruction module, and volume measurement module. These modules communicate with each other through a signal and slot mechanism to achieve seamless data transmission and efficient collaboration of the processing flow.

[0192] The image acquisition module serves as the foundational support for the system. For data sharing, this module implements cross-module data sharing through the SharedData class and uses PyQt5's signal and slot mechanism for asynchronous communication between modules. File operation assistance is provided by the File Helper class, which deeply encapsulates PyQt5's Q File Dialog component, providing a more semantic and user-friendly interface. Message prompt functionality is implemented through the Message Helper class, uniformly encapsulating PyQt5's QMessage Box component, providing four common user interaction modes: information prompts, warning prompts, error prompts, and confirmation prompts. The method design follows a consistent interface specification, accepting the parent window, title, and message content as parameters, ensuring a consistent appearance and behavior of user prompts throughout the application. The point cloud processing tool is another important component of this module, provided by the Point Cloud Utils class. This class integrates the core functionality of the Open3D library, providing basic algorithms such as point cloud normal estimation and mesh creation, balancing computational efficiency with performance.

[0193] The underwater image enhancement module uses the weight file trained according to this embodiment to enhance underwater images, utilizing image depth information for color correction and illumination equalization. For batch processing, the system supports loading multiple images at once and implements batch enhancement through the `batch_enhance` method, greatly improving the efficiency of large-scale data processing. To avoid interface lag during processing, this invention introduces the `Image Enhancement Thread` class in this module, placing computationally intensive image enhancement tasks in a separate thread. After image processing is complete, the results are returned to the main thread via a predefined signal, achieving secure communication between threads.

[0194] In 3D reconstruction, this module implements a complete reconstruction pipeline based on OpenMVG and Open MVS. The OpenMVG pipeline handles the initial feature extraction, feature matching, and sparse reconstruction, while the Open MVS pipeline handles the later dense reconstruction, mesh generation, and texture mapping. Similar to image enhancement, the 3D reconstruction task is also encapsulated in a separate Reconstruction Thread class, using multi-threaded processing to avoid interface freezes caused by long-term computation. This invention designs four key signals in this thread: update_signal for updating log information, progress_signal for updating the progress bar, preview_signal for updating the preview image, and finished_signal for notifying that reconstruction is complete. This multi-signal design ensures real-time feedback during the reconstruction process, allowing users to obtain the processing status in real time and preview the current effect without waiting for the entire process to complete before viewing the results.

[0195] The volume measurement module, based on the Open3D library, provides a complete solution from raw point cloud data to accurate volume quantization. The entire measurement process is divided into six key steps: point cloud viewing, plane segmentation, coordinate system setting, outlier removal, surface triangulation, and volume calculation. In the field of point cloud processing, visualization is an important means of understanding and analyzing 3D data. This module embeds the Open3D rendering window into the Qt interface, supporting interactive operations such as viewpoint adjustment, scaling, and rotation. It also utilizes the `visualization_timer` to implement an efficient rendering update mechanism, ensuring a smooth and immediate visual experience.

[0196] The various variations and specific examples of the three-dimensional reconstruction-based underwater target volume measurement method in the foregoing embodiments are also applicable to the three-dimensional reconstruction-based underwater target volume measurement device in this embodiment. Through the foregoing detailed description of the three-dimensional reconstruction-based underwater target volume measurement method, those skilled in the art can clearly understand the three-dimensional reconstruction-based underwater target volume measurement device in this embodiment. Therefore, for the sake of brevity, it will not be described in detail here.

[0197] Example 4

[0198] This invention proposes an electronic device, the electronic device comprising:

[0199] At least one processor, and a memory communicatively connected to said at least one processor;

[0200] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the method for measuring the volume of an underwater target based on three-dimensional reconstruction.

[0201] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0202] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and a processor may execute the program instructions to implement the methods of the software programs in the various embodiments of the present invention described above, as well as other desired functions. In addition, depending on the specific application, the electronic device may include any other suitable components.

[0203] Example 5

[0204] This invention proposes a computer-readable storage medium storing a computer program that enables a processor to execute a method for measuring the volume of an underwater target based on three-dimensional reconstruction.

[0205] In addition to the methods and apparatus described above, embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.

[0206] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of the present invention. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0207] Furthermore, embodiments of the present invention may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.

[0208] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0209] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for measuring the volume of underwater targets based on three-dimensional reconstruction, characterized in that, include: Video data is collected from underwater targets using video acquisition equipment to obtain raw images with image format data. The underwater image enhancement method based on depth estimation enhances the original image to obtain an enhanced underwater image. Based on underwater enhanced images, a realistic scene containing the target is reconstructed; Based on the 3D reconstructed point cloud, the target to be measured is segmented and obtained, and the volume parameters of the underwater target are obtained. The underwater enhanced image is used to reconstruct a realistic scene containing the target; Specifically, it includes: Sparse reconstruction is performed using incremental SFM. Evaluation and verification are performed through multi-view geometric consistency verification and bidirectional depth consistency verification. Dense point cloud files are generated by weighted calculation and fusion. The weighting factors for the weighted calculation include depth confidence weight, viewing angle weight, and resolution weight. The total weight is the product of the depth confidence weight, viewing angle weight, and resolution weight. The underwater image enhancement method model based on depth estimation enhances the original image to obtain an enhanced underwater image; specifically, it includes: Introduce depth information based on the visual Transformer and U-shape Transformer; While correcting the color cast of underwater images, local enhancement is performed on the details; Depth information of underwater images is obtained through a monocular depth estimation algorithm, and the RGB three channels and the depth channel are fused at the input layer. The convolution generator receives four-channel input and extracts multi-scale features containing depth information through successive convolution and downsampling operations; The sparse reconstruction using incremental SFM specifically includes: A pair of images is randomly selected for the initial reconstruction model. The fundamental matrix and camera intrinsic matrix are calculated by feature point matching, and the fundamental matrix is ​​converted into the intrinsic matrix. The initial reconstruction model is optimized by using an adaptive threshold selection strategy based on the anti-contrast method. The essential matrix is ​​decomposed into the camera's rotation matrix and translation vector, and the initial camera pose is recovered based on the rotation matrix and translation vector; The initial camera pose is optimized by solving the PnP problem using the LM algorithm.

2. The method for measuring the volume of an underwater target based on three-dimensional reconstruction according to claim 1, characterized in that, The process of fusing the RGB three channels with the depth channel at the input layer specifically includes: Channel-level multi-scale feature fusion receives output features from the convolutional generator, including linear projection, positional encoding, and channel multi-head attention modules. It learns the relationship between the RGB three channels and the depth channel through a channel self-attention mechanism, providing enhanced feature representations for the decoder. Spatial-level global feature modeling is used to process the encoder's low-level features, including linear projection and positional encoding. It processes serialized features through Transformer layers and uses a spatial self-attention mechanism to capture global context and the interrelationships between deep regions. The decoder receives the features processed by CMSFFT and SGFMT, reconstructs the enhanced image through upsampling and feature fusion, integrates multi-scale feature information using skip connections, and finally outputs the enhanced RGB image.

3. The method for measuring the volume of an underwater target based on three-dimensional reconstruction according to claim 2, characterized in that, The channel-level multi-scale feature fusion adopts an adversarial training mechanism, which is based on a generative adversarial network. The discriminator receives two sets of inputs: an enhanced image and a ground truth reference image. It trains to distinguish between real and fake images by comparing the two sets of samples.

4. The method for measuring the volume of an underwater target based on three-dimensional reconstruction according to claim 1, characterized in that, The process of segmenting and acquiring the target based on 3D reconstructed point cloud to obtain the volume parameters of the underwater target specifically includes: The point cloud file is segmented using the RANSAC plane fitting algorithm. The background plane is eliminated by iteratively filtering out the interior points of the background plane and the target point cloud is retained. The segmented target point cloud is denoised by voxel downsampling to simplify the point cloud data and triangulation is performed to construct a triangular mesh surface. The volume of the target object is obtained by summing the projected volumes of the triangular meshes.

5. A volume measurement device for underwater targets based on three-dimensional reconstruction, characterized in that, The method for measuring the volume of an underwater target based on three-dimensional reconstruction as described in any one of claims 1-4 includes: The image acquisition module acquires video data of underwater targets through video acquisition equipment to obtain raw images with image format data. The underwater image enhancement module uses a depth estimation-based underwater image enhancement method model to enhance the original image and obtain an enhanced underwater image. The 3D reconstruction module reconstructs a realistic scene containing the target based on underwater enhanced images; The volume measurement module, based on 3D reconstructed point cloud, segments and acquires the target to be measured, and obtains the volume parameters of the underwater target.

6. An electronic device, characterized in that, The electronic device includes: At least one processor, and a memory communicatively connected to said at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the underwater target volume measurement method based on three-dimensional reconstruction as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that enables a processor to implement the underwater target volume measurement method based on three-dimensional reconstruction as described in any one of claims 1-4.

Images