Underwater domain generalization stereo matching network training method, three-dimensional reconstruction method and device

Abstract

This application provides a training method for a generalized stereo matching network, a 3D reconstruction method, and an apparatus for underwater domains. The method includes the following steps: acquiring a left-eye image and a right-eye image, as well as a true disparity map between the left-eye and right-eye images; constructing a binocular consistency degradation model for the left-eye and right-eye images based on an ocean imaging degradation model, and then inputting the constructed images into a feature extraction network with shared weights to obtain left-eye feature maps and right-eye feature maps; constructing a cost volume based on the left-eye and right-eye feature maps, and outputting an initial disparity map through a cost aggregation network; using the weighted sum of disparity regression loss, stereo contrast feature loss, and stereo selective whitening loss as the total loss, and adaptively weighting the total loss using a quality weight map; updating the parameters of the feature extraction network and the cost aggregation network with the goal of minimizing the total loss. This application achieves highly robust 3D reconstruction in complex underwater environments.

Description

Technical Field

[0001] This application relates to the fields of computer vision and 3D reconstruction technology, specifically to an underwater domain generalized stereo matching network training method, a 3D reconstruction method, and an apparatus. Background Technology

[0002] In underwater environments, binocular vision is susceptible to feature matching distortion due to factors such as water scattering, abrupt changes in illumination, and weak texture. Existing domain-generalized stereo matching methods improve cross-domain robustness through contrastive learning, but they are trained directly on synthetic data without constructing domain-offset samples to address underwater imaging degradation mechanisms. Furthermore, they rely solely on 2D disparity supervision and lack a closed-loop mechanism to inversely constrain disparity estimation from 3D point cloud reconstruction quality, resulting in insufficient accuracy in underwater surface reconstruction. Therefore, how to construct training samples that combine the physical characteristics of underwater imaging and introduce 3D spatial quality feedback is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] In view of this, in order to overcome at least one aspect of the above problems, this application proposes an underwater domain generalized stereo matching network training method, a three-dimensional reconstruction method and device to address the technical problem of decreased matching accuracy and insufficient three-dimensional reconstruction integrity caused by domain offset in existing binocular stereo matching networks in complex underwater environments.

[0004] The underwater domain generalized stereo matching network training method according to the first aspect of this application includes the following steps: Acquire the left and right eye images after epipolar correction, as well as the true disparity map between the left and right eye images; Based on the marine imaging degradation model, the left and right eye images are constructed using binocular consistency degradation and then input into a feature extraction network with shared weights to obtain left and right eye feature maps. Based on the left and right eye feature maps, a cost volume is constructed, and an initial disparity map is output through a cost aggregation network. A three-dimensional point cloud is generated based on the initial disparity map, and a local surface quality assessment is performed on the three-dimensional point cloud to generate a quality weight map that indicates the reconstruction confidence of each region in the disparity map. The disparity regression loss is calculated based on the initial disparity map and the true disparity map, and the stereo contrast feature loss and stereo selective whitening loss are calculated based on the left eye feature map and the right eye feature map. The total loss is obtained by weighting the disparity regression loss, the stereo contrast feature loss, and the stereo selective whitening loss, and then adaptively weighting the total loss using the quality weight map. With the goal of minimizing the total loss, the parameters of the feature extraction network and the cost aggregation network are updated.

[0005] Furthermore, the ocean imaging degradation model includes at least one of water attenuation, backscattering, non-uniform illumination, bright and dark fringes, and local specular reflection; and / or, In the binocular consistency degradation structure, the left eye image and the right eye image share water attenuation parameters, backscattering parameters, and ambient light parameters, and are respectively superimposed with local reflection perturbations and illuminance perturbations related to the viewing angle.

[0006] Further, the stereo contrast feature loss is as follows: taking either the left or right eye feature map as a reference, for each anchor pixel in the reference feature map, a matching pixel is determined as a positive sample in the other side feature map based on the true disparity map, and a non-matching pixel is selected as a negative sample in the other side feature map; the difference in feature distance between the positive sample pair and the negative sample pair is calculated; and / or, The stereo-selective whitening loss is as follows: calculate the covariance matrix of the features output by the feature extraction network, and perform a whitening transformation on the covariance matrix to suppress the covariance components that are sensitive to changes in the left and right eye perspectives.

[0007] Furthermore, the quality weight map is used to perform pixel-level weighting, region-level weighting, or hierarchical weighting on the total loss; and / or, The quality indicators for the local surface quality assessment include at least one of the following: local plane fitting residual, normal vector fluctuation, point cloud porosity, and boundary continuity; and / or, The feature extraction network is ResNet, and the cost aggregation network is an aggregation network based on three-dimensional convolution.

[0008] The underwater domain generalized stereo matching three-dimensional reconstruction method according to the second aspect of this application includes the following steps: Acquire the left and right eye images of the underwater scene and the parameters of the binocular camera, and perform epipolar correction on the left and right eye images; The epipolar-corrected image is input into the feature extraction network and cost aggregation network trained using the method described in the first aspect embodiment, and a disparity map is output. Reconstruct a 3D point cloud based on the disparity map and the binocular camera parameters; The three-dimensional point cloud is quantitatively characterized to obtain information on the location, depth, length, and opening size of the defects.

[0009] Furthermore, the method also includes: optimizing the three-dimensional point cloud, generating a three-dimensional point cloud based on the disparity map, performing local surface quality assessment to generate a quality weight map, using the quality weight map to refine and correct the disparity map, and reconstructing the optimized three-dimensional point cloud based on the corrected disparity map.

[0010] Furthermore, the underwater scene is the surface of an underwater ship hull, the surface of an underwater structure, a dock lighting scene, or a marine weak texture curved surface scene.

[0011] An underwater domain generalized stereo matching network training apparatus according to a third aspect embodiment of this application includes: The data acquisition module is used to acquire the left and right eye images after epipolar correction, as well as the true disparity map between the left and right eye images; The degradation and feature extraction module is used to construct binocular consistent degradation of the left and right eye images based on the ocean imaging degradation model, and then input the constructed images into a feature extraction network with shared weights to obtain the left and right eye feature maps. The cost aggregation module is used to construct a cost volume based on the left and right eye feature maps, and output an initial disparity map through the cost aggregation network; The quality assessment module is used to generate a three-dimensional point cloud based on the initial disparity map, perform local surface quality assessment on the three-dimensional point cloud, and generate a quality weight map to indicate the reconstruction confidence of each region in the disparity map. The loss calculation module is used to calculate disparity regression loss based on the initial disparity map and the true disparity map, and to calculate stereo contrast feature loss and stereo selective whitening loss based on the left eye feature map and the right eye feature map. The weighted update module is used to adaptively weight the total loss by using the weighted sum of the disparity regression loss, the stereo contrast feature loss, and the stereo selective whitening loss, and to update the parameters of the feature extraction network and the cost aggregation network with the goal of minimizing the total loss.

[0012] The underwater domain generalized stereo matching three-dimensional reconstruction apparatus according to the fourth aspect of this application includes: The image acquisition module is used to acquire the left and right eye images of the underwater scene, as well as the parameters of the binocular camera. The epipolar correction module is used to perform epipolar correction on the left and right eye images; The three-dimensional reconstruction module is used to input the epipolar-corrected image into the feature extraction network and cost aggregation network trained by the method described in the second aspect of the present application, output a disparity map, and reconstruct a three-dimensional point cloud based on the disparity map and the binocular camera parameters. The defect characterization module is used to quantitatively characterize the defects in the three-dimensional point cloud, and obtain information on the location, depth, length and opening size of the defects.

[0013] A computer-readable storage medium according to a fifth aspect embodiment of the present application stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in the first aspect embodiment.

[0014] The underwater domain generalized stereo matching network training method of this application solves the problem of insufficient cross-domain generalization ability of existing domain generalized stereo matching networks due to direct training on synthetic data and lack of underwater degradation domain samples by constructing binocular consistency degradation of left and right images based on the marine imaging degradation model. By generating a 3D point cloud based on the initial disparity map and performing local surface quality assessment to generate a quality weight map, it solves the problem of poor matching stability in weak texture regions caused by existing training methods relying only on 2D disparity supervision and lacking 3D spatial geometric constraints. By fusing disparity regression loss, stereo contrast feature loss and stereo selective whitening loss and using the quality weight map for adaptive weighting, it solves the problem that existing training methods are difficult to take into account both the physical characteristics of underwater imaging and 3D spatial geometric constraints. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a training method for an underwater domain generalized stereo matching network according to an embodiment of this application; Figure 2 This is a block diagram of an underwater domain generalized stereo matching network training system according to an embodiment of this application; Figure 3 This is a method for generalized stereo matching and three-dimensional reconstruction of underwater domains according to embodiments of this application; Figure 4 This is a block diagram of an underwater domain generalized stereo matching 3D reconstruction system according to an embodiment of this application.

[0017] Figure Labels Underwater domain generalized stereo matching network training device 200; data acquisition module 210; degradation and feature extraction module 220; cost aggregation module 230; quality assessment module 240; loss calculation module 250; weighted update module 260; Underwater domain generalized stereo matching 3D reconstruction device 400; image acquisition module 410; epipolar correction module 420; 3D reconstruction module 430; defect characterization module 440. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to specific examples and accompanying drawings.

[0019] It should be noted that all uses of "first" and "second" in the embodiments of this application are for the purpose of distinguishing two entities with the same name but different names or different parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of this application. Subsequent embodiments will not explain this in detail.

[0020] like Figure 1 As shown, this application provides a method for training an underwater domain generalized stereo matching network to provide a domain generalized stereo matching model for the 3D reconstruction of underwater targets, including the following steps: Step S110: Obtain the left and right eye images after epipolar correction, and the true disparity map between the left and right eye images.

[0021] Specifically, for binocular measurement tasks on underwater hull curved surfaces or in complex lighting conditions in docks, the left and right eye images of the target scene are first acquired. The image acquisition device is a binocular camera system configured on an underwater robot or fixed inspection platform, and parameters such as camera baseline distance and focal length are obtained in advance through calibration.

[0022] The acquired raw binocular images need to undergo epipolar correction to ensure that the epipolar lines of corresponding spatial points in the left and right images lie on the same horizontal line, thus simplifying the two-dimensional search problem of stereo matching into a one-dimensional search problem. Epipolar correction uses the standard Bouguet algorithm, which reprojects the left and right images using the calibration parameters (rotation matrix and translation vector) of the binocular camera. The corrected left image is denoted as I. L The right eye image is denoted as I. R The polar lines of the two are strictly aligned.

[0023] Simultaneously acquire the true disparity maps D corresponding to the left and right images. gt The true disparity map can be obtained through simulation rendering or structured light scanning, and can be used as a supervision signal for network training.

[0024] Step S120: Based on the ocean imaging degradation model, the left and right eye images are constructed using binocular consistency degradation and then input into a feature extraction network with shared weights to obtain the left and right eye feature maps.

[0025] The interference of the underwater environment on optical imaging mainly stems from the absorption and scattering effects of water on light. Since the diameter of particles in water is typically much larger than the wavelength of visible light, the absorption and scattering effects of water on light are extremely pronounced. During underwater propagation, light reflected from objects is absorbed by the water and scattered at small angles by suspended particles (forward scattering), resulting in image blurring. Simultaneously, light emitted from the light source is directly captured by the camera after being scattered at large angles by impurities in the water (backscattering), leading to a decrease in contrast. Furthermore, different wavelengths of visible light attenuate at different rates in water: red light almost disappears at a depth of 5 meters, while green and blue light travel much farther.

[0026] This embodiment constructs an ocean imaging degradation model based on the Jaffe-McGlamery underwater imaging model, and constructs a binocular consistent degradation domain for the left and right eye images. The model decomposes the underwater imaging process into three components: direct reflection light, forward scattered light, and backscattered light.

[0027] Let the left and right eye images share the water attenuation parameter c(λ), backscattering parameter b, and ambient light parameter A. ∞ The water attenuation parameter c(λ) = α(λ) + b(λ), where α(λ) is the absorption coefficient and b(λ) is the scattering coefficient, both of which are functions of wavelength λ. Based on this physical characteristic, this embodiment applies wavelength-dependent attenuation factors to different wavelength channels.

[0028] The degraded left and right eye images are represented as follows: (1) (2) in, and These represent the degraded left and right eye images, respectively. and These represent the original left eye image and the original right eye image, respectively. The term representing water body attenuation follows the Beer-Lambert law. Indicates the wavelength of light The changing water body attenuation coefficient, and These represent the scene depth corresponding to each pixel in the left and right images, respectively. This represents the ambient light parameters, corresponding to the backscattering components. and These represent the local reflection perturbations and illuminance perturbations related to the observation angle superimposed on the left and right eyes, respectively.

[0029] Although the left and right eye images share the same water attenuation and backscattering parameters, the local reflection perturbation and illumination perturbation are set separately according to the difference in the observation angles of the left and right cameras: (3) (4) in, and These represent the total perturbation superimposed at pixel (x,y) in the left and right eye images, respectively. and These represent the superimposed specular reflection disturbances of the left and right eyes, respectively. and These represent the superimposed non-uniform illumination perturbations in the left and right eyes, respectively.

[0030] Since the left and right cameras observe the same curved target from different angles, the specular reflection and local illumination distribution appear differently in the left and right images. The two perturbations are set independently to simulate the imaging differences of curved targets under different viewing conditions in a real underwater environment.

[0031] The degradation effect includes at least one of the following: water body attenuation, backscattering, non-uniform illumination, bright and dark stripes, and local specular reflection. The intensity of degradation can be adjusted according to the actual water conditions (such as clear seawater, turbid nearshore waters, etc.).

[0032] To ensure the geometric correspondence between the left and right images remains unchanged before and after degradation, the spatial coordinates of pixels are not altered during the degradation process; only the pixel values ​​are subjected to degradation transformation. Therefore, the true disparity map D corresponding to the original image... gt It remains effective even after degradation and can be used as a training supervision signal. This degradation domain construction generates cross-domain stereo matching training samples. .

[0033] In this embodiment, a domain difficulty coefficient δ is also generated based on the degradation intensity. Specifically, a degradation intensity factor is defined: (5) Where, γ j λ represents the intensity coefficient of each degradation effect. j The corresponding weighting factor, δ, ranges from 0 to 1. The larger the value of δ, the more severe the degradation.

[0034] The training strategy is adjusted based on the domain difficulty coefficient δ, and the sampling probability of the training samples is as follows: (6) Wherein, α is the sampling bias coefficient, which makes samples with high degradation levels sampled with a higher probability; The ratio of clean region samples to degenerate region samples in the training batch is: (7) Where, δ bar This represents the average domain difficulty coefficient for all samples in the current batch. The weight coefficient λ in the training loss δ =1+β·δ, which makes samples with high degradation levels have a higher weight in the total loss.

[0035] Degraded left eye image L And right eye image R The input is a feature extraction network with shared weights. This feature extraction network adopts the ResNet architecture and extracts multi-scale features through multi-layered cascaded residual convolutional modules. Shared weights mean that the left and right images use the exact same set of network parameters, ensuring that the features of the left and right images are in the same feature space, providing a consistent basis for subsequent matching.

[0036] The feature extraction process can be represented as: (8) in, For feature extraction networks with shared weights, and These represent the degraded left and right eye images, respectively. θ E F represents the trainable parameters of the network. L and F R These are the feature maps for the left and right eyes, respectively.

[0037] Step S130: Construct a cost volume based on the left and right eye feature maps, and output an initial disparity map through a cost aggregation network.

[0038] Based on the left eye feature map F L and right eye feature map F R Constructing the cost volume. In this embodiment, group correlation operations are used to construct the cost volume. Specifically, F... L and F R Grouping along the channel dimension, calculating the relevant value for each group, and then concatenating the relevant values ​​from each group to form the 4D cost volume: (9) Where G is the number of groups and d is the disparity candidate value. ·, · This indicates the inner product operation.

[0039] The cost volume is input into a 3D convolutional aggregation network. Multiple cascaded 3D convolutional layers are used to regularize and aggregate the cost volume, eliminating matching noise and ambiguity. Finally, SoftArgmin regression is used to output the initial disparity map D. init : (10) Where Cagg Let σ be the aggregated cost volume, and σ be the Softmax function.

[0040] Step S140: Generate a three-dimensional point cloud based on the initial disparity map, perform local surface quality assessment on the three-dimensional point cloud, and generate a quality weight map to indicate the reconstruction confidence of each region in the disparity map.

[0041] Based on the initial disparity map D init 3D point cloud reconstruction is performed using binocular camera parameters. 3D point cloud reconstruction is based on the principle of triangulation. For the corrected binocular image, the relationship between disparity d and depth Z is: (11) Where f is the camera focal length (in pixels), and B is the binocular baseline distance. For any pixel (u, v) on the image plane, its three-dimensional coordinates (X, Y, Z) in the camera coordinate system are: (12) Among them (c u c v () represents the principal point coordinates of the camera. The 3D coordinates are calculated pixel-by-pixel to generate a dense 3D point cloud P. (13) This formula represents the dense 3D point cloud set P calculated pixel-by-pixel from the initial disparity map, where each point... For a given pixel in the image, its three-dimensional coordinates are recovered from the disparity value using triangulation principles. This point cloud set is then used for subsequent local surface quality assessment.

[0042] After obtaining the initial 3D point cloud, a local surface quality assessment is performed on the 3D point cloud. The goal of the quality assessment is to identify regions with high reconstruction quality and regions with problems (such as holes, boundary breaks, surface discontinuities, etc.) in the point cloud, providing a basis for the adaptive weighting of the training loss in subsequent training.

[0043] Specifically, the 3D point cloud is divided into local neighborhoods. For each point p in the point cloud... i Take its local neighborhood N(p) i (e.g., neighboring points within radius r or K-nearest neighbors), perform surface fitting within the local neighborhood. The local surface is fitted using the moving least squares method: (14) in, This represents the local surface function obtained through fitting. Indicates the first The fitting coefficients of the basis functions Let represent the j-th basis function, m represent the total number of basis functions, and (x, y) represent the spatial coordinates within the local neighborhood. Based on the surface fitting results, calculate the following quality indices: Local plane fitting residuals: These are calculated as the sum of squared distances from each point in the local neighborhood to the fitted surface, reflecting the flatness of the local surface and the accuracy of the fitting. Larger residuals indicate less stable disparity estimation in that region.

[0044] Normal vector fluctuation: This calculates the standard deviation of the angle between the normal vector at each point in the local neighborhood and the normal vector of the fitted surface, reflecting the consistency of the local surface orientation. A greater fluctuation indicates a more chaotic geometric structure in that region.

[0045] Point cloud porosity: The ratio of point density in a local neighborhood to the theoretical density, reflecting whether there is missing data in that area. The higher the porosity, the less complete the information in that area.

[0046] Boundary continuity: Detect the existence of discontinuous boundaries in the point cloud by calculating the gradient change rate of the local surface.

[0047] Based on the comprehensive evaluation of the above quality indicators, a quality weight map W is generated that corresponds one-to-one with the spatial location of the disparity map. q The weight value W at each position in the quality weight diagram. q (u, v) indicates the reconstruction confidence of the 3D point cloud region corresponding to the pixel. The higher the weight value, the better the reconstruction quality, and the lower the weight value, the worse the reconstruction quality (such as when it is located in a hole, boundary or reflective area).

[0048] Step S150: Calculate the disparity regression loss based on the initial disparity map and the true disparity map, and calculate the stereo contrast feature loss and stereo selective whitening loss based on the left eye feature map and the right eye feature map.

[0049] After obtaining the initial disparity map D init Quality weighting chart W q and left eye feature map F L and right eye feature map F R Then, the three training losses are calculated.

[0050] Disparity regression loss: based on the initial disparity map D init Parallax map D gt The disparity regression loss is calculated using the smoothed L1 loss function. : (15) in, , L reg Indicates the parallax regression loss. N Indicates the total number of pixels. Indicates the first The initial disparity value of each pixel. Indicates the first The true disparity value of each pixel. This represents the smoothed L1 loss function. The smoothed L1 loss is effective when the error is small ( Using a squared penalty (i.e., the behavior of L2 loss), the gradient decreases as the error decreases, resulting in a relatively smooth gradient; however, when the error is large ( It employs a linear penalty (i.e., the behavior of L1 loss), has a constant gradient, and is less sensitive to large outliers.

[0051] Stereo contrast feature loss borrows from the idea of ​​contrastive learning to bring the feature representations of matching pixel pairs closer together and push away the feature representations of non-matching pixel pairs further apart.

[0052] Specifically, taking either the left or right feature map as a reference, for each anchor pixel p in the reference feature map, according to the true disparity map D... gt Determine the matching pixel p in the feature map on the other side. + (Positive sample), and simultaneously select K non-matching pixels in the feature map on the other side along the epipolar direction: (Negative sample).

[0053] Stereo contrast feature loss is defined as : (16) Where sim(·,·) represents feature similarity (such as cosine similarity), and τ is the temperature hyperparameter. Features of the left eye anchor point. Right eye matching features (positive samples). This is the k-th non-matching feature (negative sample) of the right eye. This loss function makes the features of matching pixel pairs closer to each other in the feature space, and the features of non-matching pixel pairs farther apart, thereby enhancing the network's ability to constrain the feature consistency of matching point pairs.

[0054] Stereo-selective whitening loss: This loss is used to suppress the covariance component in features that is sensitive to changes in left and right eye perspectives, thereby improving the cross-domain generalization ability of features.

[0055] Specifically, the covariance matrix is ​​calculated for the intermediate layer features F output by the feature extraction network: (17) Perform eigenvalue decomposition on the covariance matrix Σ=QΛQ T Where Λ is the eigenvalue diagonal matrix and Q is the eigenvector matrix. Principal component directions with larger eigenvalues ​​(corresponding to directions sensitive to viewpoint changes) are selected, and covariance information in these directions is suppressed through whitening transformation. (18) Stereoselective whitening loss is a measure of the difference in features before and after whitening. : (19) in Let ||·|| be the covariance matrix of the whitened features. F It is the Frobenius norm, which is the square root of the sum of the squares of the elements of the matrix.

[0056] Step S160: Using the weighted sum of the disparity regression loss, the stereo contrast feature loss, and the stereo selective whitening loss as the total loss, adaptively weight the total loss using the quality weight map.

[0057] Using disparity regression loss L reg 3D contrast feature loss L scf and stereoselective whitening loss L ssw The weighted sum is the total loss. : (20) Where λ reg , λ scf , λ ssw These are the weighting coefficients for each loss.

[0058] Using the quality weight map W q Adaptive weighting of total loss : ,(twenty one) Quality weighting chart W q The weighting methods for the total loss include pixel-level weighting, region-level weighting, and hierarchical weighting. Pixel-level weighting weights each pixel individually; region-level weighting weights each region after dividing the image into regions; and hierarchical weighting weights feature maps at different scales separately. Through adaptive weighting, the interference of low-quality reconstructed regions on network parameter updates is reduced, while the constraint effect of high-quality geometric regions on network training is enhanced.

[0059] Step S170: Update the parameters of the feature extraction network and the cost aggregation network with the goal of minimizing the total loss.

[0060] To minimize the weighted total loss L weighted To achieve the desired result, the Adam optimizer is used to iteratively update the parameters of the feature extraction network and the cost aggregation network via gradient descent. For the convolutional neural network parameters, standard backpropagation and automatic differentiation are used to calculate the gradient. The learning rate is set to 10. -4The batch size is set to 8 or 16 based on the GPU memory. Training is performed on a synthetic dataset, and the trained network model is saved after iteration until the loss converges.

[0061] like Figure 2 As shown, this application also provides an underwater domain generalized stereo matching network training device 200 for implementing the above method. The underwater domain generalized stereo matching network training device 200 includes a data acquisition module 210, a degradation and feature extraction module 220, a cost aggregation module 230, a loss calculation module 250, and a weighted update module 260.

[0062] Specifically, the data acquisition module 210 is used to acquire the left and right eye images after epipolar correction, as well as the true disparity map between the left and right eye images. The degradation and feature extraction module 220 is used to construct binocular consistency degradation for the left and right eye images based on the ocean imaging degradation model, and then input the results into a feature extraction network with shared weights to obtain the left and right eye feature maps. The cost aggregation module 230 is used to construct a cost volume based on the left and right eye feature maps, and output an initial disparity map through a cost aggregation network. The quality assessment module 240 is used to generate a three-dimensional point cloud based on the initial disparity map and perform local evaluation on the three-dimensional point cloud. The system performs surface quality assessment and generates a quality weight map to indicate the reconstruction confidence of each region in the disparity map. The loss calculation module 250 calculates disparity regression loss based on the initial disparity map and the true disparity map, and calculates stereo contrast feature loss and stereo selective whitening loss based on the left eye feature map and the right eye feature map. The weighted update module 260 uses the weighted sum of the disparity regression loss, the stereo contrast feature loss, and the stereo selective whitening loss as the total loss, and adaptively weights the total loss using the quality weight map to minimize the total loss, thereby updating the parameters of the feature extraction network and the cost aggregation network.

[0063] In this embodiment, the underwater domain generalized stereo matching network training device 200 is used to implement each step of the above training method and can achieve the same beneficial effect. To avoid repetition, it will not be described again here.

[0064] like Figure 3 As shown, this application provides an underwater domain generalized stereo matching 3D reconstruction method, based on the feature extraction network and cost aggregation network trained by the training method described in the first embodiment, including the following steps: Step S310: Obtain the left and right eye images of the underwater scene and the parameters of the binocular camera, and perform epipolar correction on the left and right eye images.

[0065] Specifically, a binocular camera system for the underwater scene under test is installed on an underwater robot or a fixed detection platform to acquire left and right eye images of the underwater scene, and to obtain pre-calibrated binocular camera parameters (including focal length f, baseline B, principal point coordinates (c)). u c v The acquired original binocular images were subjected to epipolar correction using the Bouguet algorithm, so that the epipolar lines of corresponding spatial points in the left and right images were located on the same horizontal line.

[0066] Step S320: Input the epipolar-corrected image into the feature extraction network and cost aggregation network trained using the method described in the first embodiment, and output a disparity map.

[0067] Specifically, the epipolar-corrected left and right eye images are input into the feature extraction network and cost aggregation network trained using the first embodiment. This network, through underwater domain generalization training, has acquired cross-domain stereo matching capabilities in complex underwater environments. After forward propagation, the network outputs a disparity map D. out .

[0068] Step S330: Reconstruct the 3D point cloud based on the disparity map and the binocular camera parameters.

[0069] According to disparity map D out Reconstructing a 3D point cloud using binocular camera parameters. For the corrected binocular image, the relationship between disparity d and depth Z is Z = f·B / d. For any pixel (u, v) on the image plane, its 3D coordinates (X, Y, Z) in the camera coordinate system are: ,(twenty two) Where f represents the camera focal length (in pixels), B represents the baseline distance of the binocular cameras, and d represents the parallax value. 3D coordinates are calculated pixel-by-pixel to generate a 3D point cloud of the target scene.

[0070] In this embodiment, an optimization step for the 3D point cloud is also included: based on the disparity map D out A 3D point cloud is generated, and a quality weight map is generated by performing local surface quality assessment. The disparity map is then refined and corrected using the quality weight map. Based on the corrected disparity map, the optimized 3D point cloud is reconstructed. Disparity refinement involves fitting the matching costs of the original disparity and adjacent disparities into a quadratic curve, thereby finding a more accurate disparity implementation for the curve.

[0071] Step S340: Quantitatively characterize the defects in the three-dimensional point cloud to obtain information on the location, depth, length, and opening size of the defects.

[0072] Specifically, background suppression and adaptive segmentation are performed on the imaging envelope, and the geometric features of the segmented regions are extracted. The spatial location and geometric dimensions of the defects are calculated through surface geometric projection. The peak coordinates, -6dB boundary, and principal axis direction of the connected domain are extracted, and the defect burial depth (distance from the peak point along the CAD local normal to the outer surface), length (projection length of the principal axis of the connected domain), and opening size are calculated through surface normal projection.

[0073] In this embodiment, the target scene includes at least one of the following: underwater hull surface, underwater structure surface, complex lighting scene of a dock, and ocean weak texture surface scene.

[0074] Accordingly, such as Figure 4 As shown, this application also provides an underwater domain generalized stereo matching three-dimensional reconstruction device 400 for implementing the above reconstruction method, including an image acquisition module 410, an epipolar correction module 420, a three-dimensional reconstruction module 430, and a defect characterization module 440.

[0075] Specifically, the image acquisition module 410 is used to acquire the left and right eye images of the underwater scene and the parameters of the binocular camera. The epipolar correction module 420 is used to perform epipolar correction on the left and right eye images. The 3D reconstruction module 430 is used to input the epipolar-corrected image into the feature extraction network and cost aggregation network trained using the method described in the first embodiment, output a disparity map, and reconstruct a 3D point cloud based on the disparity map and the binocular camera parameters. The 3D reconstruction module 430 is also used to generate a 3D point cloud based on the disparity map, perform local surface quality assessment to generate a quality weight map, refine and correct the disparity map using the quality weight map, and reconstruct the optimized 3D point cloud based on the corrected disparity map. The defect characterization module 440 is used to quantitatively characterize defects in the 3D point cloud to obtain information on the location, depth, length, and opening size of defects.

[0076] In this embodiment, the underwater domain generalized stereo matching three-dimensional reconstruction device 400 is used to implement each step of the above reconstruction method and can achieve the same beneficial effect. To avoid repetition, it will not be described again here.

[0077] To verify the effectiveness of this application, experiments were conducted on a noise domain test set incorporating water scattering and abrupt changes in illumination. The experiments used PSMNet as the basic network architecture, comparing the performance of traditional PSMNet (using only disparity regression loss), methods with only SCF and SSW losses, and the complete scheme of this application (degradation model construction + SCF / SSW + 3D quality assessment). The evaluation metric was the mismatch rate (EPE>3px, i.e., the percentage of pixels with an endpoint error greater than 3 pixels). Experimental results show that the mismatch rate of traditional PSMNet is 15.4%, the mismatch rate of PSMNet with only SCF and SSW losses is reduced to approximately 8.7%, and the mismatch rate of the complete scheme of this application is 2.8%. The mismatch rate of this application is reduced by more than 5 times compared to traditional PSMNet, and also achieves approximately 3 times the performance improvement compared to the method with only SCF / SSW. These results fully verify the synergistic effect of the two innovative mechanisms—marine imaging degradation model construction and 3D point cloud quality assessment with adaptive weighting—on improving the accuracy of underwater stereo matching.

[0078] This application embodiment also provides a readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described functions. Figure 1 Any step in the corresponding method embodiment can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0079] The computer-readable storage medium of this application embodiment can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate or transmit a program for use by or in connection with an instruction execution system, apparatus or device.

[0080] The program code contained on the storage medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0081] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or terminal. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0082] The above description represents the preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A training method for an underwater domain generalized stereo matching network, characterized in that, Includes the following steps: Acquire the left and right eye images after epipolar correction, as well as the true disparity map between the left and right eye images; Based on the marine imaging degradation model, the left and right eye images are constructed using binocular consistency degradation and then input into a feature extraction network with shared weights to obtain left and right eye feature maps. Based on the left and right eye feature maps, a cost volume is constructed, and an initial disparity map is output through a cost aggregation network. A three-dimensional point cloud is generated based on the initial disparity map, and a local surface quality assessment is performed on the three-dimensional point cloud to generate a quality weight map that indicates the reconstruction confidence of each region in the disparity map. The disparity regression loss is calculated based on the initial disparity map and the true disparity map, and the stereo contrast feature loss and stereo selective whitening loss are calculated based on the left eye feature map and the right eye feature map. The total loss is obtained by weighting the disparity regression loss, the stereo contrast feature loss, and the stereo selective whitening loss, and then adaptively weighting the total loss using the quality weight map. With the goal of minimizing the total loss, the parameters of the feature extraction network and the cost aggregation network are updated.

2. The underwater domain generalized stereo matching network training method according to claim 1, characterized in that, The ocean imaging degradation model includes at least one of water attenuation, backscattering, non-uniform illumination, bright and dark fringes, and local specular reflection; and / or, In the binocular consistency degradation structure, the left eye image and the right eye image share water attenuation parameters, backscattering parameters, and ambient light parameters, and are respectively superimposed with local reflection perturbations and illuminance perturbations related to the viewing angle.

3. The underwater domain generalized stereo matching network training method according to claim 1, characterized in that, The stereo contrast feature loss is as follows: taking either the left or right feature map as a reference, for each anchor pixel in the reference feature map, a matching pixel is determined as a positive sample in the other side feature map based on the true disparity map, and a non-matching pixel is selected as a negative sample in the other side feature map; the difference in feature distance between the positive sample pair and the negative sample pair is calculated; and / or, The stereo-selective whitening loss is as follows: calculate the covariance matrix of the features output by the feature extraction network, and perform a whitening transformation on the covariance matrix to suppress the covariance components that are sensitive to changes in the left and right eye perspectives.

4. The underwater domain generalized stereo matching network training method according to claim 1, characterized in that, The quality weight map is used to perform pixel-level weighting, region-level weighting, or hierarchical weighting of the total loss; and / or, The quality indicators for the local surface quality assessment include at least one of the following: local plane fitting residual, normal vector fluctuation, point cloud porosity, and boundary continuity; and / or, The feature extraction network is ResNet, and the cost aggregation network is an aggregation network based on three-dimensional convolution.

5. A generalized stereo matching three-dimensional reconstruction method for underwater domains, characterized in that, Includes the following steps: Acquire the left and right eye images of the underwater scene and the parameters of the binocular camera, and perform epipolar correction on the left and right eye images; The epipolar-corrected image is input into the feature extraction network and cost aggregation network trained by the method described in any one of claims 1 to 4, and a disparity map is output. Reconstruct a 3D point cloud based on the disparity map and the binocular camera parameters; The three-dimensional point cloud is quantitatively characterized to obtain information on the location, depth, length, and opening size of the defects.

6. The underwater domain generalized stereo matching three-dimensional reconstruction method according to claim 5, characterized in that, Also includes: The three-dimensional point cloud is optimized by generating a three-dimensional point cloud based on the disparity map, performing local surface quality assessment to generate a quality weight map, using the quality weight map to refine and correct the disparity map, and reconstructing the optimized three-dimensional point cloud based on the corrected disparity map.

7. The underwater domain generalized stereo matching three-dimensional reconstruction method according to claim 5, characterized in that, The underwater scene refers to the surface of an underwater ship hull, the surface of an underwater structure, a dock lighting scene, or a marine weak texture curved surface scene.

8. A training device for an underwater domain generalized stereo matching network, characterized in that, include: The data acquisition module is used to acquire the left and right eye images after epipolar correction, as well as the true disparity map between the left and right eye images; The degradation and feature extraction module is used to construct binocular consistent degradation of the left and right eye images based on the ocean imaging degradation model, and then input the constructed images into a feature extraction network with shared weights to obtain the left and right eye feature maps. The cost aggregation module is used to construct a cost volume based on the left and right eye feature maps, and output an initial disparity map through the cost aggregation network; The quality assessment module is used to generate a three-dimensional point cloud based on the initial disparity map, perform local surface quality assessment on the three-dimensional point cloud, and generate a quality weight map to indicate the reconstruction confidence of each region in the disparity map. The loss calculation module is used to calculate disparity regression loss based on the initial disparity map and the true disparity map, and to calculate stereo contrast feature loss and stereo selective whitening loss based on the left eye feature map and the right eye feature map. The weighted update module is used to adaptively weight the total loss by using the weighted sum of the disparity regression loss, the stereo contrast feature loss, and the stereo selective whitening loss, and to update the parameters of the feature extraction network and the cost aggregation network with the goal of minimizing the total loss.

9. An underwater domain generalized stereo matching three-dimensional reconstruction device, characterized in that, include: The image acquisition module is used to acquire the left and right eye images of the underwater scene, as well as the parameters of the binocular camera. The epipolar correction module is used to perform epipolar correction on the left and right eye images; A 3D reconstruction module is used to input the epipolar-corrected image into a feature extraction network and a cost aggregation network trained by the method described in any one of claims 1 to 4, output a disparity map, and reconstruct a 3D point cloud based on the disparity map and the binocular camera parameters. The defect characterization module is used to quantitatively characterize the defects in the three-dimensional point cloud, and obtain information on the location, depth, length and opening size of the defects.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

Images