Visual detection method for underwater structure surface state of interferent-containing water body environment

Abstract

The invention discloses a visual detection method for the surface state of an underwater structure in a water body environment containing interferents, and the method comprises the following steps: firstly, obtaining a visual optical flow diagram through a self-supervised learning optical flow technology, carrying out the preprocessing, determining a threshold value through an Otsu method to segment a background and a foreground, and carrying out the recognition of the background and the foreground; and further searching the small-size underwater interferent based on a novel inter-frame difference method of adjacent frame alignment, and recovering an area shielded by the underwater interferent in the current frame by using STTN. Then, PVTv2 is adopted as a trunk to extract a feature pyramid of an input image, and AGFE and MFCA are integrated to improve the efficiency of underwater structure defect detection. And finally, defining a loss function for supervising the defect detection network so as to better find an optimal parameter according to an output result. The method is high in precision of detecting the surface state of the underwater structure in a water body environment containing interferents, is high in environmental adaptability, and improves the detection efficiency and precision.

Description

Technical Field

[0001] The present invention belongs to the fields of digital image processing, salient target detection and computer vision technology. Specifically, it provides a method for visually detecting the surface state of underwater structures in a water environment containing interference objects. The method can quickly and stably eliminate the influence of the water environment containing interference objects on the detection of the surface state of underwater structures and accurately segment the defective areas of underwater structures. Background Art

[0002] With the continuous development of urbanization in my country, water conservancy projects such as dams and bridges are critical infrastructure for allocating water resources and promoting economic development. However, during the long-term use of water conservancy facilities, their underwater structures are constantly eroded by water, resulting in surface defects such as cracks, spalling, and holes, which in turn create safety hazards. For example, cracks in bridge piers can gradually expand, leading to bridge collapse. Corrosion in submarine pipelines or storage tanks can cause leaks, resulting in fires or explosions. Underwater structural failures directly threaten the lives of divers, construction workers, and nearby residents. Underwater structural defect detection, as a crucial step in ensuring the safe and stable operation of water conservancy facilities, can scientifically and effectively identify potential failure risks and prevent accidents, thereby ensuring the safe operation of underwater structures and improving their reliability and stability. Therefore, regular surface inspections are crucial to prevent further deterioration of underwater structural defects.

[0003] Today, underwater vehicles equipped with cameras have become essential tools for underwater exploration missions on behalf of humans. Obtaining reliable underwater visual information is a fundamental prerequisite for dynamic environmental perception and rapid location and tracking of visual targets. However, due to the complexity of the underwater environment, visual perception is more difficult underwater than in the atmosphere.

[0004] Obtaining images of underwater structure surface defects is often affected by complex factors, including image blur caused by scattering, color distortion caused by water's selective absorption of light, and information loss caused by occlusion by underwater interfering objects. Furthermore, natural water bodies often contain various interfering objects, including algae and decaying leaves, which lead to varying degrees of visual occlusion. The movement of underwater robots disturbs the water flow, exacerbating the impact of underwater interfering objects. Existing research has largely focused on image degradation caused by water absorption and particle scattering, which can be categorized into image dehazing, color correction, and underwater image processing within a fusion framework. However, while image blur and color distortion degrade the visual quality of images, they rarely alter the image information of underwater structures and generally do not hinder the distinction between defective and normal areas on the structure's surface. Huang et al.'s research demonstrated that pre-enhancement of underwater images, such as dehazing and color correction, imposes additional computational cost but does not effectively improve the performance of defect detection models. In contrast to the effects of scattering and color distortion, occlusion caused by underwater interfering objects alters the structural features of underwater structure images, leading to false defect detection. Therefore, it is important to focus on the effects of underwater obstructions, rather than image enhancement techniques such as de-scattering and color correction. However, the information changes and loss caused by underwater obstructions have been largely overlooked. Therefore, developing an image preprocessing method that addresses the issue of underwater obstructions is crucial for high-precision detection of underwater surface conditions. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a method for visually detecting the surface status of underwater structures in a water environment containing interferences, which can identify surface defects of underwater structures more efficiently, accurately and stably.

[0006] The above purpose is achieved through the following technical solutions:

[0007] The present invention provides a method for visually detecting the surface state of an underwater structure in a water environment containing interference, comprising the following steps:

[0008] 1) Obtain a visual optical flow map through self-supervised learning of optical flow (Selflow) technology and perform preprocessing;

[0009] 2) Using Otsu's method (Otsu threshold segmentation algorithm) to determine a threshold to segment the background and foreground and detect large underwater interference objects;

[0010] 3) Based on a novel inter-frame difference method based on adjacent frame alignment, the background offset between adjacent frames is calculated according to the visualized optical flow map preprocessed in step 1), and the background changes are compensated to further search for small-sized underwater interference objects;

[0011] 4) Use the Spatio-Temporal Transformer Network (STTN) to restore the area occluded by underwater interferers in the current frame;

[0012] 5) Using PVTv2 as the backbone network to extract the feature pyramid of the input image;

[0013] 6) Integrating an asymmetric global feature enhancement module (AGFE) and a multi-feature cross aggregation module (MFCA) into the PVTv2 backbone network in step 5) to process the features of the image extracted in step 5) to improve the efficiency of underwater structural defect detection and obtain detection results;

[0014] 7) Define a loss function to supervise the defect detection network constructed in steps 5) and 6) to better find the optimal parameters based on the output results.

[0015] Furthermore, step 1) includes the following sub-steps:

[0016] A1) collecting underwater structure videos in a water environment containing interference objects and splitting the videos into continuous frame images;

[0017] A2) Let {I k-1 , I k ,I k+1} are three adjacent video frames to be processed, and {I k-1 , I k} and {I k ,I k+1} as input, and the edge map is obtained by fast edge detection of structured forest and

[0018] A3) Using UCM method and Divided into multiple closed areas. Among them, and They are and Segment mapping of ;

[0019] A4) Calculate the color contrast Ctr between each segmented region i and other regions j in the visual optical flow map i c , the formula is as follows:

[0020]

[0021] Where N is the number of segmented regions, and is the CIELAB color histogram at region i, j, ||·||2 represents the Euclidean distance, and p i and pj is the centroid position of region i, j, σ spa is the parameter of the spatial weighting scheme, and its value here is 0.4.

[0022] Furthermore, step 2) includes the following sub-steps:

[0023] B1) Using Otsu threshold segmentation algorithm to calculate the threshold for distinguishing foreground from background;

[0024] B2) According to the saliency detection theory, the greater the color contrast value, the higher the probability that the area belongs to the foreground. Therefore, the foreground area with a color contrast greater than the threshold determined in step B1) is regarded as the initial underwater interference area, that is, the salient area detected in the visualized optical flow map. and As the initial underwater interference area, large-sized underwater interferences are detected.

[0025] Furthermore, in order to better highlight the moving underwater interference in step 3), the pixel coordinates corresponding to the background area between adjacent frames should be as consistent as possible:

[0026] C1) Calculate the background offset between adjacent frames based on the visualized optical flow map: Assume Rf i It is one of the first areas of underwater interference detected. and Rf i exist and The expanded area after the 8 connected domain windows are expanded, and Rf is calculated according to the displacement values of the background areas around them. i In adjacent frames {I k-1 , I k} and {I k ,I k+1} offset compensation vector between and The formula is as follows:

[0027]

[0028] Among them, OF(·) refers to the extraction of optical flow vectors in a region, and mean(·) is the mean calculation symbol;

[0029] C2) Repeat step C1) to obtain adjacent frames {I k-1 , I k} and {I k ,I k+1 Two compensated optical flow fields between

[0030] C3) According to and Adjust the pixel coordinates (x, y) by aligning the current frame I k and adjacent frame I k-1 , I k+1 The background area in the image is used to align adjacent frames and obtain aligned adjacent frames. The formula is as follows:

[0031]

[0032] C4) through the current frame I k Aligned adjacent frames The difference between the two is calculated to obtain the detection results of small-sized underwater interference objects. The formula is as follows:

[0033]

[0034] Where T′ is the noise threshold, which is used to filter the noise of the difference image in the three RGB channels. Here, it is set to 15. AvF(·) and Bw(·) are filtering operations and binary operations, respectively.

[0035] Furthermore, step 4) includes the following sub-steps:

[0036] D1) Integrate the large-scale underwater interference object detection results obtained in step 2) and the small-scale underwater interference object detection results obtained in step 3) to obtain the final underwater interference object detection result S k , the formula is as follows:

[0037]

[0038] D2) Use the Spatio-Temporal Transformer Network (STTN) to restore the area occluded by underwater interferers in the current frame. The adjacent frames of the underwater video are used as the input of the damaged video sequence to be reconstructed, and the corresponding underwater interferer detection results are used as the corresponding frame-by-frame mask input to achieve accurate restoration of the area occluded by underwater interferers.

[0039] Furthermore, step 5) includes the following sub-steps:

[0040] E1) Use PVTv2 as the backbone network to extract the feature pyramid of the input image after removing the occlusion of underwater interference objects C a ∈{64,128,320,512}, a∈{1,2,3,4}, W and H are the width and height of the image respectively, and the multi-scale feature information is used to locate the position of the defect;

[0041] E2) Extracted features Contains rich positional relationships, convolves F1, and reduces the number of channels to 1 to obtain implicit edge features

[0042] E3) vs. F1 edge Perform upsampling to obtain the edge map prediction image Out edge .

[0043] Furthermore, step 6) includes the following sub-steps:

[0044] F1) uses the asymmetric global feature enhancement module (AGFE) to capture the features obtained in step E1) The information in the global mapping prediction map Out is obtained global ;

[0045] F2) Use the multi-feature cross aggregation module (MFCA) module to combine the features obtained in step E1) and They are fused with global features and edge features respectively, and the fused information is transferred from the pixel-level target edge to the entire target area to obtain two adjacent prior mapping prediction maps Out and Out temp .

[0046] Furthermore, step F1) includes the following sub-steps:

[0047] F1-1) uses four cascade branches {b t ,t=1,2,3,4} to capture different features. Each branch consists of a set of asymmetric convolutions. Due to the different sizes of defects in the image, different sizes of convolution kernels are used to perform non-feature convolution on each branch to extract information of different scales of the image. The output of each branch is as follows:

[0048]

[0049] in, It is a convolution operation, and the convolution kernel is m×n. For the dilated convolution operation, the convolution kernel is (2t-1)×(2t-1), and the dilation rate and padding are set to (2t-1)×(2t-1);

[0050] F1-2) Concatenate feature tensors along the first branch The fused feature tensor is passed through a 3×3 convolution module to further extract features and compress the channel dimension to 64, generating an output tensor with a unified feature representation. By adjusting the number of channels of F4 to 64 through convolution, we can get

[0051] F1-3) Cascade the outputs of the four branches and Add together and use the RELU function to get the global features of the AGFE module that embeds multi-scale information The formula is as follows:

[0052]

[0053] F1-4) In order to generate a spatially consistent single-channel representation, the global feature Passed through a 1×1 convolutional layer, retaining The spatial dimension of , while performing weighted aggregation on the input channel at each spatial position, reducing the feature dimension to 1 to obtain a single channel feature map

[0054] F1-5) uses bilinear interpolation to transform the single channel global feature map T global Restore to the original spatial resolution H×W and get the global mapping prediction map Out global .

[0055] Furthermore, in step F2), an MSFA module is constructed to perform feature fusion:

[0056] F2-1) Fusion input features In1 and In2 to obtain fusion feature R in , the formula is as follows:

[0057]

[0058] Where σ is the activation function and Up(·) is the upsampling operation;

[0059] F2-2) The input feature In1 is divided into 32 groups of candidate features with the same number of channels Through R in Perform the reverse to obtain the reverse attention-guided features in MFCA Will Periodically interpolate to the feature group separated by In1 to generate connected features P cc , the formula is as follows:

[0060]

[0061] F2-3)P cc Combined with the original input features In1 and In3 to get the output Out MFCA

[0062]

[0063] Where cat(·) represents the cascade operation, and MSFA(·) is the above-mentioned MSFA module operation;

[0064] F2-4) Repeat steps F2-1) to F2-3) twice each, and Fuse with global features and edge features respectively to obtain two different feature maps The formula is as follows:

[0065]

[0066] Among them, T′ global It's T global Double upsampling guidance map;

[0067] F2-5) Use bilinear interpolation to feature Upsample them to the original input size resolution H×W and obtain two adjacent prior mapping prediction maps Out and Out temp .

[0068] Furthermore, step 7) includes the following sub-steps:

[0069] G1) The four prediction graphs obtained in steps E3, F1, and F2 {Out edge ,Out,Out temp ,Out global Participate in the loss function calculation and define three different loss functions to supervise the defect detection network composed of steps 5) and 6): Loss e Represents the binary cross entropy loss of the boundary prediction result, Loss g Represents the mixed boundary loss of the global feature prediction result, Loss c Represents the mixed boundary loss of local feature prediction results, the formula is as follows:

[0070]

[0071] Among them, GT is the true value, and the loss function L hybrid (·) is used to guide the defect detection network to learn pixel-level, object-level and image-level features, L bce (·) is the weighted binary cross entropy loss;

[0072] G2) Add the results of these three loss functions to get the total loss Loss of back propagation and parameter update t , in order to improve the prediction accuracy of the model, the formula is as follows:

[0073] Loss t =Loss e +Loss g +Loss c (twenty one)

[0074] The final underwater structure defect detection result is the prediction graph Out output by the defect detection network after loss function optimization.

[0075] Compared with the prior art, the present invention provides a method for visually detecting the surface state of underwater structures in a water environment containing interference, which has the following beneficial effects:

[0076] (1) The detection method of the present invention differs from the traditional approach of capturing defect information based on feature extraction and edge detection. Instead, it utilizes a cascaded framework that first addresses the visual occlusion caused by underwater interference and then detects structural defect information on the object's surface. This method simultaneously overcomes issues such as image blur and color distortion, enabling flexible detection of various defects and reducing false detections and missed detections of structural defects.

[0077] (2) The recognition method of the present invention analyzes the actual situation in which visual inspection of underwater structures is subject to both image degradation and visual occlusion. By utilizing inter-frame dynamic information, the method restores obscured image details and resolves the visual occlusion problem caused by underwater interference, thereby providing reliable image data for surface condition assessment of underwater structures.

[0078] (3) The recognition method of the present invention detects underwater structural defects through edge guidance and PVTv2 network. It can identify surface defects of various shapes by identifying subtle differences between the target area and the background area, and has strong adaptability to low-quality underwater environments (such as those with color distortion and blur).

[0079] (4) The identification method of the present invention integrates the elimination of visual occlusion of underwater interference objects and the visual detection of the surface status of the structure. The proposed method can obtain accurate defect detection results in an environment containing water objects, and has high applicability and detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a flow chart of a method for visually detecting the surface state of an underwater structure in a water environment containing interferences proposed by the present invention;

[0081] Figure 2 This is a structural diagram of the underwater structure surface status detection network proposed by the present invention;

[0082] Figure 3 This is a structural diagram of the asymmetric global feature enhancement module (AGFE) proposed in the present invention;

[0083] Figure 4 This is a structural diagram of the multi-feature cross aggregation module (MFCA) proposed in the present invention;

[0084] Figure 5 This is the data set used in the verification experiment of the present invention;

[0085] Figure 6 Comparison of details before and after restoration of underwater interference occlusion using the proposed method on the USD-RSI dataset: (a) original image, (b) details of the underwater interference occlusion area, (c) underwater interference mask, (d) restored image, and (e) restored details.

[0086] Figure 7 Comparison of the underwater structural defect detection method of the present invention with four other detection models (ICON, BSANet, DGNett, CrackFormer) on the USD-VDO dataset: (a) structural defect image to be detected, (b) manually labeled ground truth image, (c) defect detection result of BSANet, (d) defect detection result of CrackFormer, (e) defect detection result of ICON, (f) defect detection result of DGNet, and (g) defect detection result of the method of the present invention;

[0087] Figure 8 Comparison of underwater structural defect detection performed directly on the USD-VDW dataset using the image of the underwater interference occlusion area restored by the method of the present invention and the original image: (a) original underwater structural defect image, (b) underwater structural defect image after restoration after occlusion of underwater interference, (c) manually marked structural defect ground truth image, (d) underwater structural defect detection result performed directly without restoration after occlusion of interference, (e) underwater structural defect detection result after restoration after occlusion of underwater interference. DETAILED DESCRIPTION

[0088] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0089] The present invention adopts the following technical solutions, combined with Figure 1 In order to complete the task of visual inspection of the surface state of underwater structures in a water environment containing interference, the present invention proposes a method for visual inspection of the surface state of underwater structures in a water environment containing interference, comprising the following steps:

[0090] 1) Obtain a visual optical flow map through self-supervised learning of optical flow (Selflow) technology and perform preprocessing;

[0091] 2) Using Otsu's method (Otsu threshold segmentation algorithm) to determine a threshold to segment the background and foreground and detect large underwater interference objects;

[0092] 3) Based on a novel inter-frame difference method based on adjacent frame alignment, the background offset between adjacent frames is calculated according to the visualized optical flow map preprocessed in step 1), and the background changes are compensated to further search for small-sized underwater interference objects;

[0093] 4) Use the Spatiotemporal Transformer Network (STTN) to restore the area occluded by underwater interferers in the current frame;

[0094] 5) Using PVTv2 as the backbone network to extract the feature pyramid of the input image;

[0095] 6) Integrating an asymmetric global feature enhancement module (AGFE) and a multi-feature cross aggregation module (MFCA) into the PVTv2 backbone network in step 5) to process the features of the image extracted in step 5) to improve the efficiency of underwater structural defect detection and obtain detection results;

[0096] 7) Define a loss function to supervise the defect detection network constructed in steps 5) and 6) to better find the optimal parameters based on the output results.

[0097] Preferably, step 1) includes the following sub-steps:

[0098] A1) collecting underwater structure videos in a water environment containing interference objects and splitting the videos into continuous frame images;

[0099] A2) Let {I k-1 , I k ,I k+1} are three adjacent video frames to be processed, and {I k-1 , I k} and {I k ,I k+1} as input, and the edge map is obtained by fast edge detection of structured forest and

[0100] A3) Using UCM method and Divided into multiple closed areas, where and They are and Segment mapping of ;

[0101] A4) Calculate the color contrast Ctr between each segmented region i and other regions j in the visual optical flow map ic , the formula is as follows:

[0102]

[0103] Where N is the number of segmented regions, and is the CIELAB color histogram at region i, j, ||·||2 represents the Euclidean distance, and p i and p j is the centroid position of region i, j, σ spa is the parameter of the spatial weighting scheme, in this embodiment σ spa The value is 0.4.

[0104] Preferably, step 2) includes the following sub-steps:

[0105] B1) Using Otsu threshold segmentation algorithm to calculate the threshold for distinguishing foreground from background;

[0106] B2) According to the saliency detection theory, the larger the color contrast value, the higher the probability that the area belongs to the foreground. Therefore, the foreground area with a color contrast greater than the threshold determined in step B1) is regarded as the initial underwater interference area, that is, the salient area detected in the visualized optical flow map. and As the initial underwater interference area, large-sized underwater interferences are detected.

[0107] Preferably, in order to better highlight the moving underwater interference object in step 3), the pixel coordinates corresponding to the background area between adjacent frames should be as consistent as possible, which mainly includes the following sub-steps:

[0108] C1) Calculate the background offset between adjacent frames based on the visualized optical flow map: Assume Rf i It is one of the first areas of underwater interference detected. and Rf i exist and The expanded area after the 8 connected domain windows are expanded, and Rf is calculated according to the displacement values of the background areas around them. i In adjacent frames {I k-1 , I k} and {I k ,I k+1} offset compensation vector between and The formula is as follows:

[0109]

[0110] Among them, OF(·) refers to the extraction of optical flow vectors in a region, and mean(·) is the mean calculation symbol;

[0111] C2) Repeat step C1) to obtain adjacent frames {I k-1 , I k} and {I k ,I k+1 Two compensated optical flow fields between

[0112] C3) According to and Adjust the pixel coordinates (x, y) by aligning the current frame I k and adjacent frame I k-1 , I k+1 The background area in the image is used to align adjacent frames and obtain aligned adjacent frames. The formula is as follows:

[0113]

[0114] C4) through the current frame I k Aligned adjacent frames The difference between the two is calculated to obtain the detection results of small-sized underwater interference objects. The formula is as follows:

[0115]

[0116] Wherein, T′ is a noise threshold, which is used to filter out noise in the difference image of the three RGB channels. In this embodiment, T′ is set to 15. AvF(·) and Bw(·) are filtering operations and binary operations, respectively.

[0117] Preferably, step 4) includes the following sub-steps:

[0118] D1) Integrate the large-scale underwater interference object detection results obtained in step 2) and the small-scale underwater interference object detection results obtained in step 3) to obtain the final underwater interference object detection result S k , the formula is as follows:

[0119]

[0120] D2) Use the Spatio-Temporal Transformer Network (STTN) to restore the area occluded by underwater interferers in the current frame. The adjacent frames of the underwater video are used as the input of the damaged video sequence to be reconstructed, and the corresponding underwater interferer detection results are used as the corresponding frame-by-frame mask input to achieve accurate restoration of the area occluded by underwater interferers.

[0121] Combine Figure 2 Preferably, step 5) includes the following sub-steps:

[0122] E1) Use PVTv2 as the backbone network to extract the feature pyramid of the input image after removing the occlusion of underwater interference objects C a ∈{64,128,320,512}, a∈{1,2,3,4}, W and H are the width and height of the image respectively, and the multi-scale feature information is used to locate the position of the defect;

[0123] E2) Extracted features Contains rich positional relationships, convolves F1, and reduces the number of channels to 1 to obtain implicit edge features

[0124] E3) vs. F1 edge Perform upsampling to obtain the edge map prediction image Out edge .

[0125] Preferably, step 6) includes the following sub-steps:

[0126] F1) uses the asymmetric global feature enhancement module (AGFE) to capture the features obtained in step E1) The information in the global mapping prediction map Out is obtained global ;

[0127] F2) Use the multi-feature cross aggregation module (MFCA) module to combine the features obtained in step E1) and They are fused with global features and edge features respectively, and the fused information is transferred from the pixel-level target edge to the entire target area to obtain two adjacent prior mapping prediction maps Out and Out temp .

[0128] Combine Figure 3 Preferably, step F1) includes the following sub-steps:

[0129] F1-1) uses four cascade branches {b t ,t=1,2,3,4} to capture different features. Each branch consists of a set of asymmetric convolutions. Due to the different sizes of defects in the image, different sizes of convolution kernels are used to perform non-feature convolution on each branch to extract information of different scales of the image. The output of each branch is as follows:

[0130]

[0131] in, It is a convolution operation, and the convolution kernel is m×n. For the dilated convolution operation, the convolution kernel is (2t-1)×(2t-1), and the dilation rate and padding are set to (2t-1)×(2t-1);

[0132] F1-2) Concatenate feature tensors along the first branch The fused feature tensor is passed through a 3×3 convolution module to further extract features and compress the channel dimension to 64, generating an output tensor with a unified feature representation. By adjusting the number of channels of F4 to 64 through convolution, we can get

[0133] F1-3) Cascade the outputs of the four branches and Add together and use the RELU function to get the global features of the AGFE module that embeds multi-scale information The formula is as follows:

[0134]

[0135] F1-4) In order to generate a spatially consistent single-channel representation, the global feature Pass through the 1×1 convolution layer, retaining The spatial dimension of , while performing weighted aggregation on the input channel at each spatial position, reducing the feature dimension to 1 to obtain a single channel feature map

[0136] F1-5) uses bilinear interpolation to transform the single channel global feature map T global Restore to the original spatial resolution H×W and get the global mapping prediction map Out global .

[0137] Combine Figure 4 Preferably, in step F2), an MSFA module is constructed to perform feature fusion, including the following sub-steps:

[0138] F2-1) Fusion input features In1 and In2 to obtain fusion feature R in , the formula is as follows:

[0139]

[0140] Where σ is the activation function and Up(·) is the upsampling operation;

[0141] F2-2) The input feature In1 is divided into 32 groups of candidate features with the same number of channels Through R in Perform the reverse to obtain the reverse attention-guided features in MFCA Will Periodically interpolate to the feature group separated by In1 to generate connected features P cc , the formula is as follows:

[0142]

[0143] F2-3)P cc Combined with the original input features In1 and In3 to get the output Out MFCA

[0144]

[0145] Where cat(·) represents the cascade operation, and MSFA(·) is the above-mentioned MSFA module operation;

[0146] F2-4) Repeat steps F2-1) to F2-3) twice each, and Fuse with global features and edge features respectively to obtain two different feature maps The formula is as follows:

[0147]

[0148] Among them, T′ global It's T global Double upsampling guidance map;

[0149] F2-5) Use bilinear interpolation to feature Upsample them to the original input size resolution H×W and obtain two adjacent prior mapping prediction maps Out and Out temp .

[0150] Preferably, step 7) includes the following sub-steps:

[0151] G1) The four prediction graphs obtained in steps E3, F1, and F2 {Out edge ,Out,Out temp ,Out global Participate in the loss function calculation and define three different loss functions to supervise the defect detection network composed of steps 5) and 6): Loss e Represents the binary cross entropy loss of the boundary prediction result, Loss g Represents the mixed boundary loss of the global feature prediction result, Loss c Represents the mixed boundary loss of local feature prediction results, the formula is as follows:

[0152]

[0153] Among them, GT is the true value, and the loss function L hybrid (·) is used to guide the defect detection network to learn pixel-level, object-level and image-level features, L bce (·) is the weighted binary cross entropy loss;

[0154] G2) Add the results of these three loss functions to get the total loss Loss of back propagation and parameter update t , in order to improve the prediction accuracy of the model, the formula is as follows:

[0155] Loss t =Loss e +Loss g +Loss c (35)

[0156] The final underwater structure defect detection result is the prediction graph Out output by the defect detection network after loss function optimization.

[0157] Combine Figure 5 Three data sets were established for experimental verification. Underwater interference object occlusion recovery experiments, surface structure defect detection comparison experiments, and quantitative analysis of the improvement in underwater structure defect detection accuracy after underwater interference object occlusion recovery were conducted on the USD-RSI data set, USD-VDO data set, and USD-VDW data set.

[0158] Combine Figure 6 On the USD-RSI dataset, details of areas occluded by suspended impurities were compared before and after restoration. In four experimental videos, cracks in underwater structures were partially obscured by selected suspended impurities. This method not only removed the occlusion, achieving a natural visual effect, but also preserved the original image information to the greatest extent possible, minimizing the damage to the original information.

[0159] Combine Figure 7 On the USD-VDO dataset, the underwater structure defect detection model in this example is compared with four other detection models (ICON, BSANet, DGNett, and Crackformer): BSANet and Crackformer can detect most defect areas, but there are obvious omissions and false detections in the results; ICON and DGNet have better detection results, but the defect areas also have blurred edges; the results obtained by the defect detection model in this example are closest to the real defect area, and have strong robustness in detecting different defect areas in various complex situations.

[0160] Combine Figure 8On the USD-VDW dataset, using this example method to remove underwater distractor occlusions significantly improves the accuracy of visual defect detection on underwater structures. In quantitative testing, compared to not performing distractor occlusion recovery, the AUC (Area Under Curve) value after performing underwater distractor occlusion recovery increased from 0.6873 to 0.7891, the F1 score increased from 0.6576 to 0.7413, and the balanced accuracy increased from 0.8980 to 0.9209, enabling the detection of uncertain surface defects on underwater structures under harsh environmental conditions.

[0161] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A method for visually detecting the surface state of an underwater structure in a water environment containing interference, characterized in that: The following steps are involved: 1) Obtain a visual optical flow map through self-supervised learning of optical flow technology and perform preprocessing; 2) Determine a threshold value to segment the background and foreground using the Otsu method and detect large underwater interference objects; 3) Based on a novel inter-frame difference method based on adjacent frame alignment, the background offset between adjacent frames is calculated according to the visualized optical flow map preprocessed in step 1), and the background changes are compensated to further search for small-sized underwater interference objects; 4) Use the spatiotemporal transformer network (STTN) to restore the area occluded by underwater interference in the current frame; 5) Using PVTv2 as the backbone network to extract the feature pyramid of the input image; 6) Integrating the asymmetric global feature enhancement module AGFE and the multi-feature cross aggregation module MFCA into the PVTv2 backbone network in step 5) to process the features of the image extracted in step 5) to improve the efficiency of underwater structural defect detection and obtain detection results; 7) Define a loss function to supervise the defect detection network constructed in steps 5) and 6) to better find the optimal model parameters based on the output results.

2. The method for visually detecting the surface state of an underwater structure in an environment containing interference according to claim 1, characterized in that: Step 1) specifically includes the following sub-steps: A1) collecting underwater structure videos in a water environment containing interference objects and splitting the videos into continuous frame images; A2) Let {I k-1 , I k ,I k+1 } are three adjacent video frames to be processed, and {I k-1 , I k } and {I k ,I k+1 } as input, and the edge map is obtained by fast edge detection of structured forest and A3) Using UCM method and Divided into multiple closed areas, where and They are and Segment mapping of ; A4) Calculate the color contrast Ctr between each segmented region i and other regions j in the visual optical flow map i c , the formula is as follows: Where N is the number of segmented regions, and is the CIELAB color histogram at region i, j, ||·||2 represents the Euclidean distance, and p i and p j is the centroid position of region i, j, σ spa is the parameter of the spatial weighting scheme, and its value here is 0.

4.

3. The method for visually detecting the surface state of an underwater structure in an environment containing interference according to claim 2, characterized in that: Step 2) includes the following sub-steps: B1) Using Otsu threshold segmentation algorithm to calculate the threshold for distinguishing foreground from background; B2) According to the saliency detection theory, the larger the color contrast value, the higher the probability that the area belongs to the foreground. Therefore, the foreground area with a color contrast greater than the threshold determined in step B1) is regarded as the initial underwater interference area, that is, the salient area detected in the visualized optical flow map. and As the initial underwater interference area, large-sized underwater interferences are detected.

4. The method for visually detecting the surface state of an underwater structure in an environment containing interference according to claim 3, characterized in that: Step 3) includes the following sub-steps: C1) Calculate the background offset between adjacent frames based on the visualized optical flow map: Assume Rf i It is one of the first areas of underwater interference detected. and Rf i exist and The expanded area after the 8 connected domain windows are expanded, and Rf is calculated according to the displacement values of the background areas around them. i In adjacent frames {I k-1 , I k } and {I k ,I k+1 } offset compensation vector between and The formula is as follows: Among them, OF(·) refers to the extraction of optical flow vectors in a region, and mean(·) is the mean calculation symbol; C2) Repeat step C1) to obtain adjacent frames {I k-1 , I k } and {I k ,I k+1 Two compensated optical flow fields between C3) According to and Adjust the pixel coordinates (x, y) by aligning the current frame I k and adjacent frame I k-1 , I k+1 The background area in the image is used to align adjacent frames and obtain aligned adjacent frames. The formula is as follows: C4) through the current frame I k Aligned adjacent frames The difference between the two is calculated to obtain the detection results of small-sized underwater interference objects. The formula is as follows: Where T′ is the noise threshold, which is used to filter the noise of the difference image in the three RGB channels. Here, it is set to 15. AvF(·) and Bw(·) are filtering operations and binary operations, respectively.

5. The method for visually detecting the surface state of an underwater structure in an environment containing interference according to claim 4, characterized in that: Step 4) includes the following sub-steps: D1) Integrate the large-scale underwater interference object detection results obtained in step 2) and the small-scale underwater interference object detection results obtained in step 3) to obtain the final underwater interference object detection result S k , the formula is as follows: D2) Use the Spatio-Temporal Transformer Network (STTN) to restore the area occluded by underwater interferers in the current frame. The adjacent frames of the underwater video are used as the input of the damaged video sequence to be reconstructed, and the corresponding underwater interferer detection results are used as the corresponding frame-by-frame mask input to achieve accurate restoration of the area occluded by underwater interferers.

6. The method for visually detecting the surface state of an underwater structure in an environment containing interference according to claim 5, characterized in that: Step 5) includes the following sub-steps: E1) Use PVTv2 as the backbone network to extract the feature pyramid of the input image after removing the occlusion of underwater interference objects C a ∈{64,128,320,512}, a∈{1,2,3,4}, W and H are the width and height of the image respectively, and the multi-scale feature information is used to locate the position of the defect; E2) Extracted features Contains rich positional relationships, convolves F1, and reduces the number of channels to 1 to obtain implicit edge features E3) vs. F1 edge Perform upsampling to obtain the edge map prediction image Out edge .

7. The method for visually detecting the surface state of an underwater structure in an environment containing interference according to claim 6, characterized in that: Step 6) includes the following sub-steps: F1) uses the asymmetric global feature enhancement module AGFE to capture the features obtained in step E1) The information in the global mapping prediction map Out is obtained global ; F2) Using the multi-feature cross aggregation module MFCA to combine the features obtained in step E1) and They are fused with global features and edge features respectively, and the fused information is transferred from the pixel-level target edge to the entire target area to obtain two adjacent prior mapping prediction maps Out and Out temp .

8. The method for visually detecting the surface state of an underwater structure in an environment containing interference according to claim 7, characterized in that: Step 7) includes the following sub-steps: G1) The four prediction graphs obtained in steps E3, F1, and F2 {Out edge ,Out,Out temp ,Out global Participate in the loss function calculation and define three different loss functions to supervise the defect detection network composed of steps 5) and 6): Loss e Represents the binary cross entropy loss of the boundary prediction result, Loss g Represents the mixed boundary loss of the global feature prediction result, Loss c Represents the mixed boundary loss of local feature prediction results, the formula is as follows: Among them, GT is the true value, and the loss function L hybrid (·) is used to guide the defect detection network to learn pixel-level, object-level and image-level features, L bce (·) is the weighted binary cross entropy loss; G2) Add the results of these three loss functions to get the total loss Loss of back propagation and parameter update t , in order to improve the prediction accuracy of the model, the formula is as follows: Loss t =Loss e +Loss g +Loss c (7) The final underwater structure defect detection result is the prediction graph Out output by the defect detection network after loss function optimization.

Images