Multi-branch multi-scale submarine cable defect polarization detection method fusing attention mechanism
By adopting a multi-branch and multi-scale method with a fusion attention mechanism in the detection of submarine cable defects, combined with CLIP pre-trained large model and swin transformer, the problem of low detection accuracy in the underwater environment is solved, and more efficient submarine cable defect detection is achieved. .
Patent Information
- Application Number
- CN202411097446.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-22
AI Technical Summary
The existing submarine cable defect detection methods have low detection accuracy and low accuracy in underwater environments, and the defect types are not complete, and the polarization image enhancement fails to fully integrate information and global features of different scales, resulting in unsatisfactory detection results.
The multi-branch multi-scale submarine cable defect polarization detection method adopts a multi-branch multi-scale submarine cable defect polarization detection method through a convolutional network and swin transformer, and combines the image encoding characteristics of the CLIP pre-trained large model to perform underwater polarization image enhancement and submarine cable defect detection. , modulate convolutional features using the cross attention mechanism and global attention features to improve detection accuracy.
It significantly improves the accuracy and efficiency of defect detection of submarine cables, can better eliminate the scattering of underwater polarization images, and accurately detects the defect type and location of submarine cables.
Smart Images

Figure CN119000688A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, specifically to a multi-branch, multi-scale submarine cable defect polarization detection method that incorporates an attention mechanism. Technical Background
[0002] Underwater cables play a vital role in marine resource development, island economic development, and national coastal security. Cable damage is frequent, causing adverse effects and losses to marine industries and island residents. Therefore, real-time underwater cable inspection is crucial. In the underwater environment, underwater images suffer from blurriness, low contrast, and color distortion due to light scattering. Traditional visual inspection methods, such as using underwater robots equipped with cameras, are commonly used for underwater cable inspection. In some harsh marine environments, manual inspection is necessary, but this method is costly, has low accuracy, and low precision. Currently, deep learning-based visual inspection methods have become the mainstream for target detection, offering advantages such as high recognition accuracy, fast detection speed, and high precision. Existing methods for detecting defects in submarine cables often employ a two-stage model: first, underwater image enhancement, then target detection methods to identify defects. Underwater image enhancement often uses generative models such as GAN networks, inputting the blurred underwater image into the generator and outputting a clear underwater image end-to-end. Paired datasets are required for training. Alternatively, a method based on an underwater imaging physical model can be used to estimate key parameters such as background scattered light in the imaging model using neural networks, followed by image restoration. Since light exhibits partial polarization after scattering through water, polarization imaging can be used to detect defects in polarized images. However, due to the complex and diverse underwater environment, acquiring datasets for underwater cables is extremely difficult, and the available datasets are limited and lack a comprehensive range of defect types.
[0003] Currently, there are very few underwater target detection algorithms based on polarization features. Furthermore, due to the low resolution and low contrast saturation of underwater imaging, most underwater target detection algorithms employ two-stage networks. First, the input underwater polarization image is enhanced, and then the target is detected. However, these polarization image enhancement methods do not fully integrate information at different scales, nor do they pay sufficient attention to global and local features, resulting in unsatisfactory enhancement effects for polarization images.
[0004] Given the unique characteristics of submarine cable defect detection, applying the CLIP pre-trained large model can effectively improve the defect detection rate. Currently, the CLIP large model is not used in submarine cable defect polarization detection technology. Summary of the Invention
[0005] To improve the accuracy of submarine cable defect detection, this invention provides a multi-branch, multi-scale submarine cable defect polarization detection method incorporating an attention mechanism. The method comprises two stages: a polarization image enhancement stage and a submarine cable detection stage. Convolutional features and global attention features of the image are extracted using convolutional networks and a Swing transformer on different dimensions, and the global attention features are used to modulate the convolutional features. Simultaneously, image encoding features from a CLIP pre-trained large model are fused to further enhance the submarine cable defect detection rate.
[0006] The specific technical solution of this invention has the following steps:
[0007] Step 1: Constructing the underwater cable polarization image dataset:
[0008] Step 1.1 Take clear images of the cable defects in clean water using a regular camera; manually label the target location and type of the cable defects on the clear images and use this as the ground truth value y. i The location of the submarine cable defect includes the center coordinates, height, and width of the defect.
[0009] Step 1.2 Add mud and milk to clear water to simulate a turbid water environment. Use a polarization camera to capture linear underwater polarization images of cable defects in turbid water with polarization angles of 0°, 45°, and 90°. Adjust the exposure time of the polarization camera to keep the exposure intensity of the linear underwater polarization images consistent with that of the clear images. Replace the cable and repeat steps 1.1 and 1.2 until a dataset of more than 1,000 underwater cable polarization images is obtained.
[0010] Step 2: Build an underwater polarization image enhancement module to enhance the polarization image of the underwater submarine cable:
[0011] The underwater polarization image enhancement module consists of three parts: CLIP image encoder, swintransformer network, and UNet feature fusion module;
[0012] The polarization image of the underwater cable is input into the CLIP image encoder to obtain the CLIP image encoding vector C. The CLIP image encoder has the function of encoding a large number of different image contents. Encoding the polarization image of the underwater cable using the CLIP image encoder can greatly improve the quality of the image encoding features.
[0013] The polarization image of the underwater cable is input into a Swin transformer network to obtain attention features S at different scales. i The Swing Transformer network extracts attention features from different dimensions, including global image features, to prevent local distortion.
[0014] The polarization image of the underwater cable, the CLIP image encoding vector C, and the attention features S at different scales are used to... i The input is fed into the UNet feature fusion module; the UNet feature fusion module first performs convolution, then downsampling, to extract convolutional features CON at different spatial scales. i By fusing convolutional features, convolutional attention features, and cross-attention features of different dimensions through convolutional layers and upsampling operations, an underwater polarization enhancement feature map is obtained.
[0015] The underwater polarization enhancement feature map acquisition process is as follows:
[0016] The CLIP image encoding vector C and convolutional features CON at different spatial scales are used. i The input is fed into the cross-attention module, where cross-attention feature A is calculated. The process for calculating cross-attention feature A is as follows:
[0017] Q i =CNN(CON i )
[0018] K = MLP(C)
[0019] V = MLP(C)
[0020]
[0021] Where CNN represents convolutional operation, MLP represents fully connected layer, and Q... i V represents the query vector, K represents the value vector, and d represents the key vector. k Denotes the dimension of K, K T This represents the transpose of K, where i is 1;
[0022] Utilizing attention features S at different scales i Modulate convolutional features, and combine attention features S at different scales. i Convolutional features at different spatial scales CON i The input is fed into the feature modulation module to obtain the modulated convolutional attention features SC. i The modulation process is represented as follows:
[0023]
[0024] Wherein, CNN stands for convolution operation. This represents matrix addition. Represents the Hadamard product of matrices;
[0025] Step 3: Following the method in Step 2, image enhancement is performed on the underwater polarization maps at three polarization angles of 0°, 45°, and 90° to obtain underwater polarization enhancement feature maps at the three polarization angles of 0°, 45°, and 90° respectively.
[0026] Step 4: The underwater polarization enhancement feature maps at three polarization angles of 0°, 45°, and 90° are stitched together according to channels to obtain the underwater polarization enhancement fusion feature map;
[0027] Step 5: Determine the type and location of submarine cable defects using the submarine cable defect detection module.
[0028] The submarine cable defect detection module consists of a transformer and a fully connected layer. The underwater polarization enhancement fusion feature map is input into the submarine cable defect detection module. The transformer first outputs a prediction vector of the submarine cable defect, and the prediction vector is then output as the type and location of the submarine cable defect through the fully connected layer.
[0029] Step 6: Construct the loss function L, and update the network parameters by calculating the value of the loss function until the transformer converges smoothly.
[0030] First, the transformer outputs N prediction vectors, where N is set to twice the number of cable defect types. These vectors are then passed through a fully connected layer to obtain the predicted cable defect types and locations. i The true value representing the target type and location of submarine cable defects is expressed by... This represents N predicted values of submarine cable defect type and location obtained after passing through the fully connected layer; the set of N true values and y i The difference is used To fill, For objects of the defect type that do not have a corresponding match, the Hungarian matching algorithm is used to solve for the set y. i and Best match:
[0031]
[0032] in, It is y i The predicted value with index σ(i) Matching cost;
[0033] The matching cost considers the similarity between the predicted cable defect type and the predicted cable defect location and the actual value, where each element y of the actual value... i =(c i ,b i ), where c i It is a type of defect in submarine cables; b i It is the location of the target defect in the submarine cable; the defect type c of the submarine cable is... iThe probability is defined as Define the prediction box as but:
[0034]
[0035] L box (.) represents the box loss, defined as:
[0036]
[0037] Among them, L iou (.) represents the IOU loss, and ||.||1 represents the L1 norm. It is the predicted target location of the submarine cable defect;
[0038] Define the loss function L as:
[0039]
[0040] Step 7: Detect the predicted submarine cable defects: Use the updated transformer network from Step 6 to detect the predicted submarine cable targets as in Step 5.
[0041] An electronic device includes: one or more processors; and a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the submarine cable defect detection method described in steps one to seven above.
[0042] A computer-readable storage medium storing program code that can be called by a processor to execute the submarine cable defect detection method described in steps one through seven above.
[0043] Compared with existing technologies, the beneficial technical effects of the present invention are:
[0044] The submarine cable defect polarization detection method of this invention combines polarization image processing, image enhancement, CLIP image encoder, and target detection, and designs a multi-branch, multi-scale network. Convolutional and attention features are extracted in different dimensions, and the attention features are used to modulate the convolutional features, which can better eliminate scattering in underwater polarization images. A cross-attention mechanism is used, with the image encoding features of the CLIP image encoder guiding the underwater polarization image enhancement module, improving the quality of the underwater polarization image. A submarine cable defect detection module is constructed using a transformer and fully connected layers, which can accurately detect the type and location of submarine cable defects. Attached Figure Description
[0045] Figure 1A flowchart of a method for detecting polarization defects in submarine cables;
[0046] Figure 2 The overall architecture of the submarine cable defect polarization detection network;
[0047] Figure 3 This is a structural diagram of the underwater polarization image enhancement module;
[0048] Figure 4 Here is a structural diagram of the cross-attention module;
[0049] Figure 5 This is a structural diagram of the feature modulation module;
[0050] Figure 6 This is a structural diagram of a submarine cable defect detection module. Detailed Implementation
[0051] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.
[0052] This embodiment discloses a multi-branch, multi-scale submarine cable defect polarization detection method incorporating an attention mechanism. First, an underwater submarine cable polarization image dataset is constructed. Then, a multi-branch, multi-scale submarine cable defect polarization detection network incorporating an attention mechanism is built, comprising two stages: an underwater polarization image enhancement stage and a submarine cable defect detection stage. Convolutional features and global attention features of the images are extracted using convolutional networks and a Swing transformer on different dimensions, and the global attention features are used to modulate the convolutional features. Simultaneously, image encoding features from a CLIP pre-trained large model are fused to improve the submarine cable defect detection rate. The specific submarine cable defect detection process is as follows: Figure 1 and Figure 2 As shown, the specific steps include:
[0053] S1: Construction of underwater cable polarization image dataset:
[0054] S11: Place the submarine cable in clean water;
[0055] S12: In a dark room, in an underwater environment without natural light, incandescent lamps are used as the active light source;
[0056] S13: Take clear images of underwater submarine cables using a regular camera;
[0057] S14: Manually label the location and type of submarine cable defects in clear images and use them as the true values; where the location of the submarine cable defect target includes the center coordinates of the submarine cable defect target, the height and width of the submarine cable defect target;
[0058] S15: Add mud and skim milk to clear water to simulate a turbid underwater environment, and use a polarization camera to capture linear underwater polarization images at three polarization angles: 0°, 45°, and 90°. The protein particles in skim milk are similar in size to suspended particles in seawater, which can cause strong scattering, ensuring that the collected images are similar to underwater images of the real environment. Adjust the camera exposure time to keep the exposure intensity of the linear underwater polarization images and the clear images at the same level.
[0059] S16: Replace the submarine cable and repeat S11 to S15 until a dataset of 1000 underwater polarization images is obtained.
[0060] S2: Build an underwater polarization image enhancement module to enhance the polarization image of underwater submarine cables:
[0061] The underwater polarization image enhancement module consists of three parts: CLIP image encoder, swintransformer, and UNet feature fusion module.
[0062] S21: Input the polarization image of the underwater cable into the CLIP image encoder to obtain the CLIP image encoding vector C. The CLIP image encoder has the function of encoding a large number of different image contents. Encoding the polarization image of the underwater cable using the CLIP image encoder can greatly improve the quality of the image encoding features.
[0063] S22: Input the polarization image of the underwater cable into the Swin transformer to obtain attention features S at different scales. i The swin transformer extracts attention features from different dimensions, including global image features, to prevent local distortion.
[0064] S23 combines underwater cable polarization images, CLIP image encoding vector C, and attention features S at different scales. i The input is fed into the UNet feature fusion module, which is as follows: Figure 3 As shown, in this embodiment, the UNet feature fusion module is configured with 4 layers, and all convolutional layers in the UNet feature fusion module use 3*3 convolutional kernels. The convolutional layers and downsampling in the UNet feature fusion module work together to extract convolutional features CON at different spatial scales. i , i = {1, 2, 3, 4}, where i represents the different layers of the UNet feature fusion module, CON i This represents the convolutional features at different layers in the UNet feature fusion module. Convolutional features CON i It contains rich local texture information.
[0065] S24: Combine the CLIP image encoding vector C with convolutional features CON at different spatial scales. i The input is fed into the cross-attention module, and the structure diagram of the cross-attention module is as follows: Figure 4 As shown, the process of calculating cross-attention feature A is expressed as follows:
[0066] Q i =CNN(CON i )
[0067] K = MLP(C)
[0068] V = MLP(C)
[0069]
[0070] Where CNN represents convolutional operation, MLP represents fully connected layer, and Q... i Let K represent the query vector, V represent the key vector, and d represent the value vector. k Denotes the dimension of K, K T This represents the transpose of K. In the example, d... k Set 32, i = 1, to represent the convolutional feature CON1 of the first layer.
[0071] S25: Utilizing Attention Features at Different Scales i Feature modulation convolution features, combining attention features S at different scales. i Convolutional features at different spatial scales CON i The input is fed into the feature modulation module, and the structure diagram of the feature modulation module is as follows: Figure 5 As shown, the modulated convolutional attention features SC are obtained. i This process can be represented as:
[0072]
[0073] Wherein, CNN stands for convolution operation. This represents matrix addition. Represents the Hadamard product of matrices;
[0074] S26: By fusing convolutional features, convolutional attention features, and cross-attention features of different dimensions through convolutional layers and upsampling operations, an underwater polarization enhancement feature map is obtained.
[0075] S3: Image enhancement of underwater polarization maps at three polarization angles of 0°, 45°, and 90°: Perform image enhancement according to step two to obtain underwater polarization enhancement feature maps at three polarization angles of 0°, 45°, and 90° respectively.
[0076] S4: The underwater polarization enhancement feature maps at three polarization angles of 0°, 45°, and 90° are stitched together according to the channels to obtain the underwater polarization enhancement fusion feature map.
[0077] S5: Determining the type and location of submarine cable defects using the submarine cable defect detection module: The submarine cable defect detection module consists of a transformer and a fully connected layer; the underwater polarization enhancement fusion feature map is input into the submarine cable defect detection module, first passing through the transformer to output a prediction vector of the submarine cable defect, and then passing through the fully connected layer to output the type and location of the submarine cable defect. The structure diagram of the submarine cable defect detection module is shown below. Figure 6 As shown.
[0078] S6: Construct the loss function L, and update the network parameters by calculating the value of the loss function.
[0079] First, a transformer outputs N prediction vectors, where N is set to twice the number of cable defect types. These vectors are then passed through a fully connected layer to obtain the predicted cable defect types and locations. Using y... i The true value representing the target type and location of submarine cable defects is expressed by... Represents N predicted values of submarine cable defect type and location obtained after passing through the fully connected layer; using y i Fill with a set of N true values. For objects of the defect type that do not have a corresponding match, the Hungarian matching algorithm is first used to solve for the set y. i and Best match:
[0080]
[0081] in It is y i The predicted value with index s(i) Matching cost.
[0082] The matching cost considers the similarity between the predicted cable defect type and the predicted cable defect location and the actual value. Each element y of the actual value... i =(c i ,b i ), where c i It is a type of defect in submarine cables; b i It is the location of the target defect in the submarine cable; the defect type c of the submarine cable is... i The probability is defined as Define the prediction box as but:
[0083]
[0084] Lbox (.) represents the box loss, defined as:
[0085]
[0086] Among them, L iou (.) represents the IOU loss, and ||.||1 represents the L1 norm. It is the predicted target location of the submarine cable defect.
[0087] Define the loss function L as:
[0088]
[0089] S7: Detect predicted defects in submarine cables.
[0090] This embodiment builds a multi-branch, multi-scale submarine cable defect polarization detection network with an attention mechanism on the PyTorch platform. Python version 3.9 is used, and hardware acceleration is achieved using an NVIDIA RTX 2080 graphics card. The initial learning rate of the network is set to 1×10⁻⁶. -4 To ensure smooth network convergence, the learning rate remained constant for the first 100 rounds, and a cosine decay strategy was implemented for the last 100 rounds. The batch size was set to 8, the number of training rounds was 200, and the Adam optimizer was used to optimize the network parameters.
[0091] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.
[0092] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
Claims
1. A multi-branch multi-scale submarine cable defect polarization detection method integrating attention mechanism, characterized in that: The following steps are involved: Step 1: Construction of underwater cable polarization image dataset: Step 1.1 Use an ordinary camera to take a clear picture of the submarine cable defect in clear water; manually mark the target position of the submarine cable defect and the type of defect on the clear image and use it as the true value y i ; Wherein, the submarine cable defect target position includes the center position coordinates, height and width of the submarine cable defect target; Step 1.2: Add silt and milk to clean water to simulate the turbid water environment, and use a polarization camera to capture linear underwater polarization images of the submarine cable defects in turbid water with polarization angles of 0°, 45°, and 90° respectively; adjust the exposure time of the polarization camera to keep the exposure intensity of the linear underwater polarization image and the clear image consistent; replace the submarine cable, and repeat steps 1.1 and 1.2 until more than 1000 underwater submarine cable polarization image datasets are obtained; Step 2: Build an underwater polarization image enhancement module to enhance the polarization image of the underwater submarine cable; Step 3, performing image enhancement on underwater polarization images at three polarization angles of 0°, 45°, and 90°: performing image enhancement according to step 2, and obtaining underwater polarization enhancement feature images at three polarization angles of 0°, 45°, and 90° respectively; Step 4: splicing the underwater polarization enhancement feature maps at three polarization angles of 0°, 45°, and 90° according to the channels to obtain an underwater polarization enhancement fusion feature map; Step 5: Determine the type and location of submarine cable defects through the submarine cable defect detection module: Step 6: Construct the loss function L, and update the network parameters by calculating the value of the loss function until the transformer converges smoothly. Step 7: Detect the predicted submarine cable defect target: Use the transformer network updated in step 6 to detect the predicted submarine cable target according to step 5.
2. The multi-branch multi-scale submarine cable defect polarization detection method integrating attention mechanism according to claim 1 is characterized in that: In step 2, the underwater polarization image enhancement module consists of three parts, namely, CLIP image encoder, swin transformer network, and UNet feature fusion module; Input the underwater cable polarization image into the CLIP image encoder to obtain the CLIP image coding vector C; the CLIP image encoder has the function of encoding a large number of different image contents. Using the CLIP image encoder to encode the underwater cable polarization image can greatly improve the quality of image coding features; The polarization image of the underwater cable is input into the swin transformer network to obtain attention features S of different scales i ; The swin transformer network extracts attention features of different dimensions, and the attention features contain global features of the image to prevent local distortion; The underwater cable polarization image, CLIP image encoding vector C and attention features S of different scales are i Input into the UNet feature fusion module; the UNet feature fusion module first performs convolution and then downsamples to extract convolution features of different spatial scales CON i ; Convolutional features, convolutional attention features, and cross-attention features of different dimensions are fused through convolutional layers and upsampling operations to obtain an underwater polarization enhanced feature map.
3. The multi-branch multi-scale submarine cable defect polarization detection method integrating attention mechanism according to claim 2 is characterized in that: The underwater polarization enhancement feature map acquisition process is as follows: The CLIP image encoding vector C and the convolution features CON of different spatial scales are combined i Input into the cross-attention module and calculate the cross-attention feature A. The process of calculating the cross-attention feature A is as follows: Q i =CNN(CON i ) K=MLP(C) V=MLP(C) Among them, CNN represents the convolution operation, MLP represents the fully connected layer, and Q i represents the query vector, V represents the value vector, K represents the key vector, and d k represents the dimension of K, K T represents the transpose of K, i is 1; Using attention features S of different scales i Modulate the convolutional features and convert the attention features S of different scales i and convolutional features of different spatial scales CON i Input into the feature modulation module to obtain the modulated convolutional attention feature SC i , the modulation process is expressed as: Among them, CNN represents the convolution operation, represents matrix addition, represents the Hadamard product of matrices.
4. The multi-branch multi-scale submarine cable defect polarization detection method integrating attention mechanism according to claim 1 is characterized in that: In step five, the submarine cable defect detection module is composed of a transformer and a fully connected layer; the underwater polarization enhanced fusion feature map is input into the submarine cable defect detection module, first outputted by the transformer as a prediction vector of the submarine cable defect, and the prediction vector is outputted by the fully connected layer as the type and location of the submarine cable defect.
5. The multi-branch multi-scale submarine cable defect polarization detection method integrating attention mechanism according to claim 1 is characterized in that: In step 6, the loss function L is constructed as follows: First, the transformer outputs N prediction vectors, where N is set to twice the number of submarine cable defect types. After passing through the fully connected layer, the predicted submarine cable defect type and location are obtained; y i Indicates the true value of the type and location of the submarine cable defect target, represents the N predicted values of submarine cable defect types and locations obtained after passing through the fully connected layer; the N true value sets and y i The difference of To fill, There is no corresponding defect type object; use the Hungarian matching algorithm to solve the set y i and Best match for: in, Yes i and the predicted value indexed by σ(i) Matching costs; The matching cost takes into account the similarity between the predicted cable defect type and the predicted cable defect location and the true value. Each element y of the true value i =(c i ,b i ), where c i is the type of submarine cable defect; b i is the position of the submarine cable defect target; the submarine cable defect type c i The probability is defined as The prediction box is defined as but: L box (.) is the box loss, defined as: Among them, L iou (.) represents IOU loss, ||.||1 represents L1 norm, is the predicted target position of submarine cable defects; The loss function L is defined as:
6. An electronic device comprising one or more processors; and a memory; and one or more programs, characterized in that: The one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 5.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 5.
Citation Information
Cited By
Method and device for detecting apparent defects of polaroid
CN119198565A
Reflection-type enameled wire appearance defect intelligent monitoring device and method
CN120121623A
Cable surface spraying defect detection method and system based on vision assistance
CN120495297A
Polarization image fusion method and system based on global perception and multi-branch heterogeneous attention
CN121120426A
