Submarine cable pipeline detection method based on underwater acoustic and optical fusion

Abstract

The present invention discloses a submarine cable pipeline detection method based on underwater acoustic-optical fusion, which relates to the field of underwater detection. The method comprises: acquiring underwater sonar image data and underwater optical image data through underwater sensor equipment; performing preprocessing extraction to obtain sonar rectangular image data and standard optical image data; using a parallel dual-branch convolutional-Transformer network to perform feature extraction to obtain an acoustic modal embedding feature set and an optical modal embedding feature set; performing attention perception and weighted fusion to obtain acoustic-optical multimodal fusion-level features; decoding through a multimodal decoder structure to obtain acoustic-optical fusion pixel-level features; and performing detection and classification using a detection regression head and a classification head to obtain submarine cable pipeline detection results. The method solves the technical problems of insufficient detection accuracy and robustness in existing submarine cable pipeline detection, achieving the technical effects of improving detection accuracy and enhancing detection robustness.

Description

Technical Field

[0001] The present application relates to the field of underwater detection, and in particular to a method for detecting submarine cables and pipelines based on underwater acoustic and optical fusion. Background Art

[0002] Underwater inspection tasks, especially the inspection of submarine cables and pipelines, are important technical requirements in the fields of marine engineering, underwater resource exploration, and environmental monitoring. However, due to the particularity of the underwater environment, such as insufficient light, turbid water, biological interference and other factors, underwater inspection tasks face many challenges. In the inspection of submarine cables and pipelines, traditional methods mainly rely on single acoustic or optical sensors. Acoustic sensors can penetrate turbid water and capture acoustic information of the seabed, but their accuracy in target recognition is limited; while optical sensors can provide high-resolution image data, their performance will be greatly reduced in the case of insufficient light or turbid water. Both acoustic and optical data are severely affected by underwater environmental factors, making it difficult for single modal data to provide sufficient information for accurate detection.

[0003] Among the current related technologies, submarine cable pipeline detection has technical problems such as insufficient detection accuracy and robustness. Summary of the Invention

[0004] The present application provides a submarine cable pipeline detection method based on underwater acoustic-optical fusion, utilizes underwater sensor equipment to simultaneously collect acoustic image and optical image data, preprocesses the collected data, extracts high-quality sonar rectangular image data and standard optical image data, and uses a parallel dual-branch convolution-Transformer network to extract features of the two modal data. Through attention perception and weighted fusion mechanism, the acoustic modal embedding feature set and the optical modal embedding feature set are fused to obtain acoustic-optical multimodal fusion-level features, and the fused features are decoded using a multimodal decoder structure to obtain acoustic-optical fusion pixel-level features. Detection and classification are performed through detection regression head and classification head, and other technical means. By fusing the data of the two modalities of acoustic image and optical image, the complementary advantages of the two modalities in representing submarine cable pipelines are fully utilized, thereby achieving the technical effect of improving detection accuracy and enhancing detection robustness.

[0005] The present application provides a submarine cable pipeline detection method based on underwater acoustic-optical fusion, comprising: acquiring underwater sonar image data and underwater optical image data containing a target submarine cable pipeline through underwater sensor equipment; preprocessing and extracting the underwater sonar image data and underwater optical image data respectively to obtain sonar rectangular image data and standard optical image data; using a parallel dual-branch convolution-Transformer network to extract features from the sonar rectangular image data and the standard optical image data respectively to obtain an acoustic modal embedding feature set and an optical modal embedding feature set; performing attention perception and weighted fusion on the acoustic modal embedding feature set and the optical modal embedding feature set to obtain acoustic-optical multimodal fusion-level features; decoding the acoustic-optical multimodal fusion-level features through a multimodal decoder structure to obtain acoustic-optical fusion pixel-level features; and detecting and classifying the acoustic-optical fusion pixel-level features using a detection regression head and a classification head to obtain a submarine cable pipeline detection result.

[0006] In a possible implementation, the sonar rectangular image data and the standard underwater optical image data are obtained, and the following processing is performed: a rectangular image conversion formula is constructed: ,in, They represent the input underwater sonar image data and the preprocessed sonar rectangular image data, respectively. are the height and width of the image, Represents the pre-processing reconstruction algorithm set, which includes angle conversion , traverse the projection matrix ; Based on the rectangular image conversion formula, the underwater sonar image data is preprocessed by window conversion to obtain the sonar rectangular image data; and a lightweight underwater image enhancement algorithm is constructed: , ,in, represents the pre-processed standard optical image data, is an activation function for deep learning. and It is a convolution layer with a convolution kernel size of 7×7 and a maximum pooling layer. Represents input underwater optical image data expanded along the RGB three-color space; the underwater optical image data is preprocessed based on the lightweight underwater image enhancement algorithm to obtain the standard optical image data.

[0007] In a possible implementation, the acoustic modality embedding feature set and the optical modality embedding feature set are obtained, and the following processing is performed: an acoustic coder in the parallel dual-branch convolution-Transformer network is used to construct an acoustic coding extraction formula as follows: ,in, represents the embedding vector extracted by the convolution-Transformer structure, is the sonar rectangular image data, is the corresponding result of the embedding vector after the acoustic encoder, Represents the convolutional-Transformer used for inference, where the encoder contains fully connected layers , represents the activation function for deep learning, is a scalar parameter used to scale the softmax output; based on the acoustic coding extraction formula, the sonar rectangular image data is encoded and extracted to obtain the acoustic modality embedding feature set; through the optical encoder in the parallel dual-branch convolution-Transformer network, the optical coding extraction formula is constructed as follows: ,in, 、 、 Respectively represent the first and The feature representation matrix on up to the Nth image block is obtained by The corresponding result generated after the optical encoder is 、 and , represents a weight operator for balancing multiple image blocks; encoding and extracting the standard optical image data based on the optical encoding extraction formula to obtain the optical modality embedding feature set.

[0008] In a possible implementation, the acoustic-optical multimodal fusion-level features are obtained by performing the following processing: performing attention perception on the acoustic modality embedding feature set and the optical modality embedding feature set based on a dynamic attention fusion network to obtain a multimodal feature attention map; and constructing a feature alignment formula: ,in Represents the KL divergence loss function used, and its corresponding inputs are and , which is the first feature regions and optical feature embedding feature regions, where N is the total number of feature regions; performing feature alignment on the acoustic modal embedding feature set and the optical modal embedding feature set based on the feature alignment formula; and constructing a multimodal feature fusion formula: ,in, is the optical modality embedding feature set, is the acoustic modal embedding feature set, is a weight map; the multimodal feature attention map is used as the weight map, and the acoustic modal embedding feature set and the optical modal embedding feature set after feature alignment are fused based on the feature fusion formula to obtain the acoustic-optical multimodal fusion level feature.

[0009] In a possible implementation, the multimodal feature attention map is obtained and the following processing is performed: According to the dynamic attention fusion network, the attention fusion network formula is constructed as follows: ,in, Represents the attention map corresponding to the acoustic modality embedding feature set, represents the attention map corresponding to the optical modality embedding feature set, represents the maximum pooling layer in deep learning, represents the average pooling layer in deep learning, Indicates that the convolution kernel size is convolutional layer; performing attention perception on the acoustic modality embedding feature set and the optical modality embedding feature set based on the attention fusion network formula to obtain an acoustic embedding feature attention map and an optical embedding feature attention map; performing superposition fusion and channel confusion on the acoustic embedding feature attention map and the optical embedding feature attention map to obtain the multimodal feature attention map.

[0010] In a possible implementation, the multimodal feature attention map is obtained and the following processing is performed: a superposition fusion formula is constructed: ,in, represents the consistent attention map of superposition fusion, represents the fusion weight of the acoustic embedding feature attention map, Represents the fusion weight of the optical embedding feature attention map, which is dynamically updated during the learning process; based on the superposition fusion formula, the acoustic embedding feature attention map and the optical embedding feature attention map are superimposed and fused to generate a consistent attention map; construct the channel confusion formula: ,in, Represents multiple groups of convolution layers with 7 convolution kernels. is the activation function used in deep learning, ; Based on the channel confusion formula, the consistent attention map is subjected to channel confusion to obtain the multimodal feature attention map.

[0011] In a possible implementation, the acousto-optic fusion pixel-level features are obtained by performing the following processing: using the multimodal decoder structure to introduce differential convolution to obtain a feature decoding process, wherein the feature decoding process is specifically as follows: ,in, is a conventional Vanilla convolution operation, Represents multiple differential convolution operations, including central differential and angular differential convolution operations; constructs a reparameterization process: ,in, Representing convolution kernels corresponding to multiple convolution operations; based on the feature decoding process and the reparameterization process, the acoustic-optical multimodal fusion level feature is feature decoded to obtain the acoustic-optical fusion pixel-level feature.

[0012] In a possible implementation, the submarine cable pipeline detection result is obtained, and the following processing is performed: a prediction box regression function is constructed based on the detection regression head: ,in, Represents the center point b of the predicted box and the center point of the real box The Euclidean distance, The length of the diagonal representing the minimum circumscribed moment between two center points, Represents the intersection-over-union ratio of the predicted box and the true box, Representation parameters The corresponding impact factor, represents a parameter for measuring aspect ratio consistency; performing anchor frame detection on the acousto-optic fusion pixel-level features based on the prediction frame regression function to obtain a target feature prediction frame; constructing a target classification process based on the classification head, the target classification process is specifically as follows: ,in, Indicates the The probability that an image belongs to each category, is the fully connected layer, is a classification function; based on the target classification process, the target feature prediction frame is classified and determined to obtain the submarine cable pipeline detection result.

[0013] The submarine cable pipeline detection method based on underwater acoustic-optical fusion proposed in this application is to first collect underwater sonar image data and underwater optical image data containing the target submarine cable pipeline through underwater sensor equipment, then pre-process and extract the underwater sonar image data and underwater optical image data respectively to obtain sonar rectangular image data and standard optical image data, and then use a parallel two-branch convolution-Transformer network to extract features of the sonar rectangular image data and the standard optical image data respectively to obtain acoustic modal embedding feature set and optical modal embedding feature set, and then perform attention perception and weighted fusion on the acoustic modal embedding feature set and the optical modal embedding feature set to obtain acoustic-optical multimodal fusion-level features, and then decode the acoustic-optical multimodal fusion-level features through a multimodal decoder structure to obtain acoustic-optical fusion pixel-level features, and finally use the detection regression head and the classification head to detect and classify the acoustic-optical fusion pixel-level features to obtain the submarine cable pipeline detection results, thereby achieving the technical effect of improving detection accuracy and enhancing detection robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention are briefly introduced below. Flowcharts are used in this application to illustrate the operations performed by the methods according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in precise order. Instead, various steps may be processed in reverse order or simultaneously as needed. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0015] Figure 1 A schematic flow chart of a submarine cable pipeline detection method based on underwater acoustic-optical fusion provided in an embodiment of the present application.

[0016] Figure 2 Schematic diagram of the process of preprocessing and extracting underwater sonar image data and underwater optical image data in the submarine cable pipeline detection method based on underwater acoustic and optical fusion provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0018] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0019] In the following description, reference is made to “some embodiments” which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. The terms “including” and “having” and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used herein are for the purpose of describing the embodiments of this application only.

[0020] The present application provides a method for detecting submarine cables based on underwater acoustic and optical fusion. Figure 1 As shown, the method includes:

[0021] Step S100: Acquire underwater sonar image data and underwater optical image data containing the target submarine cable pipeline through underwater sensor equipment.

[0022] Specifically, the underwater sensor device refers to a sensor device capable of operating in an underwater environment and used to collect sonar and optical image data. The underwater sensor device, which integrates sonar and optical sensors, is deployed on the seabed or nearby waters and activated to simultaneously collect sonar and optical image data.

[0023] Step S200 , pre-processing and extracting the underwater sonar image data and the underwater optical image data respectively to obtain sonar rectangular image data and standard optical image data.

[0024] Specifically, image processing algorithms are used to preprocess the collected data to improve image quality. Underwater sonar image data is subjected to denoising and enhancement to obtain clear rectangular sonar image data. This preprocessed sonar image data has a clear rectangular shape and features. Underwater optical image data is corrected and enhanced to obtain standard optical image data. This standard optical image data is preprocessed optical image data that meets certain standards and specifications.

[0025] like Figure 2 As shown, in a possible implementation, the sonar rectangular image data and the standard underwater optical image data are obtained, and step S200 further includes step S210, constructing a rectangular image conversion formula: ,in, They represent the input underwater sonar image data and the preprocessed sonar rectangular image data, respectively. are the height and width of the image, Represents the pre-processing reconstruction algorithm set, which includes angle conversion , traverse the projection matrix Specifically, a rectangular image conversion formula is constructed using a mathematical formula and an image processing algorithm. The rectangular image conversion formula is used to convert the input forward-looking sonar sector image data into a rectangular shape. The preprocessing and reconstruction algorithm is an image processing algorithm for preprocessing and reconstructing the image.

[0026] Step S220: Window conversion preprocessing is performed on the underwater sonar image data based on the rectangular image conversion formula to obtain the sonar rectangular image data. Specifically, the window conversion preprocessing is performed using image processing software or an algorithm. The rectangular image conversion formula constructed in step S210 is applied to the input underwater sonar image data. Window conversion preprocessing is performed on the image through steps such as angle conversion and traversing the projection matrix in the algorithm, and the preprocessed sonar rectangular image data is output.

[0027] Step S230: Construct a lightweight underwater image enhancement algorithm: , ,in, represents the pre-processed standard optical image data, is an activation function for deep learning. and It is a convolution layer with a convolution kernel size of 7×7 and a maximum pooling layer. Represents input underwater optical image data expanded along the RGB color space. Specifically, a lightweight underwater image enhancement algorithm is constructed using deep learning algorithms and image processing techniques. This lightweight underwater image enhancement algorithm is used to improve the quality of underwater optical images. A convolutional layer with a 7×7 kernel size and a max pooling layer are used to extract image features and perform dimensionality reduction. The input underwater optical image data is expanded along the RGB color space, enabling the algorithm to process color images and reduce the effects of color cast caused by underwater light refraction and scattering.

[0028] Step S240, pre-processing the underwater optical image data based on the lightweight underwater image enhancement algorithm to obtain the standard optical image data. Specifically, the lightweight underwater image enhancement algorithm is pre-processed using image processing software or an algorithm. The lightweight underwater image enhancement algorithm constructed in step S230 is applied to the input underwater optical image data, and the image is pre-processed through steps such as the convolution layer, the maximum pooling layer, and the activation function in the algorithm, and the pre-processed standard optical image data is output. Underwater images are often affected by factors such as illumination and turbidity, resulting in poor image quality. This implementation method pre-processes sonar images and optical images by constructing a rectangular image conversion formula and a lightweight underwater image enhancement algorithm, thereby improving image quality, making them more suitable for subsequent feature extraction and detection classification, and improving the detection accuracy and efficiency of submarine cables and pipelines.

[0029] Step S300: Use a parallel dual-branch convolution-Transformer network to perform feature extraction on the sonar rectangular image data and the standard optical image data respectively to obtain an acoustic modality embedding feature set and an optical modality embedding feature set.

[0030] Specifically, a parallel dual-branch convolutional-Transformer network is used for feature extraction. A parallel dual-branch network refers to a network structure with two parallel branches, each processing a different type of image data. A convolutional neural network (CNN) is a deep learning network structure used for image feature extraction; a Transformer network is a deep learning network based on a self-attention mechanism that can process sequence data and extract global features. Sonar rectangular image data and standard optical image data are input into the parallel dual-branch network, respectively. A convolutional neural network (CNN) is used to extract local features of the image, while a Transformer network is used to extract global features and contextual information of the image. The extracted features are then combined into an acoustic modality embedding feature set and an optical modality embedding feature set.

[0031] In one possible implementation, the step of obtaining the acoustic modality embedding feature set and the optical modality embedding feature set, step S300 further includes step S310, constructing an acoustic coding extraction formula as follows through the acoustic encoder in the parallel dual-branch convolution-Transformer network:

[0032] ,in, represents the embedding vector extracted by the convolution-Transformer structure, is the sonar rectangular image data, is the corresponding result of the embedding vector after the acoustic encoder, Represents the convolutional-Transformer used for inference, where the encoder contains fully connected layers , represents the activation function for deep learning, is a scalar parameter used to scale the softmax output. Specifically, the acoustic encoder contains a fully connected layer and an activation function in deep learning to increase the nonlinear expression ability of the network. Used to stabilize the training process.

[0033] In step S320, the sonar rectangular image data is encoded and extracted based on the acoustic coding extraction formula to obtain the acoustic modal embedding feature set. Specifically, the sonar rectangular image data is fed into an acoustic encoder. Based on the acoustic coding extraction formula, image features are extracted through convolution operations and a Transformer structure. The acoustic modal embedding feature set is output as input for subsequent steps.

[0034] In step S330, an optical code extraction formula is constructed by using the optical encoder in the parallel dual-branch convolution-Transformer network as follows:

[0035] ,in, 、 、 Respectively represent the first and The feature representation matrix on up to the Nth image block is obtained by The corresponding result generated after the optical encoder is 、 and , Represents a weighting operator used to balance multiple image blocks. Specifically, the optical encoder infers information about how image features vary across different blocks, segments the input image (i.e., standard optical image data) into multiple blocks, and performs feature extraction on each block. The weighting operator balances the features across multiple blocks, ensuring comprehensiveness and accuracy of feature extraction.

[0036] Step S340, encoding and extracting the standard optical image data based on the optical coding extraction formula to obtain the optical modality embedding feature set. Specifically, the standard optical image data is used as input and sent to the optical encoder, and the image is divided into multiple image blocks according to the optical coding extraction formula, and feature extraction is performed on each image block respectively, and the features of each image block are balanced using a weight operator to obtain the optical modality embedding feature set. This implementation method can process sonar images and optical images simultaneously through a parallel dual-branch structure, thereby improving detection efficiency. The convolution-Transformer structure combines the advantages of CNN and Transformer, and can not only process local features but also capture global dependencies. This enables the network to adapt to seabed environments and submarine cable pipeline morphologies of different complexities, thereby improving detection accuracy and robustness.

[0037] Step S400: performing attention perception and weighted fusion on the acoustic modality embedding feature set and the optical modality embedding feature set to obtain acoustic-optical multimodal fusion-level features.

[0038] Specifically, feature fusion is performed using an attention mechanism, which simulates human attention to information and improves the accuracy of feature extraction and fusion. Attention is applied to the acoustic and optical modality embedding feature sets, and the weight of each feature is calculated. The features are then weighted and fused based on the weights to obtain acoustic-optical multimodal fusion-level features.

[0039] In one possible implementation, the step of obtaining the acoustic-optical multimodal fusion-level features, step S400 further includes step S410, performing attention perception on the acoustic modal embedding feature set and the optical modal embedding feature set based on a dynamic attention fusion network, and obtaining a multimodal feature attention map. Specifically, the dynamic attention fusion network is a network structure that can dynamically assign different weights according to the importance of input features, and is used to help the model pay attention to more valuable information in different modalities. Based on the dynamic attention fusion network, the acoustic modal embedding feature set and the optical modal embedding feature set are subjected to attention perception to generate a multimodal feature attention map, which reflects the importance of different feature areas in different modalities.

[0040] Step S420: construct a feature alignment formula: ,in Represents the KL divergence loss function used, and its corresponding inputs are and , which is the first feature regions and optical feature embedding feature regions, where N is the total number of feature regions. Specifically, a feature alignment formula is constructed and the KL divergence loss function is used to minimize the difference between the acoustic feature embedding and the optical feature embedding, so that they are aligned in space and channels, ensuring that features of different modalities can be compared in the same feature space.

[0041] Step S430: Performing a feature alignment operation on the acoustic modal embedding feature set and the optical modal embedding feature set based on the feature alignment formula. Specifically, performing a feature alignment operation on the acoustic modal embedding feature set and the optical modal embedding feature set based on the feature alignment formula minimizes the difference between the acoustic feature embedding and the optical feature embedding by optimizing the KL divergence loss function, enabling comparison and fusion in the same feature space.

[0042] Step S440: Construct a multimodal feature fusion formula: ,in, is the optical modality embedding feature set, is the acoustic modal embedding feature set, Specifically, the multimodal feature fusion formula is used to fuse features from different modalities according to certain rules or weights to generate a comprehensive feature representation, and the fusion is performed based on the weight graph.

[0043] Step S450, using the multimodal feature attention map as a weight map, based on the feature fusion formula, the acoustic modal embedding feature set and the optical modal embedding feature set after feature alignment are fused to obtain the acoustic-optical multimodal fusion-level feature. Specifically, using the multimodal feature attention map as a weight map, based on the multimodal feature fusion formula, the acoustic modal embedding feature set and the optical modal embedding feature set after feature alignment are fused to generate a comprehensive acoustic-optical multimodal fusion-level feature, which contains both acoustic modal information and optical modal information, and the information is weighted according to their importance. This implementation method enables the model to capture more details and features by fusing information from different modalities, thereby improving the accuracy of submarine cable pipeline detection.

[0044] In a possible implementation, the step S410 of obtaining the multimodal feature attention map further includes a step S411 of constructing an attention fusion network formula according to the dynamic attention fusion network as follows: ,in, Represents the attention map corresponding to the acoustic modality embedding feature set, represents the attention map corresponding to the optical modality embedding feature set, represents the maximum pooling layer in deep learning, represents the average pooling layer in deep learning, Indicates that the convolution kernel size is Specifically, the attention fusion network formula captures the correlation and importance between the acoustic and optical modal embedding feature sets by computing the attention map between the two modal embedding feature sets. The maximum pooling layer and the average pooling layer are used to extract global information about features from different angles, while the convolutional layer is used to adjust the feature dimension and fuse the features from the maximum pooling layer and the average pooling layer.

[0045] Step S412: Attention is perceived on the acoustic modality embedding feature set and the optical modality embedding feature set based on the attention fusion network formula to obtain an acoustic embedding feature attention map and an optical embedding feature attention map. Specifically, the acoustic modality embedding feature set and the optical modality embedding feature set are respectively input into the attention fusion network formula, global information is extracted through a maximum pooling layer and an average pooling layer, and features from the two pooling layers are fused using a convolutional layer to generate an acoustic embedding feature attention map and an optical embedding feature attention map.

[0046] Step S413, the acoustic embedded feature attention map and the optical embedded feature attention map are superimposed, fused and channel-confused to obtain the multimodal feature attention map. Specifically, the acoustic embedded feature attention map and the optical embedded feature attention map are superimposed in the channel dimension, and the channel confusion operation is performed on the superimposed feature map, that is, the channel order of the feature map is disrupted to further enhance the interaction and fusion effect of the features, and finally, a multimodal feature attention map is obtained, which reflects the correlation and importance between the acoustic modal and optical modal features. This implementation method can capture the correlation and importance between the two modal features by calculating the attention map between the acoustic modal and optical modal features, thereby effectively utilizing multimodal information to improve detection accuracy.

[0047] In a possible implementation, the step of obtaining the multimodal feature attention map further includes step S4131, constructing a superposition fusion formula: ,in, represents the consistent attention map of superposition fusion, represents the fusion weight of the acoustic embedding feature attention map, represents the fusion weight of the optical embedding feature attention map, which is dynamically updated during the learning process. Specifically, the fusion weight is dynamically updated during the learning process to balance the contributions of acoustic and optical modal features in the superposition fusion process.

[0048] Step S4132: The acoustic embedding feature attention map and the optical embedding feature attention map are superimposed and fused based on the superposition fusion formula to generate a consistent attention map. Specifically, the acoustic embedding feature attention map and the optical embedding feature attention map are used as inputs, substituted into the superposition fusion formula, and the superimposed and fused consistent attention map is calculated based on the dynamically updated fusion weights.

[0049] Step S4133: Construct a channel confusion formula: ,in, Represents multiple groups of convolution layers with 7 convolution kernels. is the activation function used in deep learning, Specifically, the channel confusion formula is used to enhance the interaction and fusion of features, further explore more valuable multimodal information in the consistent attention map, and avoid noise interference caused by modal fusion. Through multiple sets of convolution-activation operations, the channels of the consistent attention map are shuffled and recombined. After each convolution operation, the activation function is used to increase nonlinearity. In the formula, Indicates channel splicing.

[0050] In step S4134, channel confusion is performed on the consistent attention map based on the channel confusion formula to obtain the multimodal feature attention map. Specifically, the consistent attention map is used as input and substituted into the channel confusion formula. Through multiple sets of convolution-activation operations, the channels of the input features are shuffled and recombined until the final multimodal feature attention map is obtained. This implementation method obtains a more accurate and comprehensive multimodal feature attention map through superposition fusion and channel confusion, thereby improving the accuracy and robustness of submarine cable pipeline detection.

[0051] Step S500: decoding the acousto-optic multimodal fusion level features through a multimodal decoder structure to obtain acousto-optic fusion pixel-level features.

[0052] Specifically, decoding is performed using a multimodal decoder structure, which refers to a network structure capable of processing and decoding features from multiple modalities. The acousto-optic multimodal fusion-level features are input into the multimodal decoder structure, which then decodes the features to obtain acousto-optic fusion pixel-level features, i.e., features that reflect the information of each pixel in the image.

[0053] In one possible implementation, the step S500 of obtaining the acousto-optic fusion pixel-level features further includes a step S510 of introducing differential convolution using the multimodal decoder structure to obtain a feature decoding process, wherein the feature decoding process is specifically as follows: ,in, is a conventional Vanilla convolution operation, Represents multi-difference convolution operations, including center difference and angular difference convolution operations. Specifically, the conventional Vanilla convolution operation is a standard convolution operation used to extract local features in the feature map. Differential convolution is a special convolution operation that captures finer spatial structure information by calculating the differences between adjacent pixels (or pixels in a specific direction) in the input feature map. Multi-difference convolution operations include center difference and angular difference convolution operations. Center difference convolution calculates the difference between the center pixel and its neighboring pixels, and angular difference convolution focuses on capturing the features of image edges or corners. Combining the conventional Vanilla convolution operation with the multi-difference convolution operation forms a feature decoding process that can more comprehensively capture and process multi-scale and multi-directional information in the image.

[0054] Step S520: Construct a re-parameterization process: ,in, Represents the convolution kernels corresponding to various convolution operations. Specifically, by constructing a reparameterization process, multiple convolution operations (including regular convolution and differential convolution) are integrated into a unified framework. Reparameterization technology allows the use of multiple convolution kernels for feature extraction during training, while these convolution kernels are merged into one during inference to reduce computational complexity and memory consumption, thereby simplifying the model structure without sacrificing performance. The convolution kernels corresponding to the various convolution operations include kernels for regular convolution and kernels for differential convolution.

[0055] Step S530, based on the feature decoding process and the reparameterization process, the acoustic-optical multimodal fusion level feature is feature decoded to obtain the acoustic-optical fusion pixel-level feature. Specifically, the feature decoding process in step S510 and the reparameterization process in step S520 are combined to decode the acoustic-optical multimodal fusion level feature to obtain the acoustic-optical fusion pixel-level feature. This implementation method can capture the spatial structure information in the image more finely by introducing differential convolution, thereby enhancing the feature extraction capability. The application of reparameterization technology allows the use of multiple convolution kernels for feature extraction during training, and simplifies the model structure during inference, reduces the amount of calculation and memory consumption, and optimizes computational efficiency.

[0056] Step S600: Use the detection regression head and the classification head to detect and classify the acousto-optic fusion pixel-level features to obtain the submarine cable pipeline detection results.

[0057] Specifically, detection and classification are performed using a detection regression head and a classification head. The detection regression head refers to the network component used for position regression and can determine the target's position in the image. The classification head refers to the network component used for classification and can determine the target's type or status. The acoustic-optical fusion pixel-level features are input into the detection regression head and the classification head. The detection regression head performs position regression on the features to determine the location of the submarine cable pipeline. The classification head classifies the features to determine the type or status of the submarine cable pipeline. The detection results, including information such as the location, type, and status of the submarine cable pipeline, are output. The embodiment of the present application utilizes underwater sensor equipment to simultaneously collect acoustic image and optical image data, preprocesses the collected data, extracts high-quality sonar rectangular image data and standard optical image data, and uses a parallel dual-branch convolution-Transformer network to extract features of the two modal data. Through attention perception and weighted fusion mechanism, the acoustic modal embedding feature set and the optical modal embedding feature set are fused to obtain acoustic-optical multimodal fusion-level features, and the fused features are decoded using a multimodal decoder structure to obtain acoustic-optical fusion pixel-level features. Detection and classification are performed through detection regression head and classification head, and other technical means. By fusing the data of the two modalities of acoustic image and optical image, the complementary advantages of the two modalities in representing submarine cable pipelines are fully utilized, thereby achieving the technical effect of improving detection accuracy and enhancing detection robustness.

[0058] In a possible implementation, the step S600 of obtaining the submarine cable pipeline detection result further includes a step S610 of constructing a prediction box regression function based on the detection regression head: ,in, Represents the center point b of the predicted box and the center point of the real box The Euclidean distance, The length of the diagonal representing the minimum circumscribed moment between two center points, Represents the intersection-over-union ratio of the predicted box and the true box, Representation parameters The corresponding impact factor, Represents a parameter that measures aspect ratio consistency. Specifically, the prediction box regression function calculates the difference between the predicted box (i.e., the target location box predicted by the model) and the ground-truth box (i.e., the actual target location box) and minimizes this difference. The Euclidean distance between the center point of the predicted box and the center point of the ground-truth box reflects the positional deviation. The length of the minimum circumscribed moment diagonal measures the scale deviation between the predicted box and the ground-truth box. The ratio of the intersection area of ​​the predicted box to the union area of ​​the predicted box and the ground-truth box reflects the degree of shape matching between the two.

[0059] Step S620: Anchor box detection is performed on the acousto-optic fusion pixel-level features based on the prediction box regression function to obtain a target feature prediction box. Specifically, anchor box detection is performed on the acousto-optic fusion pixel-level features using the constructed prediction box regression function. Anchor boxes are a series of pre-set rectangular boxes of different scales and aspect ratios that serve as the starting point for target detection. By calculating the difference between each anchor box and the true box and applying the prediction box regression function for optimization, the target feature prediction box is ultimately obtained.

[0060] Step S630: construct a target classification process based on the classification header. The target classification process is specifically as follows: ,in, Indicates the The probability that an image belongs to each category, is the fully connected layer, is a classification function. Specifically, the target classification process is used to classify the target feature prediction box. This process includes a fully connected layer for feature extraction and a classification function for mapping features to class probabilities. By calculating the probability that each image belongs to each class, the target feature prediction box is classified.

[0061] Step S640, classify and determine the target feature prediction frame based on the target classification process to obtain the submarine cable pipeline detection result. Specifically, the target feature prediction frame is classified and determined using the constructed target classification process, and the features of the target feature prediction frame are input into the fully connected layer. After the features are extracted, the probabilities of each category are calculated by the classification function. Then, according to the set threshold or the maximum probability principle, the category to which the target feature prediction frame belongs is determined, thereby obtaining the submarine cable pipeline detection result. This implementation method optimizes the position and scale of the prediction frame by constructing a prediction frame regression function, so that it more accurately approaches the real frame; and classifies and determines the prediction frame by constructing a target classification process to determine the category to which it belongs. This combined method not only ensures the accuracy of detection, but also improves the robustness of detection.

[0062] The above specific embodiments do not constitute a limitation to the scope of protection of this application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of this application should be included in the scope of protection of this application. In some cases, the actions or steps recorded in this application can be performed in an order different from that in the embodiments and can still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A submarine cable pipeline detection method based on underwater acoustic and optical fusion, characterized in that: The method comprises: Acquire underwater sonar image data and underwater optical image data containing the target submarine cable pipeline through underwater sensor equipment; Preprocessing and extracting the underwater sonar image data and the underwater optical image data respectively to obtain sonar rectangular image data and standard optical image data; Using a parallel dual-branch convolution-Transformer network to perform feature extraction on the sonar rectangular image data and the standard optical image data, respectively, to obtain an acoustic modal embedding feature set and an optical modal embedding feature set; Performing attention perception and weighted fusion on the acoustic modality embedding feature set and the optical modality embedding feature set to obtain acoustic-optical multimodal fusion-level features; Decoding the acousto-optic multimodal fusion level features through a multimodal decoder structure to obtain acousto-optic fusion pixel-level features; Use the detection regression head and the classification head to detect and classify the acoustic-optical fusion pixel-level features to obtain submarine cable pipeline detection results; The obtaining of the acoustic-optical multimodal fusion-level features includes: Performing attention perception on the acoustic modality embedding feature set and the optical modality embedding feature set based on a dynamic attention fusion network to obtain a multimodal feature attention map; Construct feature alignment formula: ,in Represents the KL divergence loss function used, and its corresponding inputs are and , which is the first feature regions and optical feature embedding feature areas, is the total number of feature regions; Performing a feature alignment operation on the acoustic mode embedding feature set and the optical mode embedding feature set based on the feature alignment formula; Construct multimodal feature fusion formula: ,in, is the optical modality embedding feature set, is the acoustic modal embedding feature set, is the weight graph; Using the multimodal feature attention map as a weight map, the acoustic modality embedding feature set and the optical modality embedding feature set after feature alignment are fused based on the feature fusion formula to obtain the acoustic-optical multimodal fusion level feature; The obtaining of the multimodal feature attention map includes: According to the dynamic attention fusion network, the attention fusion network formula is constructed as follows: , ,in, represents the attention map corresponding to the acoustic modality embedding feature set, represents the attention map corresponding to the optical modality embedding feature set, represents the maximum pooling layer in deep learning, represents the average pooling layer in deep learning, Indicates that the convolution kernel size is Convolutional layers; Performing attention perception on the acoustic modality embedding feature set and the optical modality embedding feature set based on the attention fusion network formula to obtain an acoustic embedding feature attention map and an optical embedding feature attention map; Performing superposition fusion and channel confusion on the acoustic embedding feature attention map and the optical embedding feature attention map to obtain the multimodal feature attention map; The obtaining of the multimodal feature attention map comprises: Construct the superposition fusion formula: ,in, represents the consistent attention map of superposition fusion, represents the fusion weight of the acoustic embedding feature attention map, represents the fusion weight of the optical embedding feature attention map, which is dynamically updated during the learning process; Superimposing and fusing the acoustic embedding feature attention map and the optical embedding feature attention map based on the superposition fusion formula to generate a consistent attention map; Construct channel confusion formula: ,in, Represents multiple groups of convolution layers with a convolution kernel of 7. is the activation function used in deep learning, ; Channel confusion is performed on the consistent attention map based on the channel confusion formula to obtain the multimodal feature attention map.

2. The submarine cable pipeline detection method based on underwater sound and light fusion according to claim 1 is characterized in that: The obtaining of sonar rectangular image data and standard underwater optical image data comprises: Construct a rectangular image transformation formula: ,in, They represent the input underwater sonar image data and the preprocessed sonar rectangular image data, respectively. are the height and width of the image, Represents the pre-processing reconstruction algorithm set, which includes angle conversion , traverse the projection matrix ; Performing window conversion preprocessing on the underwater sonar image data based on the rectangular image conversion formula to obtain the sonar rectangular image data; Building a lightweight underwater image enhancement algorithm: ,in, represents the pre-processed standard optical image data, is an activation function for deep learning. and It is a convolution layer with a convolution kernel size of 7×7 and a maximum pooling layer. Represents the input underwater optical image data expanded along the RGB three-color space; The underwater optical image data is preprocessed based on the lightweight underwater image enhancement algorithm to obtain the standard optical image data.

3. The submarine cable pipeline detection method based on underwater sound and light fusion according to claim 2 is characterized in that: The obtaining of the acoustic mode embedding feature set and the optical mode embedding feature set comprises: The acoustic code extraction formula is constructed by using the acoustic encoder in the parallel dual-branch convolution-Transformer network as follows: ,in, represents the embedding vector extracted by the convolution-Transformer structure, is the sonar rectangular image data, is the corresponding result of the embedding vector after the acoustic encoder, Represents the convolutional-Transformer used for inference, where the encoder contains fully connected layers , represents the activation function for deep learning, is a scalar parameter used to scale the output of softmax; Encoding and extracting the sonar rectangular image data based on the acoustic coding extraction formula to obtain the acoustic modal embedding feature set; The optical code extraction formula is constructed by using the optical encoder in the parallel dual-branch convolution-Transformer network as follows: ,in, 、 、 Respectively represent the first and The feature representation matrix on up to the Nth image block is obtained by The corresponding result generated after the optical encoder is 、 and , Represents a weight operator, used to balance multiple image blocks; The standard optical image data is coded and extracted based on the optical coding extraction formula to obtain the optical modality embedding feature set.

4. The method for detecting submarine cables and pipelines based on underwater acoustic and optical fusion according to claim 1, wherein: The obtaining of the acousto-optic fusion pixel-level features includes: The multimodal decoder structure is used to introduce differential convolution to obtain a feature decoding process, which is specifically as follows: ,in, is a conventional Vanilla convolution operation, Represents multiple differential convolution operations, including central differential and angular differential convolution operations; Build a reparameterized process: ,in, Represents the convolution kernels corresponding to various convolution operations; The acousto-optic multimodal fusion level feature is feature decoded based on the feature decoding process and the reparameterization process to obtain the acousto-optic fusion pixel level feature.

5. The method for detecting submarine cables and pipelines based on underwater acoustic and optical fusion according to claim 4, characterized in that: The obtaining of submarine cable pipeline detection results includes: Based on the detection regression head, a prediction box regression function is constructed: ,in, Represents the center point b of the predicted box and the center point of the real box The Euclidean distance, The length of the diagonal representing the minimum circumscribed moment between two center points, Represents the intersection-over-union ratio of the predicted box and the true box, Representation parameters The corresponding impact factor, It represents the parameter that measures the consistency of aspect ratio; Perform anchor frame detection on the acousto-optic fusion pixel-level features based on the prediction frame regression function to obtain a target feature prediction frame; According to the classification header, a target classification process is constructed. The target classification process is specifically as follows: ,in, Indicates the The probability that an image belongs to each category, is the fully connected layer, is the classification function; The target feature prediction frame is classified and determined based on the target classification process to obtain the submarine cable pipeline detection result.

Images