A shape-adaptive pipeline detection method for side-scan sonar images

Abstract

The present invention belongs to the technical field of underwater detection and pipeline detection, and specifically relates to a shape-adaptive side-scan sonar image pipeline detection method, comprising the following steps: 1) acquiring side-scan sonar pipeline image data to be detected and sending it to a feature extraction module for feature extraction; 2) constructing the feature extraction module to gradually extract feature vector data from the side-scan sonar pipeline image data and send the data to a transformer module; 3) establishing a transformer model and modifying an introduced object query vector to obtain the modified object query vector and a detection result generated by the feature extraction module; 4) introducing a transformer model with a feedback mechanism, outputting a q_s vector as a supplement to the object query vector; obtaining an updated object query vector and inputting it again into the transformer model; 5) the transformer model outputting whether a pipeline target exists in the detection image and m coordinate points corresponding to the pipeline target; and 6) obtaining a boundary map of the detected pipeline target based on the m output coordinate points of the pipeline target.

Description

Technical Field

[0001] The invention belongs to the technical field of underwater detection and pipeline detection, and in particular to a pipeline detection method using adaptive shape side-scan sonar images. Background Art

[0002] Side-scan sonar technology, an important means of detecting seafloor topography and obstacles, is widely used in fields such as marine engineering, underwater archaeology, and pipeline inspection. By transmitting sound waves and receiving reflected signals from the seafloor, this technology generates high-resolution two-dimensional images, revealing detailed features of the seafloor topography, pipeline layout, and surrounding environment. However, in practical applications, the interpretation and processing of side-scan sonar images still face numerous limitations, hindering further improvements in detection efficiency and accuracy.

[0003] Traditional side-scan sonar image analysis methods rely primarily on manual visual interpretation. This approach requires professionals with extensive experience and expertise to meticulously analyze the images to accurately identify the location, morphology, and potential defects of pipelines. However, manual interpretation is not only time-consuming and labor-intensive, but also susceptible to subjective factors, leading to instability and uncertainty in detection results. Furthermore, automated processing methods based on target detection algorithms have also been applied to side-scan sonar image analysis to a certain extent. However, when detecting targets such as pipelines, these methods typically only output a bounding box of the target and are unable to accurately depict the target's actual shape and location. This box-like detection result has significant limitations. Not only is the position inaccurate and susceptible to the complex seabed environment (such as mud cover and biological adhesion), but it also cannot adapt to changes in the target's morphology (such as bends and branches in the pipeline), further reducing the accuracy and reliability of the detection results.

[0004] Therefore, a method is urgently needed to more efficiently and accurately interpret side-scan sonar images, automatically identifying and precisely delineating the shape and location of targets such as submarine pipelines. This method should eliminate the need for manual assistance and external equipment, enabling an automated and continuous inspection process to meet the growing demand for submarine exploration. Furthermore, the method should be highly stable and reliable, adaptable to complex and changing submarine environments, and provide strong support for submarine pipeline safety monitoring and condition assessment. Summary of the Invention

[0005] The present invention aims to provide a shape-adaptive pipeline detection method for side-scan sonar images. This method directly outputs the detection results from the image, including the presence of a target and multiple coordinates of the target. By connecting these coordinates, a boundary map of the detected target can be obtained.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned object is: a method for detecting pipelines using side-scan sonar images with adaptive shapes, comprising the following steps:

[0007] 1) The side-scan sonar obtains the pipeline image data to be detected and sends it to the feature extraction module for feature extraction, and then sends the extracted pipeline image data to the transformer module;

[0008] 2) Construct a feature extraction module using a convolutional neural network. The input image is processed layer by layer through the convolutional layer and pooling layer structure. Feature vector data is gradually extracted from the input side-scan sonar pipeline image data and sent to the transformer module.

[0009] 3) Establishing a transformer model through the transformer module and modifying the introduced Object Query vector to obtain the modified Object Query vector and the detection results generated by the feature extraction module;

[0010] 4) The modified Object Query vector and the detection results generated by the feature extraction module are input into the transformer model with the feedback mechanism. The output q_s vector is used as a supplement to the Object Query vector. The updated Object Query vector is obtained and input into the transformer model again.

[0011] 5) The transformer model outputs the presence of a pipeline target in the detection image and the m coordinate points corresponding to the pipeline target;

[0012] 6) Based on the output m coordinate points of the pipeline target, a boundary map of the detected pipeline target is obtained through a connection operation.

[0013] The shape of the modified Object Query vector is adjusted to a combination of the number of objects, the number of key points of each object, and the corresponding feature dimensions.

[0014] The step 2) is specifically as follows:

[0015] The transformer model includes: an encoder and a decoder;

[0016] The encoder is used to process the input feature vector data and convert it into a series of encoded vectors;

[0017] The feature vector data includes: low-level features of color, texture, and shape in the image, as well as high-level semantic features;

[0018] The decoder generates detection results based on the encoding vector and the subsequently introduced Object Query.

[0019] In step 3), the introduced Object Query vector is modified to obtain a modified ObjectQuery vector, specifically:

[0020] 1-1) Determine the number of pipeline targets to be processed, the number of key points for each pipeline target, and the feature dimensions of each key point;

[0021] Among them, we determine the number of targets, that is, the number of pipes that may appear in the image; the number of key points of each target, that is, the endpoints, midpoints and set landmarks of the pipe; the feature dimension, which is the length of the vector used to represent the target feature;

[0022] 1-2) Calculate the number of targets and total number of key points:

[0023] Calculate the product of the number of targets and the number of key points of each target to get the total number of key points to determine the length of the new Object Query vector shape;

[0024] The Object Query is a learnable vector used to guide the feature extraction module to generate detection results;

[0025] The step 4) is specifically as follows:

[0026] 2-1) Input the modified Object Query into the transformer model. The output of the transformer model is connected to a trainable MLP module to output the q_s vector.

[0027] The shape of the output q_s vector is: (number of targets, number of key points, number of features);

[0028] 2-2) Directly add the q_s vector to the original Object Query vector to form an updated ObjectQuery vector, which is then fed back into the transformer model.

[0029] The addition operation is performed element-wise, that is, each element in q_s will be added to the original ObjectQuery element at the corresponding position. Then, the original ObjectQuery will be updated to a new vector containing more information about the target and its key points. The updated ObjectQuery will be input into the Transformer model again for iterative processing.

[0030] In step 5), the transformer model output detects whether there is a pipeline target in the image, specifically:

[0031] 3-1) Output feature vector f of the transformer model; input feature vector f into the MLP module, change the dimension of feature vector f, and convert the transformer output vector to a dimension that matches the number of categories. Here, the output is a vector of shape (number of targets, 2);

[0032] Among them, the 2 in (number of targets, 2) corresponds to two categories, namely: the target is a pipeline and is not a pipeline;

[0033] 3-2) The output vector (target number, 2) is passed to the Softmax layer, which converts the output into a probability distribution so that the sum of the two probabilities of the target output is 1, obtaining the final classification prediction.

[0034] In step 6), the transformer model outputs m coordinate points corresponding to the pipeline target, specifically:

[0035] 4-1) Output feature vector f of the transformer model; input feature vector f into another MLP module in parallel with the classification task to change the dimension;

[0036] 4-2) Convert the feature vector f output by the transformer model to a dimension that matches the number of keypoint coordinates, and output a vector of shape (number of targets, number of keypoints, coordinate values); where the coordinate values ​​refer to the x-coordinate and y-coordinate, thus obtaining the final coordinate point prediction.

[0037] The shape of the feature vector f output by the transformer model is: (number of targets, number of key points, number of features).

[0038] An object detection model for a shape-adaptive side-scan sonar image pipeline detection method, comprising: a feature extraction module, a Transformer module, and an output layer;

[0039] The feature extraction module is used to extract feature information including low-level features such as color, texture, and shape in the input side-scan sonar pipeline image, as well as higher-level semantic features; the output feature map is sent to the Transformer module;

[0040] The Transformer module processes the feature map through the self-attention mechanism, outputs the feature vector f, and inputs it into the MLP module. It also receives the updated Object Query vector and generates the final feature vector f to the output layer to determine the coordinate points of the pipeline target and whether the pipeline target exists.

[0041] The MLP module is used to introduce a feedback mechanism. The input is the transformer’s output feature vector f with the shape of (number of targets, number of key points, number of features). The output is a q_s vector as a complement to the q vector, thus achieving a comprehensive feature representation of the feature map.

[0042] The output layer is used to directly predict the coordinate points of the pipeline target and whether the pipeline target exists based on the output of the encoder and decoder of the Transformer module.

[0043] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for detecting pipelines using side-scan sonar images with adaptive shapes is implemented.

[0044] The present invention has the following beneficial effects and advantages:

[0045] 1. The present invention can directly output the detection results of the image, including whether the target exists and multiple coordinate points of the target. By connecting these coordinate points, the boundary map of the detected target can be obtained.

[0046] 2. Based on the DETR (Detection Transformer) algorithm, the present invention modifies the shape of the Object Query and expands the feature dimension to achieve effective detection of targets and their key points. It also adds a feedback mechanism to improve the detection accuracy.

[0047] 3. The present invention realizes effective detection of targets and their key points by modifying the shape of Object Query, expanding the feature dimension and introducing a feedback mechanism, which significantly improves the accuracy and robustness of detection.

[0048] 4. The method of the present invention can directly output the coordinate points of the target without the need for subsequent complex post-processing steps, thereby simplifying the detection process and improving detection efficiency.

[0049] 5. By introducing key point reference points and explicit supervision strategies, the present invention accelerates the convergence speed of the model and improves training efficiency.

[0050] 6. When detecting a target, the method of the present invention can simultaneously consider the overall shape and key point information of the target, and continuously optimize the detection results in combination with a feedback mechanism, thereby more accurately depicting the boundary map of the target, providing strong support for subsequent image analysis and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flow chart of a method for pipeline detection using adaptive shape side-scan sonar images according to the present invention;

[0052] Figure 2Schematic diagram of the principle of Object Query vector modification of the present invention;

[0053] Figure 3 An architecture diagram of a shape-adaptive side-scan sonar image pipeline detection system of the present invention;

[0054] Figure 4 The method of the present invention obtains a comparison image of the detection target and the conventional detection frame. DETAILED DESCRIPTION

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0056] like Figure 1 FIG. 1 is a flow chart of a method for detecting pipelines using a side-scan sonar image with an adaptive shape according to the present invention. The method comprises the following steps:

[0057] Step 1: Obtain the side-scan sonar image data to be detected;

[0058] Step 2: Build a feature extraction module. The feature extraction module is a pre-processing component of the detection model, responsible for extracting useful feature information from the input side-scan sonar pipeline imagery. This feature information typically includes low-level features such as color, texture, and shape, as well as higher-level semantic features. The feature extraction module can be implemented using deep learning models such as convolutional neural networks (CNNs). These models process the input image layer by layer through structures such as convolutional layers and pooling layers, gradually extracting feature information from the image.

[0059] Step 3: Build a transformer model. The model architecture typically consists of two parts: an encoder and a decoder. The encoder processes the input feature vector data and converts it into a series of encoded vectors; the decoder generates detection results based on these encoded vectors and the object query.

[0060] like Figure 2 The figure shows a schematic diagram of the principle of the Transformer model of the present invention to modify the Object Query vector, where the Object Query is a set of learnable vectors used to guide the decoder to generate detection results. In order to adapt to the complex shape and characteristics of pipe targets in side-scan sonar images, the Object Query is modified. To do this, it is necessary to determine the number of targets (i.e., the number of pipes that may appear in the image), the number of key points of each target (such as the endpoints and midpoints of the pipes), and the feature dimension (the length of the vector used to represent the target features). The shape of the Object Query is adjusted to a combination of the number of targets, the number of key points of each target, and the corresponding feature dimension.

[0061] Modify the imported Object Query vector to obtain the modified Object Query vector, specifically:

[0062] 1-1) Determine the number of pipeline targets to be processed, the number of key points for each pipeline target, and the feature dimensions of each key point;

[0063] Among them, we determine the number of targets, that is, the number of pipes that may appear in the image; the number of key points of each target, that is, the endpoints, midpoints and set landmarks of the pipe; the feature dimension, which is the length of the vector used to represent the target feature;

[0064] 1-2) Calculate the number of targets and total number of key points:

[0065] Calculate the product of the number of targets and the number of key points of each target to get the total number of key points to determine the length of the new Object Query vector shape;

[0066] The Object Query is a learnable vector used to guide the feature extraction module to generate detection results;

[0067] Step 4: After the image data is processed by the feature extraction module, feature vector data is generated. These feature vectors are then input into the encoder of the Transformer model to produce a series of encoded vectors. These encoded vectors and the modified object query are then input into the decoder to generate detection results. After the decoder generates preliminary detection results, these results are input into a trainable MLP (multi-layer perceptron) module. The MLP module further processes these preliminary detection results to extract more information about the object and its key points. Specifically, the MLP module outputs a vector q_s of the shape (number of objects, number of key points, number of features) as supplementary information to the object query. This information is used to subsequently optimize the detection results.

[0068] Step 5: Add the q_s output from the MLP module to the original object query. This addition is performed element-wise: each element in q_s is added to the corresponding element in the original object query. This updates the original object query to a new vector containing more information about the object and its keypoints. The updated object query is then fed back into the Transformer model for iterative processing.

[0069] Step 6: The model outputs detection results in two branches, corresponding to the classification task and the regression task. In the classification task, the model's goal is to identify whether a pipeline target exists in the image. The transformer model outputs a feature vector f with the shape of (number of targets, number of keypoints, number of features). f is fed into the output layer, which changes the dimension and transforms the transformer output vector to match the number of categories. The output is a vector with the shape of (number of targets, 2). The 2 corresponds to the two categories: pipeline and not pipeline. The output is then fed into the Softmax layer, which converts the output into a probability distribution so that the sum of the two probabilities of the target output is 1, resulting in the final classification prediction. f is fed into another output layer, which changes the dimension and transforms the transformer output vector f to match the number of keypoint coordinates. The output is a vector with the shape of (number of targets, number of keypoints, coordinate values), where the coordinate values ​​refer to the x and y coordinates. This results in the final coordinate point prediction.

[0070] Step 7: According to whether the output is a pipeline target and the coordinate point, the boundary map of the detection target is obtained through line operation, such as Figure 4 shown.

[0071] The improved target detection model is built based on the DETR algorithm, and the following key improvements are made:

[0072] (1) The shape of Object Query is modified to accommodate more information about the object and its key points;

[0073] (2) The feature dimension is expanded to better capture the detailed features of the target;

[0074] (3) A feedback mechanism is introduced to process the transformer output through the MLP module and combine it with the original ObjectQuery to further improve the detection accuracy;

[0075] (4) The output layer of the model is adjusted to directly output the coordinate points of the target without the need for subsequent complex post-processing steps.

[0076] like Figure 3 FIG. 1 is an architecture diagram of a shape-adaptive side-scan sonar image pipeline detection system according to the present invention. The target detection model of the shape-adaptive side-scan sonar image pipeline detection method according to the present invention includes: a feature extraction module, a Transformer module, and an output layer.

[0077] The feature extraction module is used to extract feature information including low-level features such as color, texture, and shape in the input side-scan sonar pipeline image, as well as higher-level semantic features; the output feature map is sent to the Transformer module;

[0078] The Transformer module processes the feature map through the self-attention mechanism, outputs the feature vector f, and inputs it into the MLP module. It also receives the updated Object Query vector and generates the final feature vector f to the output layer to determine the coordinate points of the pipeline target and whether the pipeline target exists.

[0079] The MLP module is used to introduce a feedback mechanism. The input is the transformer’s output feature vector f with the shape of (number of targets, number of key points, number of features). The output is a q_s vector as a complement to the q vector, thus achieving a comprehensive feature representation of the feature map.

[0080] The output layer is used to directly predict the coordinate points of the pipeline target and whether the pipeline target exists based on the output of the encoder and decoder of the Transformer module.

[0081] Furthermore, the modification of the Object Query and the implementation of the feedback mechanism include:

[0082] (1) Determine the number of targets, that is, the number of target types or instances that may appear in the image to be detected;

[0083] (2) Determine the number of key points for each object, which depends on the specific shape and characteristics of the object;

[0084] (3) Adjust the shape of the Object Query based on the number of targets and key points, and expand the feature dimension to match these requirements;

[0085] (4) The output of the transformer is connected to the MLP module to obtain q_s, which is used as a supplement to the Object Query and added to it to form the updated Object Query.

[0086] During the training process, the present invention also adopts the following strategies to improve the performance of the model:

[0087] (1) Using randomly initialized keypoint reference points as the initial positions, these reference points are continuously learned and updated during the training process;

[0088] (2) Explicitly supervise the positions of key points through key point reference points to accelerate the convergence of the model;

[0089] (3) The multi-head attention mechanism and cross-attention mechanism are used to enable the model to capture the feature information in the image more effectively.

[0090] Those skilled in the art will understand that the above description is only a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of the present disclosure may be combined or coupled in various ways, even if such a combination or coupling is not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

[0091] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A shape-adaptive side-scan sonar image pipeline detection method, characterized in that: The following steps are involved: 1) The side-scan sonar obtains the image data of the side-scan sonar pipeline to be detected and sends it to the feature extraction module for feature extraction; 2) Construct a feature extraction module using a convolutional neural network. The input image is processed layer by layer through convolutional and pooling layers. Feature vector data is gradually extracted from the input side-scan sonar pipeline image data and sent to the transformer module. 3) Build a transformer model through the transformer module and modify the introduced object query vector to obtain the modified object query vector and the result of feature vector data conversion; The shape of the modified Object Query vector is adjusted to a combination of the number of targets, the number of key points of each target, and the corresponding feature dimensions; 4) The modified Object Query vector and the result of the feature vector data conversion are input into the transformer model with the feedback mechanism. The output q_s vector is used as a supplement to the Object Query vector. The updated Object Query vector is obtained and input into the transformer model again. Step 4) is as follows: 2-1) Input the modified Object Query into the transformer model. The output of the transformer model is connected to a trainable MLP module to output the q_s vector. The shape of the output q_s vector is: (number of targets, number of key points, number of features); 2-2) Directly add the q_s vector to the original Object Query vector to form an updated Object Query vector, which is then fed back into the Transformer model. The addition operation is performed element-wise, that is, each element in q_s will be added to the original ObjectQuery element at the corresponding position. Then, the original ObjectQuery will be updated to a new vector containing more information about the target and its key points. The updated ObjectQuery will be input into the Transformer model again for iterative processing; 5) The transformer model outputs the presence of a pipeline target in the detection image and the m coordinate points corresponding to the pipeline target; 6) Based on the output m coordinate points of the pipeline target, a boundary map of the detected pipeline target is obtained through line connection operation.

2. The method for detecting pipelines using side-scan sonar images with adaptive shapes according to claim 1, wherein: The step 2) is specifically as follows: The transformer model includes: an encoder and a decoder; The encoder is used to process the input feature vector data and convert it into a series of encoded vectors; The feature vector data includes: low-level features of color, texture, and shape in the image, as well as high-level semantic features; The decoder generates detection results based on the encoding vector and the subsequently introduced Object Query.

3. The method for detecting pipelines using adaptive side-scan sonar images according to claim 2, wherein: In step 3), the introduced Object Query vector is modified to obtain a modified Object Query vector, specifically: 1-1) Determine the number of pipeline targets to be processed, the number of key points for each pipeline target, and the feature dimensions of each key point; Among them, we determine the number of targets, that is, the number of pipes that may appear in the image; the number of key points of each target, that is, the endpoints, midpoints and set landmarks of the pipe; the feature dimension, which is the length of the vector used to represent the target feature; 1-2) Calculate the number of targets and total number of key points: Calculate the product of the number of targets and the number of key points of each target to get the total number of key points to determine the length of the new Object Query vector shape; Object Query is a learnable vector used to guide the decoder to generate detection results.

4. The method for detecting pipelines using adaptive side-scan sonar images according to claim 1, wherein: In step 5), the transformer model output detects whether there is a pipeline target in the image, specifically: 3-1) Output feature vector f of the transformer model; input feature vector f into the MLP module, change the dimension of feature vector f, and convert the transformer output vector to a dimension that matches the number of categories. Here, the output is a vector with shape (number of targets, 2); Among them, the 2 in (number of targets, 2) corresponds to two categories, namely: the target is a pipeline and is not a pipeline; 3-2) The output vector (target number, 2) is passed to the Softmax layer. The Softmax layer converts the output into a probability distribution so that the sum of the two probabilities of the target output is 1, obtaining the final classification prediction.

5. The method for detecting pipelines using side-scan sonar images with adaptive shape according to claim 1, wherein: In step 5), the transformer model outputs m coordinate points corresponding to the pipeline target, specifically: 4-1) Output feature vector f of the transformer model; input feature vector f into another MLP module in parallel with the classification task to change the dimension; 4-2) Convert the feature vector f output by the transformer model to a dimension that matches the number of keypoint coordinates, and output a vector of shape (number of targets, number of keypoints, coordinate values); the coordinate values ​​refer to the x-coordinate and y-coordinate, thus obtaining the final coordinate point prediction.

6. The method for detecting pipelines using adaptive shape side-scan sonar images according to claim 4 or 5, characterized in that: The shape of the feature vector f output by the transformer model is: (number of targets, number of key points, number of features).

7. The target detection model of the shape-adaptive side-scan sonar image pipeline detection method according to claim 1, characterized in that: include: Feature extraction module, Transformer module, output layer; The feature extraction module is used to extract feature information including low-level features such as color, texture, and shape in the input side-scan sonar pipeline image, as well as higher-level semantic features; the output feature map is sent to the Transformer module; The Transformer module processes the feature map through the self-attention mechanism, outputs the feature vector f, and inputs it into the MLP module. It also receives the updated Object Query vector and generates the final feature vector f to the output layer to determine the coordinate points of the pipeline target and whether the pipeline target exists. The MLP module is used to introduce a feedback mechanism. The input is the transformer’s output feature vector f with the shape of (number of targets, number of key points, number of features). The output is a q_s vector as a complement to the q vector, thus achieving a comprehensive feature representation of the feature map. The output layer is used to directly predict the coordinate points of the pipeline target and whether the pipeline target exists based on the output of the encoder and decoder of the Transformer module.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the method for detecting pipelines using side-scan sonar images with adaptive shapes as described in any one of claims 1 to 6 is implemented.

Images