Shape-adaptive side-scan sonar image pipeline detection method

Abstract

The invention belongs to the technical field of underwater detection and pipeline detection, and particularly relates to a shape-adaptive side-scan sonar image pipeline detection method, which comprises the following steps of: 1) sending acquired side-scan sonar pipeline image data to be detected to a feature extraction module for feature extraction; 2) constructing a feature extraction module, gradually extracting feature vector data from the side-scan sonar pipeline image data, and sending the feature vector data to a transformer module; (3) a transform model is established, the introduced Object Query vector is modified, and the modified Object Query vector and a detection result generated by the feature extraction module are obtained; 4) by introducing a transform model of a feedback mechanism, taking an output qs vector as a supplement of an Object Query vector; obtaining an updated Object Query vector, and inputting the updated Object Query vector into the transform model again; (5) the transformer model outputs whether a pipeline target exists in the detection image or not and m coordinate points corresponding to the pipeline target; and 6) obtaining a boundary diagram of the detected pipeline target according to the m output coordinate points of the pipeline target.

Description

Technical Field

[0001] The invention belongs to the technical field of underwater detection and pipeline detection, and in particular is a pipeline detection method using side-scan sonar images with an adaptive shape. Background Art

[0002] Side-scan sonar technology, as an important means of detecting seabed topography and obstacles, is widely used in marine engineering, underwater archaeology, pipeline inspection and other fields. This technology generates high-resolution two-dimensional images by emitting sound waves and receiving reflected signals from the seabed, which can reveal the characteristics of the seabed topography, pipeline layout and its surrounding environment in detail. However, in practical applications, there are still many limitations in the analysis and processing of side-scan sonar images, which restricts the further improvement of its detection efficiency and accuracy.

[0003] Traditional side-scan sonar image analysis methods mainly rely on manual visual interpretation, which requires professionals with rich experience and expertise to conduct detailed analysis of the image to accurately identify the location, shape and potential defects of the pipeline. However, manual interpretation is not only time-consuming and laborious, but also easily affected by subjective factors, resulting in instability and uncertainty in the detection results. In addition, automated processing methods based on target detection algorithms have also been applied to side-scan sonar image analysis to a certain extent. However, when detecting targets such as pipelines, these methods can usually only output the bounding box of the target and cannot accurately depict the actual shape and position of the target. This frame-like detection result has obvious limitations. Not only is the position inaccurate, it is easily affected by the complex seabed environment (such as mud and sand coverage, biological adhesion, etc.), and it cannot adapt to changes in the target morphology (such as pipe bending, branching, etc.), further reducing the accuracy and reliability of the detection results.

[0004] Therefore, there is an urgent need for a method that can more efficiently and accurately analyze side-scan sonar images, automatically identify and accurately depict the shape and location of targets such as submarine pipelines. This method should be able to get rid of the dependence on manual assistance and external equipment, and realize an automated and continuous detection process to meet the growing demand for submarine detection. At the same time, this method should also have high stability and reliability, be able to adapt to the complex and changing submarine environment, and provide strong support for the safety monitoring and status assessment of submarine pipelines. Summary of the invention

[0005] The purpose of the present invention is to provide a pipeline detection method with adaptive shape of side scan sonar images. The method can directly output the detection results of the image, including whether the target exists and multiple coordinate points of the target. By connecting these coordinate points, the boundary map of the detected target can be obtained.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a method for detecting pipelines using side-scan sonar images with an adaptive shape, comprising the following steps:

[0007] 1) The side scan sonar obtains the side scan sonar pipeline image data to be detected and sends it to the feature extraction module for feature extraction, and sends the extracted pipeline image data to the transformer module;

[0008] 2) Construct a feature extraction module through a convolutional neural network, process the input image layer by layer through the convolution layer and pooling layer structure, gradually extract feature vector data from the input side scan sonar pipeline image data, and send it to the transformer module;

[0009] 3) Establishing a transformer model through the transformer module, and modifying the introduced Object Query vector, obtaining the modified Object Query vector and the detection result generated by the feature extraction module;

[0010] 4) The modified Object Query vector and the detection results generated by the feature extraction module are input into the transformer model with the feedback mechanism, and the output q_s vector is used as a supplement to the Object Query vector; the updated Object Query vector is obtained and input into the transformer model again;

[0011] 5) The transformer model outputs the detection of whether there is a pipeline target in the image and the m coordinate points corresponding to the pipeline target;

[0012] 6) According to the outputted m coordinate points of the pipeline target, a boundary map of the detected pipeline target is obtained by connecting lines.

[0013] The shape of the modified Object Query vector is adjusted to a combination of the number of objects, the number of key points of each object, and the corresponding feature dimensions.

[0014] The step 2) is specifically:

[0015] The transformer model includes: an encoder and a decoder;

[0016] The encoder is used to process the input feature vector data and convert it into a series of encoded vectors;

[0017] The feature vector data includes: low-level features of color, texture, shape and high-level semantic features in the image;

[0018] The decoder generates detection results based on the encoding vector and the subsequently introduced Object Query.

[0019] In step 3), the introduced Object Query vector is modified to obtain a modified ObjectQuery vector, specifically:

[0020] 1-1) Determine the number of pipeline targets that need to be processed, the number of key points of each pipeline target, and the feature dimension of each key point;

[0021] Among them, the number of targets is determined, that is, the number of pipes that may appear in the image; the number of key points of each target, that is, the endpoints, midpoints and set landmarks of the pipe; the feature dimension, which is the length of the vector used to represent the target feature;

[0022] 1-2) Calculate the number of targets and total number of key points:

[0023] Calculate the product of the number of targets and the number of key points of each target to get the total number of key points to determine the length of the new Object Query vector shape;

[0024] The Object Query is a learnable vector used to guide the feature extraction module to generate detection results;

[0025] The step 4) is specifically:

[0026] 2-1) Input the modified Object Query into the transformer model. The output of the transformer model is connected to a trainable MLP module to output the q_s vector.

[0027] The shape of the output q_s vector is: (number of targets, number of key points, number of features);

[0028] 2-2) The q_s vector is directly added to the original Object Query vector to form an updated ObjectQuery vector, which is input into the transformer model again;

[0029] The addition operation is performed element by element, that is, each element in q_s will be added to the original ObjectQuery element at the corresponding position, and then the original ObjectQuery will be updated to a new vector containing more information about the target and its key points. The updated ObjectQuery will be input into the Transformer model again for iterative processing.

[0030] In step 5), the transformer model output detects whether there is a pipeline target in the image, specifically:

[0031] 3-1) Output feature vector f of the transformer model; input feature vector f into the MLP module, change the dimension of feature vector f, and convert the output vector of the transformer to a dimension that matches the number of categories. Here, the output is a vector with a shape of (number of targets, 2);

[0032] Among them, the 2 in (number of targets, 2) corresponds to two categories, namely: the target is a pipeline and is not a pipeline;

[0033] 3-2) The output vector (target number, 2) is transmitted to the Softmax layer, which converts the output into a probability distribution so that the sum of the two probabilities of the target output is 1, and the final classification prediction is obtained.

[0034] In step 6), the transformer model outputs m coordinate points corresponding to the pipeline target, specifically:

[0035] 4-1) Output feature vector f of the transformer model; input feature vector f into another MLP module parallel to the classification task to change the dimension;

[0036] 4-2) Convert the feature vector f output by the transformer model into a dimension that matches the number of key point coordinates, and output a vector of shape (number of targets, number of key points, coordinate values); the coordinate values ​​refer to the x-coordinate and the y-coordinate, that is, the final coordinate point prediction is obtained.

[0037] The shape of the feature vector f output by the transformer model is: (number of targets, number of key points, number of features).

[0038] An object detection model of a side-scan sonar image pipeline detection method with an adaptive shape, comprising: a feature extraction module, a Transformer module, and an output layer;

[0039] The feature extraction module is used to extract the feature information of the underlying features including color, texture, shape, and higher-level semantic features from the input side-scan sonar pipeline image; the output feature map is sent to the Transformer module;

[0040] The Transformer module processes the feature map through the self-attention mechanism, outputs the feature vector f, and inputs it into the MLP module; at the same time, it receives the updated Object Query vector and generates the final feature vector f to the output layer to calculate the coordinate points of the pipeline target and determine whether there is a pipeline target;

[0041] The MLP module is used to introduce a feedback mechanism. The input is the feature vector f of the transformer output shape (number of targets, number of key points, number of features), and the output is the q_s vector as a supplement to the q vector to achieve a comprehensive feature representation of the feature map.

[0042] The output layer is used to directly predict the coordinate points of the pipeline target and whether the pipeline target exists based on the output of the encoder and decoder of the Transformer module.

[0043] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for detecting pipelines using side-scan sonar images with an adaptive shape is implemented.

[0044] The present invention has the following beneficial effects and advantages:

[0045] 1. The present invention can directly output the detection results of the image, including whether the target exists and multiple coordinate points of the target. By connecting these coordinate points, the boundary map of the detected target can be obtained.

[0046] 2. The present invention is based on the DETR (Detection Transformer) algorithm, modifies the shape of the Object Query, expands the feature dimension to achieve effective detection of the target and its key points, and also adds a feedback mechanism to improve the detection accuracy.

[0047] 3. The present invention realizes effective detection of targets and their key points by modifying the shape of Object Query, expanding feature dimensions and introducing a feedback mechanism, thereby significantly improving the accuracy and robustness of detection.

[0048] 4. The method of the present invention can directly output the coordinate points of the target without the need for subsequent complex post-processing steps, thereby simplifying the detection process and improving detection efficiency.

[0049] 5. By introducing key point reference points and explicit supervision strategies, the present invention accelerates the convergence speed of the model and improves the training efficiency.

[0050] 6. When detecting a target, the method of the present invention can simultaneously consider the overall shape and key point information of the target, and continuously optimize the detection results in combination with a feedback mechanism, thereby more accurately depicting the boundary map of the target, providing strong support for subsequent image analysis and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flow chart of a method for pipeline detection using adaptive shape side-scan sonar images of the present invention;

[0052] Figure 2Schematic diagram of the principle of Object Query vector modification of the present invention;

[0053] Figure 3 An architecture diagram of a side-scan sonar image pipeline detection system with an adaptive shape according to the present invention;

[0054] Figure 4 The method of the present invention obtains a comparison image of the detection target and the conventional detection frame. DETAILED DESCRIPTION

[0055] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0056] like Figure 1 As shown, it is a flow chart of a method for detecting pipelines using a side-scan sonar image with an adaptive shape according to the present invention. The method for detecting pipelines using a side-scan sonar image with an adaptive shape according to the present invention comprises the following steps:

[0057] Step 1: Obtain the side scan sonar image data to be detected;

[0058] Step 2: Build a feature extraction module. The feature extraction module is the pre-working part of the detection model, responsible for extracting useful feature information from the input side scan sonar pipeline image. This feature information usually includes low-level features such as color, texture, shape, and higher-level semantic features in the image. The feature extraction module can be implemented through deep learning models such as convolutional neural networks (CNN). These models process the input image layer by layer through structures such as convolutional layers and pooling layers, and gradually extract feature information from the image.

[0059] Step 3: Build a transformer model. The model architecture usually consists of two parts: an encoder and a decoder. The encoder is responsible for processing the input feature vector data and converting it into a series of encoded vectors; the decoder generates detection results based on these encoded vectors and the Object Query.

[0060] like Figure 2 As shown, the schematic diagram of the principle of the Transformer model of the present invention to modify the Object Query vector, wherein the Object Query is a set of learnable vectors used to guide the decoder to generate detection results. In order to adapt to the complex shape and features of the pipeline target in the side scan sonar image, the Object Query is modified. For this purpose, it is necessary to determine the number of targets (i.e., the number of pipelines that may appear in the image), the number of key points of each target (such as the endpoints and midpoints of the pipeline, etc.), and the feature dimension (the length of the vector used to represent the target feature). The shape of the Object Query is adjusted to a combination of the number of targets, the number of key points of each target, and the corresponding feature dimension;

[0061] The introduced Object Query vector is modified to obtain a modified Object Query vector, which is as follows:

[0062] 1-1) Determine the number of pipeline targets that need to be processed, the number of key points of each pipeline target, and the feature dimension of each key point;

[0063] Among them, the number of targets is determined, that is, the number of pipes that may appear in the image; the number of key points of each target, that is, the endpoints, midpoints and set landmarks of the pipe; the feature dimension, which is the length of the vector used to represent the target feature;

[0064] 1-2) Calculate the number of targets and total number of key points:

[0065] Calculate the product of the number of targets and the number of key points of each target to get the total number of key points to determine the length of the new Object Query vector shape;

[0066] The Object Query is a learnable vector used to guide the feature extraction module to generate detection results;

[0067] Step 4: After the image data is processed by the feature extraction module, feature vector data is obtained, which will be input into the encoder of the Transformer model to obtain a series of encoded vectors. Then, these encoded vectors and the modified Object Query will be input into the decoder together to generate detection results. After the decoder generates preliminary detection results, these results will be input into a trainable MLP (multi-layer perceptron) module. The MLP module will further process these preliminary detection results to extract more information about the target and its key points. Specifically, the MLP module will output a vector q_s of the shape of (number of targets, number of key points, number of features) as supplementary information to the Object Query. This information will be used for subsequent optimization of detection results.

[0068] Step 5: Add the q_s output by the MLP module to the original Object Query. The addition operation is performed element by element, that is, each element in q_s will be added to the original Object Query element at the corresponding position. In this way, the original Object Query will be updated to a new vector containing more information about the target and its key points. The updated Object Query will be input into the Transformer model again for iterative processing.

[0069] Step 6: The model outputs the detection results, which contain two branches, corresponding to the classification task and the regression task. In the classification task, the goal of the model is to identify whether there is a pipeline target in the image. The output of the transformer model is a feature vector f of the shape of (number of targets, number of key points, number of features). Input f into the output layer, change the dimension, and convert the output vector of the transformer to a dimension that matches the number of categories. Here, the output is a vector of the shape of (number of targets, 2). The 2 corresponds to two categories: pipeline and not pipeline. The output is then fed into the Softmax layer, which converts the output into a probability distribution so that the sum of the two probabilities of the target output is 1, and the final classification prediction is obtained. Input f into another output layer that is parallel to the classification task, change the dimension, and convert the output vector f of the transformer to a dimension that matches the number of coordinates of the key points. The output is a vector of the shape of (number of targets, number of key points, coordinate values), where the coordinate values ​​refer to the x and y coordinates. The final coordinate point prediction is thus obtained.

[0070] Step 7: According to whether the output is a pipeline target and the coordinate point, the boundary map of the detection target is obtained through line operation, such as Figure 4 shown.

[0071] The improved target detection model is built based on the DETR algorithm, and the following key improvements are made:

[0072] (1) The shape of the Object Query is modified to accommodate more information about the object and its key points;

[0073] (2) The feature dimension is expanded to better capture the detailed features of the target;

[0074] (3) A feedback mechanism is introduced to process the transformer output through the MLP module and combine it with the original ObjectQuery to further improve the detection accuracy;

[0075] (4) The output layer of the model is adjusted to directly output the coordinate points of the target without the need for subsequent complex post-processing steps.

[0076] like Figure 3 As shown, it is an architecture diagram of a side-scan sonar image pipeline detection system of the present invention, and a target detection model of a side-scan sonar image pipeline detection method of the present invention of an adaptive shape includes: a feature extraction module, a Transformer module, and an output layer;

[0077] The feature extraction module is used to extract the feature information of the underlying features including color, texture, shape, and higher-level semantic features from the input side-scan sonar pipeline image; the output feature map is sent to the Transformer module;

[0078] The Transformer module processes the feature map through the self-attention mechanism, outputs the feature vector f, and inputs it into the MLP module; at the same time, it receives the updated Object Query vector and generates the final feature vector f to the output layer to calculate the coordinate points of the pipeline target and determine whether there is a pipeline target;

[0079] The MLP module is used to introduce a feedback mechanism. The input is the feature vector f of the transformer output shape (number of targets, number of key points, number of features), and the output is the q_s vector as a supplement to the q vector to achieve a comprehensive feature representation of the feature map.

[0080] The output layer is used to directly predict the coordinate points of the pipeline target and whether the pipeline target exists based on the output of the encoder and decoder of the Transformer module.

[0081] Furthermore, the modification of the Object Query and the implementation of the feedback mechanism include:

[0082] (1) Determine the number of targets, that is, the number of target types or instances that may appear in the image to be detected;

[0083] (2) Determine the number of key points of each object, which depends on the specific shape and characteristics of the object;

[0084] (3) Adjust the shape of the Object Query based on the number of targets and key points, and expand the feature dimension to match these requirements;

[0085] (4) Connect the output of the transformer to the MLP module to obtain q_s, which is used as a supplement to the Object Query and added to it to form the updated Object Query.

[0086] During the training process, the present invention also adopts the following strategies to improve the performance of the model:

[0087] (1) Use randomly initialized key point reference points as the initial positions, which are continuously learned and updated during the training process;

[0088] (2) Explicitly supervise the positions of key points through key point reference points to accelerate the convergence of the model;

[0089] (3) The multi-head attention mechanism and cross-attention mechanism are used to enable the model to capture the feature information in the image more effectively.

[0090] Those skilled in the art will appreciate that the above are only preferred embodiments of the present invention, and the various embodiments of the present disclosure and / or the features described in the claims may be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments, or perform equivalent substitutions on some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0091] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A shape-adaptive side-scan sonar image pipeline detection method, characterized in that: The following steps are involved: 1) The side scan sonar obtains the side scan sonar pipeline image data to be detected and sends it to the feature extraction module for feature extraction; 2) Construct a feature extraction module through a convolutional neural network, process the input image layer by layer through the convolution layer and pooling layer structure, gradually extract feature vector data from the input side scan sonar pipeline image data, and send it to the transformer module; 3) Establishing a transformer model through the transformer module, and modifying the introduced Object Query vector, obtaining the modified Object Query vector and the detection result generated by the feature extraction module; 4) The modified Object Query vector and the detection results generated by the feature extraction module are input into the transformer model with the feedback mechanism, and the output q_s vector is used as a supplement to the Object Query vector; the updated Object Query vector is obtained and input into the transformer model again; 5) The transformer model outputs the detection of whether there is a pipeline target in the image and the m coordinate points corresponding to the pipeline target; 6) According to the outputted m coordinate points of the pipeline target, a boundary map of the detected pipeline target is obtained by connecting lines.

2. The method for detecting pipelines using side-scan sonar images with adaptive shapes according to claim 1, characterized in that: The shape of the modified Object Query vector is adjusted to a combination of the number of objects, the number of key points of each object, and the corresponding feature dimensions.

3. The method for detecting pipelines using side-scan sonar images with adaptive shapes according to claim 1, characterized in that: The step 2) is specifically: The transformer model includes: an encoder and a decoder; The encoder is used to process the input feature vector data and convert it into a series of encoded vectors; The feature vector data includes: low-level features of color, texture, shape and high-level semantic features in the image; The decoder generates detection results based on the encoding vector and the subsequently introduced Object Query.

4. The method for detecting pipelines using side-scan sonar images with adaptive shapes according to claim 1, characterized in that: In step 3), the introduced Object Query vector is modified to obtain a modified Object Query vector, specifically: 1-1) Determine the number of pipeline targets that need to be processed, the number of key points of each pipeline target, and the feature dimension of each key point; Among them, the number of targets is determined, that is, the number of pipes that may appear in the image; the number of key points of each target, that is, the endpoints, midpoints and set landmarks of the pipe; the feature dimension, which is the length of the vector used to represent the target feature; 1-2) Calculate the number of targets and total number of key points: Calculate the product of the number of targets and the number of key points of each target to get the total number of key points to determine the length of the new Object Query vector shape; The Object Query is a learnable vector used to guide the feature extraction module to generate detection results.

5. The method for detecting pipelines using side-scan sonar images with adaptive shapes according to claim 1, characterized in that: The step 4) is specifically: 2-1) Input the modified Object Query into the transformer model. The output of the transformer model is connected to a trainable MLP module to output the q_s vector. The shape of the output q_s vector is: (number of targets, number of key points, number of features); 2-2) The q_s vector is directly added to the original Object Query vector to form an updated Object Query vector, which is input into the transformer model again; The addition operation is performed element by element, that is, each element in q_s will be added to the original ObjectQuery element at the corresponding position, and then the original ObjectQuery will be updated to a new vector containing more information about the target and its key points. The updated ObjectQuery will be input into the Transformer model again for iterative processing.

6. The method for detecting pipelines using side-scan sonar images with adaptive shapes according to claim 1, characterized in that: In step 5), the transformer model output detects whether there is a pipeline target in the image, specifically: 3-1) Output feature vector f of the transformer model; input feature vector f into the MLP module, change the dimension of feature vector f, and convert the output vector of the transformer to a dimension that matches the number of categories. Here, the output is a vector with a shape of (number of targets, 2); Among them, the 2 in (number of targets, 2) corresponds to two categories, namely: the target is a pipeline and is not a pipeline; 3-2) The output vector (target number, 2) is transmitted to the Softmax layer, which converts the output into a probability distribution so that the sum of the two probabilities of the target output is 1, and the final classification prediction is obtained.

7. The method for detecting pipelines using side-scan sonar images with adaptive shapes according to claim 1, characterized in that: In step 5), the transformer model outputs m coordinate points corresponding to the pipeline target, specifically: 4-1) Output feature vector f of the transformer model; input feature vector f into another MLP module parallel to the classification task to change the dimension; 4-2) Convert the feature vector f output by the transformer model into a dimension that matches the number of key point coordinates, and output a vector of shape (number of targets, number of key points, coordinate values); the coordinate values ​​refer to the x-coordinate and the y-coordinate, that is, the final coordinate point prediction is obtained.

8. The method for detecting pipelines using side-scan sonar images with adaptive shapes according to claim 6 or 7, characterized in that: The shape of the feature vector f output by the transformer model is: (number of targets, number of key points, number of features).

9. The target detection model of the adaptive shape side scan sonar image pipeline detection method according to claim 1, characterized in that: include: Feature extraction module, Transformer module, output layer; The feature extraction module is used to extract the feature information of the underlying features including color, texture, shape, and higher-level semantic features from the input side-scan sonar pipeline image; the output feature map is sent to the Transformer module; The Transformer module processes the feature map through the self-attention mechanism, outputs the feature vector f, and inputs it into the MLP module; at the same time, it receives the updated Object Query vector and generates the final feature vector f to the output layer to calculate the coordinate points of the pipeline target and determine whether there is a pipeline target; The MLP module is used to introduce a feedback mechanism. The input is the feature vector f of the transformer output shape (number of targets, number of key points, number of features), and the output is the q_s vector as a supplement to the q vector to achieve a comprehensive feature representation of the feature map. The output layer is used to directly predict the coordinate points of the pipeline target and whether the pipeline target exists based on the output of the encoder and decoder of the Transformer module.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method for detecting pipelines using side-scan sonar images with an adaptive shape as described in any one of claims 1 to 8 is implemented.

Images