A method for processing submarine sonar images based on semantic segmentation

Abstract

The present invention discloses a method for processing submarine sonar images based on semantic segmentation, comprising the following steps: S1. obtaining a preprocessed sonar image block; S2. generating a priori map of submarine landform boundaries corresponding pixel by pixel to the preprocessed sonar image block; S3. constructing an improved U-Net dual-branch semantic segmentation network, comprising a semantic branch and a boundary branch, and configuring a landform mutual feedback gating module at each scale jump layer; S4. obtaining a fused feature map; S5. deploying the map to an autonomous underwater vehicle to form an onboard inference model; and S6. utilizing the onboard inference model to perform online semantic segmentation inference on the preprocessed sonar image block acquired in real time, outputting a semantic category map and a boundary vector map corresponding to the preprocessed sonar image block. The present invention improves the average mIoU metric of the model in scenarios with small targets and multiple categories.

Description

Technical Field

[0001] The present invention relates to the field of sonar technology, and in particular to a seabed sonar image processing method based on semantic segmentation. Background Art

[0002] With the continuous expansion of marine resource development, seabed geological exploration and underwater engineering application scenarios, seabed landform recognition and mapping technology based on sonar imaging has gradually become an important means of modern marine information acquisition and environmental assessment. At present, the mainstream seabed sonar image semantic segmentation method mostly uses deep convolutional neural networks to automatically process sonar images, dividing the original sonar images into different categories of areas such as sediments, bedrock, and artificial structures, providing a data basis for geological analysis, resource exploration and underwater operations.

[0003] However, existing technologies have exposed many shortcomings in practical applications. First, sonar images are naturally affected by high noise, speckle, and shadows, which make the terrain boundaries appear blurred or broken in the imaging results. Traditional U-Net segmentation methods often render real hard boundaries as wide transition zones when processing such data, which makes it difficult to meet the requirements of subsequent terrain vectorization and accurate GIS entry. In addition, existing network structures generally lack the integration of prior knowledge of physical terrain and often rely only on pixel-level gradients or simple attention mechanisms. As a result, when the model faces terrain slope breaks, complex shadows, and multi-scale heterogeneous targets, the segmentation results are prone to pseudo-edges, contour gaps, and category misclassification, which seriously affects the engineering value of terrain mapping and feature extraction.

[0004] Secondly, the acquisition of submarine sonar data is limited by the high cost of ship time and the difficulty of field collection. The sample size of public datasets is limited, and there are significant differences between different survey equipment, regions, and annotation standards. This makes existing methods difficult to generalize in cross-regional and cross-sensor scenarios or requires reliance on large amounts of pixel-level fully annotated data, reducing the actual deployment efficiency and adaptability of the algorithm. At the same time, existing networks based on single-branch or simple multi-branch structures are generally unable to meet the dual requirements of global category discrimination and sub-meter boundary positioning, and lack the ability to retain high-frequency details in sonar echoes. As a result, the output terrain segmentation map is difficult to provide high-quality decision support for subsequent underwater navigation, resource screening, and environmental monitoring.

[0005] Therefore, there is an urgent need for an innovative method that can improve segmentation accuracy, enhance boundary expression, reduce annotation dependence and have real-time reasoning capabilities to better serve the actual needs of marine geological surveys and intelligent underwater operations. Summary of the Invention

[0006] One purpose of the present invention is to propose a submarine sonar image processing method based on semantic segmentation. In the present invention, the average mIoU index of the model is improved in small target and multi-category mixed scenes.

[0007] A method for processing submarine sonar images based on semantic segmentation according to an embodiment of the present invention includes the following steps:

[0008] S1. Collecting raw seabed sonar image data for standardization, preprocessing the standardized sonar image blocks to obtain preprocessed sonar image blocks;

[0009] S2. Calculate terrain slope features based on the preprocessed sonar image blocks, fuse the features, and generate a priori map of the seafloor topography boundary corresponding to the preprocessed sonar image blocks pixel by pixel;

[0010] S3. Build an improved U-Net dual-branch semantic segmentation network, which includes a semantic branch and a boundary branch.

[0011] S4. Configure a geomorphic mutual feedback gating module at each scale skip layer. Embed the seabed geomorphic boundary prior map into the multi-scale skip structure of the improved U-Net dual-branch semantic segmentation network through the geomorphic mutual feedback gating module. Perform weighted fusion on the semantic feature maps output by the semantic branches to obtain a fused feature map.

[0012] S5. Using preprocessed sonar image patches, a prior map of seafloor topography boundaries, and corresponding pixel-level category labels as training data, a modified U-Net two-branch semantic segmentation network is trained using a joint loss function. The trained segmentation model is then deployed on the autonomous underwater vehicle to form an onboard inference model.

[0013] S6. Use the onboard inference model to perform online semantic segmentation inference on the pre-processed sonar image patches acquired in real time, and output the semantic category map and boundary vector map corresponding to the pre-processed sonar image patches.

[0014] Optionally, the S1 includes the following steps:

[0015] S11. Use seabed detection equipment to collect original seabed sonar image data to form a set of original images ,in, Indicates the Original seafloor sonar images, is the total number of collected images;

[0016] S12. For the original image set Perform size adjustment and image segmentation according to the preset segment size Each original image Divide into several non-overlapping or partially overlapping sub-blocks to generate a set of standardized sonar image blocks;

[0017] S13. performing an adaptive spatial filtering operation on the standardized sonar image block set, constructing a local response window, and applying a weighted denoising model in each window to obtain a denoised image block;

[0018] S14. Perform texture preservation enhancement processing on each denoised image block and construct a set of structural response operators in multiple directions , where the set of structural response operators includes horizontal, vertical and two diagonal directions, corresponding to the direction angles , extract the gradient response features of the image block in each direction, compare the response strengths in different directions and select the maximum response direction for enhancement, thus generating an image block with enhanced texture structure;

[0019] S15. Perform grayscale dynamic range equalization processing on the image block with enhanced texture structure, map the original pixel grayscale value to the equalized grayscale value, form a pre-processed sonar image block, and combine them to form a pre-processed sonar image block set. .

[0020] Optionally, the S2 includes the following steps:

[0021] S21. Calculate the local grayscale gradient amplitude map of each pixel in the preprocessed sonar image block ,The local gray gradient amplitude map is used to characterize the gray level change degree of the ,preprocessed sonar image block in the horizontal and vertical directions;

[0022] S22. Calculate the shadow direction response map of each pixel in the preprocessed sonar image block ,The shadow direction response map is used to characterize the sonar shadow saliency of pixel points in multiple directions;

[0023] S23. For the historical digital elevation model data corresponding to the spatial position of the pre-processed sonar image block To calculate the slope, define the slope gradient component of each terrain grid cell as and , calculate the slope magnitude map ,The slope magnitude map is used to measure the size of the terrain slope at each pixel;

[0024] S24. Normalize the local grayscale gradient amplitude map, the shadow direction response map, and the slope amplitude map to obtain normalized grayscale gradient maps. , Normalized shadow response diagram and normalized slope magnitude plots ;

[0025] S25. Construct a pixel-level fusion function to perform weighted fusion of the normalized grayscale gradient map, the normalized shadow response map, and the normalized slope amplitude map according to a preset weight coefficient to generate a priori map of the seabed landform boundary. ;

[0026] S26. Prior map of the seabed topography boundary of all pre-processed sonar image blocks Combining and constructing a priori atlas of seafloor landform boundaries .

[0027] Optionally, S3 includes the following steps:

[0028] S31. Construct a multi-scale sonar feature encoder with input as a set of pre-processed sonar image patches and a priori atlas of seafloor landform boundaries At each scale, the multi-scale sonar feature encoder uses a parallel stack of depth-separable convolution and band-preserving convolution to generate the encoded feature map ;

[0029] S32. At each scale, in the encoding feature map Then the landform attention unit is connected in series. The landform attention unit forms a landform feature vector by the global average mapping of the normalized grayscale gradient map and the normalized slope amplitude map. The landform feature vector is linearly transformed and normalized with the probability distribution constraint through the learnable mapping matrix to obtain the landform attention weight map. ;

[0030] S33. Construct a semantic branch and modulate the topographic attention weight of the encoded feature map at each scale. The modulated semantic feature map is obtained by element-wise multiplication of each pixel in the encoded feature map with the topographic attention weight of the corresponding position. , used to highlight key areas containing sediment boundaries, fault lines, or artificial structures;

[0031] S34. Construct a boundary branch. The boundary branch takes the corresponding seabed landform boundary prior map as input, performs a multi-scale dilated convolution operation to generate a preliminary boundary feature map, and performs a local contrast enhancement operation on the preliminary boundary feature map. The local contrast enhancement operation applies a large-scale average pooling to the boundary prior map and subtracts it from the dilated convolution result to obtain a locally enhanced boundary feature map of the boundary response. .

[0032] Optionally, the S4 includes the following steps:

[0033] S41. Construct a terrain mutual feedback gating module in each scale skip layer. The terrain mutual feedback gating module jointly analyzes the modulation semantic feature map and the boundary initial feature map, extracts features through independent learnable convolution kernels, and then adds them together. After Sigmoid normalization function processing, pixel-level mutual feedback coefficients are generated. ;

[0034] S42. The modulation semantic feature map and the boundary initial feature map at each scale are calculated using the pixel-level mutual feedback coefficient Perform weighted fusion for weights to generate fusion feature maps .

[0035] Optionally, the S5 includes the following steps:

[0036] S51. Construct a training dataset consisting of a preprocessed sonar image patch set, a priori atlas of corresponding seabed landform boundaries, and a pixel-by-pixel category label atlas. Together they constitute, Indicates the semantic category label of each pixel in the preprocessed sonar image block. The category label covers the semantic items of seabed sediments, bedrock structures, artificial structures and background areas.

[0037] S52. Define multi-component joint loss function As the objective function of supervised training, the multi-component joint loss function is composed of the category cross entropy loss , Dice loss in boundary area Preserving loss with topological consistency Weighted calculation;

[0038] S53. Using multi-component joint loss function End-to-end supervised training is performed on the constructed improved U-Net two-branch semantic segmentation network. During the training process, the parameter weights of the semantic branch and the boundary branch are optimized simultaneously until the improved U-Net two-branch semantic segmentation network converges on the validation set, obtaining a trained segmentation model.

[0039] S54. Perform structural optimization and model compression on the trained segmentation model, replace redundant convolution modules with depthwise separable convolution, and fuse the boundary-guided weight map to form a lightweight network structure. Deploy the segmentation model to the onboard graphics processing unit of the autonomous underwater vehicle or remote-controlled underwater platform to form an onboard inference model.

[0040] Optionally, the category cross entropy loss Final predicted pixel-wise category label map used to measure the output of the semantic branch and the actual pixel-by-pixel category label map The classification consistency between them on the entire image:

[0041] ;

[0042] in, represents the height of each pre-processed sonar image block, Indicates the width of each pre-processed sonar image block, Represents the total number of categories in the segmentation task, including seabed sediments, bedrock structures, artificial structures, and background categories. Indicates the horizontal coordinate index of the current pixel, Indicates the vertical coordinate index of the current pixel, represents the semantic category index, Indicates location The pixel at belongs to The true value label of the class, Represents the improved U-Net dual-branch semantic segmentation network at position Place opposite The predicted probability value of the class.

[0043] Optionally, the boundary area Dice loss It is used to supervise the accuracy of the boundary branch in the landform boundary area, and obtain the seabed landform boundary prior map through the binary boundary mask map. Extract the boundary region:

[0044] ;

[0045] in, Represents the boundary prediction map output by the improved U-Net two-branch semantic segmentation network, Represents a binary boundary mask map, which is derived from the seabed landform boundary prior map Extracted according to the preset threshold, It represents the sum of the binary boundary mask map and the predicted boundary map after pixel-level multiplication. Represents the number of pixels belonging to the boundary area in the binary boundary mask image, Represents the sum of all pixel confidences in the predicted boundary map.

[0046] Optionally, the topology consistency preservation loss It is used to constrain the consistency of the spatial topological relationship between the contour structure output by the improved U-Net two-branch semantic segmentation network and the true contour. The topological consistency preservation loss is obtained by respectively calculating the Euler number of each category area output by the improved U-Net two-branch semantic segmentation network and the Euler number of the corresponding category area in the true label map, and calculating the absolute value of the difference between the two Euler numbers and taking the average value.

[0047] Optionally, the S6 includes the following steps:

[0048] S61. Real-time connection between the onboard inference model and the sonar data acquisition system of the seabed detection equipment;

[0049] S62. Utilize the onboard inference model to batch or stream input the real-time acquired, standardized, and preprocessed sonar image patch sets, and perform the online semantic segmentation inference process. For each preprocessed sonar image patch, parallel inference outputs the corresponding semantic category map and boundary vector map. The semantic category map is the pixel-level category distribution result consistent with the spatial size of the input sonar image patch, and the boundary vector map is a multi-channel feature map that encodes the strength and direction of each pixel boundary response.

[0050] The beneficial effects of the present invention are:

[0051] (1) The present invention explicitly introduces the seabed landform boundary prior map into the network structure, and uses the multi-source physical information of normalized grayscale gradient, shadow direction and slope amplitude to achieve the joint modeling of physical landform features and depth semantic features through the fusion of landform attention unit and multi-scale feature encoder. It can not only actively focus on the key slope breaks, faults, and complex return shadow areas in the sonar image, but also accurately suppress the false contours caused by noise and speckle interference, so that the model can obtain sub-meter segmentation resolution at the landform boundary, significantly improving the spatial accuracy of landform elements.

[0052] (2) The present invention designs a geomorphic mutual feedback gating module, which realizes the pixel-level weight dynamic fusion of the modulated semantic feature map and the boundary initial feature map. Through the mutual feedback learning of multi-scale geomorphic responses, the network can adaptively adjust the attention of semantic category areas and boundary areas, taking into account both global category discrimination and fine-grained boundary recognition. It is suitable for the accurate extraction of thin layers of seabed sediments, debris flow fans, and weak boundary targets of artificial structures. In the scenario of small targets and multi-category mixed scenes, the average mIoU index of the model is improved.

[0053] (3) This paper adopts depthwise separable convolution, band-preserving convolution, and a lightweight model structure compression design, enabling the trained segmentation model to run in real time at an inference speed of 8–12 fps on an embedded platform of an autonomous underwater vehicle. Furthermore, by combining a boundary prior map with a joint loss optimization strategy, high-quality training can be achieved with minimal pixel-level annotation or only coarse boundary annotation, significantly reducing the annotation cost and adaptability threshold of submarine sonar data. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0055] Figure 1This is a flow chart of a submarine sonar image processing method based on semantic segmentation proposed by the present invention;

[0056] Figure 2 This is a block diagram of the improved U-Net dual-branch semantic segmentation network structure in the submarine sonar image processing method based on semantic segmentation proposed in the present invention. DETAILED DESCRIPTION

[0057] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0058] refer to Figure 1-Figure 2 , a submarine sonar image processing method based on semantic segmentation, comprising the following steps:

[0059] S1. Collect raw seafloor sonar image data, slice and standardize it according to a uniform pixel resolution and file format to obtain standardized sonar image blocks. Preprocess the standardized sonar image blocks by using adaptive spatial filtering to remove speckle noise, perform texture-preserving enhancement, and perform grayscale dynamic range equalization to obtain preprocessed sonar image blocks.

[0060] S2. Calculate terrain slope characteristics based on the local grayscale gradient of the preprocessed sonar image blocks, sonar shadow direction information, and historical digital elevation model data, fuse these features, and generate a priori map of the seafloor landform boundary corresponding to the preprocessed sonar image blocks pixel by pixel;

[0061] S3. Construct an improved U-Net dual-branch semantic segmentation network. The improved U-Net dual-branch semantic segmentation network includes a semantic branch and a boundary branch. The semantic branch uses preprocessed sonar image blocks as input to extract seabed sediment features, bedrock structure features, and artificial structure features. The boundary branch uses a priori seabed landform boundary map as input to extract boundary saliency features.

[0062] S4. Configure a geomorphic mutual feedback gating module at each scale skip layer. Embed the seabed geomorphic boundary prior map into the multi-scale skip structure of the improved U-Net dual-branch semantic segmentation network through the geomorphic mutual feedback gating module. Perform weighted fusion on the semantic feature maps output by the semantic branches to obtain a fused feature map.

[0063] S5. Using preprocessed sonar image patches, a priori maps of seafloor topography boundaries, and corresponding pixel-level category labels as training data, a modified U-Net two-branch semantic segmentation network is supervised trained using a joint loss function consisting of category cross-entropy loss, boundary region Dice loss, and topological consistency loss. The trained segmentation model is then deployed on the autonomous underwater vehicle to form an onboard inference model.

[0064] S6. Use the onboard inference model to perform online semantic segmentation inference on the pre-processed sonar image patches acquired in real time, and output the semantic category map and boundary vector map corresponding to the pre-processed sonar image patches.

[0065] In this embodiment, S1 includes the following steps:

[0066] S11. Use seabed detection equipment to collect original seabed sonar image data. The original seabed sonar image data is a two-dimensional grayscale image. The unified image resolution and frequency band parameters are used in the collection process to form the original image set. ,in, Indicates the Original seafloor sonar images, is the total number of collected images;

[0067] S12. For the original image set Perform size adjustment and image segmentation according to the preset segment size Each original image Divide into several non-overlapping or partially overlapping sub-blocks to generate a standardized sonar image block set ,in, Indicates the The first image image blocks, and are the width and height of the image block respectively;

[0068] S13. Standardized sonar image patch set Perform adaptive spatial filtering operations to construct local response windows , apply the weighted denoising model in each window to obtain the denoised image patch:

[0069] ;

[0070] in, Represents the grayscale value of the pixel in the window, represents the weight coefficient based on local variance and structure preservation, is the center pixel value after denoising;

[0071] S14. Perform texture preservation enhancement processing on each denoised image block and construct a set of structural response operators in multiple directions , where the set of structural response operators includes horizontal, vertical and two diagonal directions, corresponding to the direction angles , extract the gradient response features of the image block in each direction, compare the response strengths in different directions and select the maximum response direction for enhancement, thus generating an image block with enhanced texture structure;

[0072] S15. Perform grayscale dynamic range equalization on the image block with enhanced texture structure. Use grayscale transformation based on local histogram to construct a grayscale mapping function of pixel value by calculating the frequency of occurrence of each grayscale value in the image block in the entire image block. Map the original pixel grayscale value to the equalized grayscale value to form a preprocessed sonar image block, and combine them to form a preprocessed sonar image block set. .

[0073] The grayscale mapping function is constructed by counting the frequency of pixels at each grayscale level in each standardized sonar image block to form a local grayscale histogram, and remapping each grayscale level according to the cumulative distribution function. The mapping function is expressed as follows: the original pixel grayscale value is linearly mapped to the target grayscale interval according to its cumulative probability in this image block, realizing grayscale redistribution, enhancing local contrast and compressing noise intervals. The output grayscale value of each pixel is equal to its cumulative probability multiplied by the upper limit of the grayscale level, rounded down to obtain an integer grayscale level.

[0074] In this embodiment, S2 includes the following steps:

[0075] S21. Calculate the local grayscale gradient amplitude map of each pixel in the preprocessed sonar image block The local grayscale gradient amplitude map is used to characterize the grayscale change degree of the preprocessed sonar image block in the horizontal and vertical directions. The local grayscale gradient amplitude map is obtained by calculating the gradient component of the image in the horizontal direction and the gradient component in the vertical direction respectively, and taking the square root of the sum of the squares of the gradient components in the two directions.

[0076] S22. Calculate the shadow direction response map of each pixel in the preprocessed sonar image block The shadow direction response map is used to characterize the sonar shadow significance of pixels in multiple directions. The shadow direction response map is calculated by applying linear directional enhancement filters in multiple preset directions to the preprocessed sonar image blocks, calculating the directional response value of the pixel in each direction, and selecting the maximum value from all directional response values ​​as the response intensity of the corresponding pixel in the shadow direction response map;

[0077] S23. For the historical digital elevation model data corresponding to the spatial position of the pre-processed sonar image block To calculate the slope, define the slope gradient component of each terrain grid cell as and , calculate the slope magnitude map , the slope magnitude map is used to measure the size of the terrain slope at each pixel point:

[0078] ;

[0079] S24. Normalize the local grayscale gradient amplitude map, the shadow direction response map, and the slope amplitude map to obtain normalized grayscale gradient maps. , Normalized shadow response diagram and normalized slope magnitude plots ;

[0080] S25. Construct a pixel-level fusion function to perform weighted fusion of the normalized grayscale gradient map, the normalized shadow response map, and the normalized slope amplitude map according to a preset weight coefficient to generate a priori map of the seabed landform boundary. , used to indicate the significance of each pixel belonging to the landform boundary. During the fusion process, the boundary prior value of each pixel is obtained by multiplying the normalized gray gradient value, the normalized shadow response value, and the normalized slope amplitude by the corresponding fusion weight coefficient;

[0081] The pixel-level fusion function normalizes the three types of feature maps, namely grayscale gradient, shadow response, and slope amplitude, to the range of [0,1] for each preprocessed sonar image block. An independent fusion weight is assigned to each feature map at the pixel level. The fusion weight can be obtained based on scene requirements, empirical knowledge, or through prior data statistics. For each pixel in the image, the fusion function calculates the product of the three normalized eigenvalues ​​and the corresponding weight, and sums the three to obtain the final pixel-level prior value.

[0082] S26. Prior map of the seabed topography boundary of all pre-processed sonar image blocks Combining and constructing a priori atlas of seafloor landform boundaries ,Each seabed topography boundary prior map corresponds one-to-one to the corresponding pre-processed sonar image block in the spatial dimension.

[0083] In this embodiment, S3 includes the following steps:

[0084] S31. Construct a multi-scale sonar feature encoder with input as a set of pre-processed sonar image patches and a priori atlas of seafloor landform boundaries At each scale, the multi-scale sonar feature encoder uses a parallel stack of depth-separable convolution and band-preserving convolution to generate the encoded feature map , where the sampling rate of the band-preserving convolution is kept consistent with the emission frequency of the sonar signal, which is used to prevent the high-frequency detail features in the sonar image from being smoothed or mistakenly clipped during the convolution process, thereby preserving the structural integrity of the sonar echo texture;

[0085] S32. At each scale, in the encoding feature map Then the landform attention unit is connected in series. The landform attention unit forms a landform feature vector by the global average mapping of the normalized grayscale gradient map and the normalized slope amplitude map. The landform feature vector is linearly transformed and normalized with the probability distribution constraint through the learnable mapping matrix to obtain the landform attention weight map. ,The landform attention weight map is used to indicate which areas are more likely to be landform boundaries or edge areas at the current scale;

[0086] S33. Construct a semantic branch and modulate the topographic attention weight of the encoded feature map at each scale. The modulated semantic feature map is obtained by element-wise multiplication of each pixel in the encoded feature map with the topographic attention weight of the corresponding position. , which is used to highlight key areas containing sediment boundaries, fault lines or artificial structures, and suppress noisy areas or low-response areas, so that the areas of interest of the semantic branch are more concentrated on the pixel distribution with geomorphic structural significance;

[0087] S34. Construct a boundary branch. The boundary branch takes the corresponding seabed landform boundary prior map as input, performs a multi-scale dilated convolution operation to generate a preliminary boundary feature map, and performs a local contrast enhancement operation on the preliminary boundary feature map. The local contrast enhancement operation applies a large-scale average pooling to the boundary prior map and subtracts it from the dilated convolution result to obtain a locally enhanced boundary feature map of the boundary response. ,The local contrast enhancement operation is used to increase the grayscale difference between the weak boundary region and its neighborhood in the image block, thereby enhancing the boundary information.,The initial boundary feature map is used to describe the texture and response features of the,image that is suspected to belong to the boundary region;

[0088] ;

[0089] in, is the dilated convolution operator, the expansion rate and scale correspond, is the window size Average pooling.

[0090] In this embodiment, S4 includes the following steps:

[0091] S41. Construct a terrain mutual feedback gating module in each scale skip layer. The terrain mutual feedback gating module jointly analyzes the modulation semantic feature map and the boundary initial feature map, extracts features through independent learnable convolution kernels, and then adds them together. After Sigmoid normalization function processing, pixel-level mutual feedback coefficients are generated. , pixel-level mutual feedback coefficient Used to measure the contribution of semantic features and boundary features to the fusion results at each spatial position;

[0092] The terrain mutual feedback gating module is a pixel-level, dynamic fusion mechanism proposed on the basis of an improved U-Net dual-branch semantic segmentation network to meet the needs of complex expression of multi-source heterogeneous terrain features and high-precision boundary discrimination in submarine sonar images. The terrain mutual feedback gating module mainly acts on the multi-scale skip layer of the network, constructing cross-branch information feedback and weighted modulation pathways between semantic branches and boundary branches, realizing deep coupling and dynamic complementarity of semantic and boundary features.

[0093] The fusion process of the terrain mutual feedback gating module is not just a simple feature splicing or linear superposition, but through convolution extraction, nonlinear normalization and spatial adaptive weighting, it fully utilizes the physical saliency and semantic discrimination information of the terrain boundary area in the sonar image to achieve dynamic adjustment and enhance the key pixel response, significantly improving the segmentation effect of weak boundaries, small targets and complex terrain areas, and effectively alleviating the problems of category misclassification and contour breakage caused by boundary fuzzy and noise interference in traditional single-branch or static fusion methods.

[0094] S42. The modulation semantic feature map and the boundary initial feature map at each scale are calculated using the pixel-level mutual feedback coefficient Perform weighted fusion for weights to generate fusion feature maps , the final value of each pixel in the fusion feature map is the sum of the modulation semantic feature map and the boundary initial feature map multiplied by the pixel-level mutual feedback coefficient and its complement;

[0095] ;

[0096] In this embodiment, S5 includes the following steps:

[0097] S51. Construct a training dataset consisting of a preprocessed sonar image patch set, a priori atlas of corresponding seabed landform boundaries, and a pixel-by-pixel category label atlas. Together they constitute, Indicates the semantic category label of each pixel in the preprocessed sonar image block. The category label covers the semantic items of seabed sediments, bedrock structures, artificial structures and background areas.

[0098] S52. Define multi-component joint loss function As the objective function of supervised training, the multi-component joint loss function is composed of the category cross entropy loss , Dice loss in boundary area Preserving loss with topological consistency Weighted calculation;

[0099] S53. Using multi-component joint loss function End-to-end supervised training is performed on the constructed improved U-Net two-branch semantic segmentation network. During the training process, the parameter weights of the semantic branch and the boundary branch are optimized simultaneously until the improved U-Net two-branch semantic segmentation network converges on the validation set, obtaining a trained segmentation model.

[0100] S54. Perform structural optimization and model compression on the trained segmentation model, replace redundant convolution modules with depthwise separable convolution, and fuse the boundary-guided weight map to form a lightweight network structure. Deploy the segmentation model to the onboard graphics processing unit of the autonomous underwater vehicle or remote-controlled underwater platform to form an onboard inference model.

[0101] In this implementation, the category cross entropy loss Final predicted pixel-wise category label map used to measure the output of the semantic branch and the actual pixel-by-pixel category label map The classification consistency between them on the entire image:

[0102] ;

[0103] in, represents the height of each pre-processed sonar image block, Indicates the width of each pre-processed sonar image block, Represents the total number of categories in the segmentation task, including seabed sediments, bedrock structures, artificial structures, and background categories. Indicates the horizontal coordinate index of the current pixel, the value range is , Indicates the vertical coordinate index of the current pixel, the value range is , Represents the semantic category index, the value range is , Indicates location The pixel at belongs to The true value label of the class, if it belongs to the class, the value is 1, otherwise it is 0, Represents the improved U-Net dual-branch semantic segmentation network at position Place opposite The predicted probability value of the class ranges from 0 to 1.

[0104] In this embodiment, the Dice loss in the boundary area It is used to supervise the accuracy of the boundary branch in the landform boundary area, and obtain the seabed landform boundary prior map through the binary boundary mask map. Extract the boundary region:

[0105] ;

[0106] in, Represents the boundary prediction map output by the improved U-Net two-branch semantic segmentation network, Represents a binary boundary mask map, which is derived from the seabed landform boundary prior map Extracted according to the preset threshold, Each pixel value in 1 indicates that it belongs to the boundary area, and 0 indicates that it belongs to the non-boundary area. It represents the sum of the pixel-level product of the binary boundary mask map and the predicted boundary map, and the sum of the overlapping areas between the model predicted boundary and the true boundary. Represents the number of pixels belonging to the boundary area in the binary boundary mask image, Represents the sum of all pixel confidences in the predicted boundary map.

[0107] In this embodiment, the topological consistency is maintained. It is used to constrain the consistency of the spatial topological relationship between the contour structure output by the improved U-Net two-branch semantic segmentation network and the true contour. The topological consistency preservation loss is obtained by respectively calculating the Euler number of each category area output by the improved U-Net two-branch semantic segmentation network and the Euler number of the corresponding category area in the true label map, and calculating the absolute value of the difference between the two Euler numbers and taking the average value. The Euler number is used to measure the number of connected domains, the number of holes and the integrity of the topological structure of each category area. The topological consistency preservation loss takes the average of the Euler number differences of all categories as the final loss value. The Euler number calculation method remains consistent within each category area.

[0108] In this embodiment, S6 includes the following steps:

[0109] S61. Real-time connection between the onboard inference model and the sonar data acquisition system of the seabed detection equipment;

[0110] S62. Utilize the onboard inference model to perform online semantic segmentation inference on batch or streaming input of a set of normalized and preprocessed sonar image patches acquired in real time. For each preprocessed sonar image patch, parallel inference outputs are generated for the corresponding semantic category map and boundary vector map. The semantic category map is a pixel-level category distribution consistent with the spatial dimensions of the input sonar image patch, and the boundary vector map is a multi-channel feature map that encodes the strength and direction of each pixel boundary response.

[0111] S63. Perform real-time post-processing on the online semantic segmentation inference results. The semantic category map is thresholded at the pixel level to obtain the final category labels. The category labels include seabed sediment area, bedrock structure area, artificial structure area, and background area.

[0112] The seabed sediment area category label corresponds to the pixel point with the maximum category probability belonging to the sediment category and low response intensity and uniform grayscale in the seabed landform boundary prior map.

[0113] The bedrock structure area category label corresponds to the pixel point with the maximum category probability belonging to the bedrock structure category and the pixel is in the local extreme value area of ​​the slope amplitude.

[0114] The artificial structure region category label corresponds to the pixel point with the maximum category probability belonging to the artificial structure category and the pixel belongs to the regular texture area or has a significant boundary enhancement response.

[0115] The background area category label corresponds to the pixel points whose maximum probability of all categories is lower than the preset confidence threshold or does not conform to other category structures or landform features;

[0116] The boundary vector map extracts boundary pixel sets through non-maximum suppression and threshold discrimination to obtain the spatial distribution information of the landform boundary. The boundary type in the boundary vector map maintains a corresponding relationship with the structural consistency of the semantic category map:

[0117] The sediment boundary type corresponds to the semantic category map where one side of the boundary pixel is the seabed sediment area category label, and the other side is the background area category label or other category label, and the response value of this boundary in the seabed landform boundary prior map is in the medium-low range;

[0118] The structural boundary type corresponds to the boundary pixel connection area being in the high slope response area or the gradient extreme value area, and the semantic category map distinguishes the border position of the bedrock structure area category label and the seabed sediment area category label or the artificial structure area category label;

[0119] The artificial boundary type corresponds to the boundary presenting a long straight line or a regular closed shape, and overlaps with the high confidence area of ​​the artificial structure area category label, and the boundary response is continuous and the boundary closure degree is high in the local structure contrast enhancement result.

[0120] Example 1: A marine survey team carried out a mission from April to June 2024. The operating area had a water depth of 400 to 1100 meters and a sea area of ​​approximately 130 square kilometers, covering muddy sedimentary fans, fault zones, and artificially laid pipeline areas. The survey team used a certain type of multi-beam sonar (operating frequency 300kHz), synchronous side-scan sonar equipment (resolution 0.25 meters / pixel), and an underwater high-performance GPU (NVIDIA Jetson AGX Orin, 32GB RAM) on the AUV platform to acquire and process large-scale underwater landform data in real time.

[0121] During the data collection phase, the AUVs were deployed on equidistant routes at a speed of 3 knots, with a single voyage lasting approximately 12 hours. Approximately 11,000 raw sonar images were collected each time, each with a size of 2048 × 4096 pixels. These raw data exhibited significant sonar shadowing, speckle noise, and echo banding. The model was trained and tested using the same batch of sonar images collected using a traditional U-Net, SEAUNet, FPUA-UNet, and the proposed segmentation method based on an improved U-Net and enhanced seafloor topography boundaries.

[0122] In the data preprocessing stage, the resolution of all original sonar images was unified to 0.25 meters per pixel. Adaptive spatial filtering was used to remove speckle noise and cut the original images into small blocks of 512×512 pixels. A total of 4,800 standardized training samples were collected and constructed for the mud sediment area and fault zone area respectively. Local slope, grayscale gradient and main shadow direction information were automatically extracted from historical survey data to generate a priori map of the landform boundary. The total number of samples in the priori map was consistent with the image blocks.

[0123] During the model training phase, the proposed method was compared with three traditional segmentation methods. Each model had a training set of 4,000 samples and a test set of 800 samples. Each image block was manually labeled with pixel-level categories and boundaries. All models were trained and tested on the same hardware platform (NVIDIA Jetson AGX Orin). Training used the Adam optimizer, with an initial learning rate of 0.001 and a maximum of 50 iterations, each taking an average of 40 minutes.

[0124] Using the method of the present invention, the input of the segmentation network is the pre-processed sonar image block, the landform boundary prior map and the pixel-level annotation. The backbone adopts a multi-scale feature extraction structure of depth-separable convolution and band-preserving convolution, and displays the fusion of slope, gradient and shadow prior. The semantic branch focuses on the extraction of sediments, bedrock and artificial target areas. The boundary branch uses the landform prior to dynamically guide the segmentation contour. The multi-scale landform mutual feedback gating module performs pixel-level weighted fusion of semantics and boundary responses to achieve refined contour expression. Model training adopts the joint optimization of category cross entropy, boundary Dice and topological consistency loss. The final inference model is compressed and deployed on the AUV onboard GPU to achieve full-process automatic inference and real-time output.

[0125] In actual testing, 800 test image blocks were collected, representing typical segmentation scenarios including sedimentary fan transition zones, fault boundaries, artificially laid pipeline areas, and debris flow sectors. Using manual annotation as a benchmark, the performance of each method in terms of mean intersection over union (mIoU), boundary pixel accuracy (BoundaryF1), boundary Hausdorff distance (HD, in pixels), and inference speed (fps) is summarized in Table 1:

[0126] Table 1 Data comparison between the present invention and three traditional segmentation methods

[0127]

[0128] In addition, in terms of feature preservation and small target detection, the method of the present invention significantly improves the detection rate of fault zones and slender pipeline structures. Compared with manual labeling, the average small target recall rate is improved from 62.8% of the traditional U-Net to 80.4% of the present invention, and the boundaries of the segmented fault lines are continuous, without obvious transition zones or fracture artifacts. In the area of ​​artificial structures (such as laid pipelines), the end-to-end connectivity rate (the degree of overlap between the connected length and the real structure) detected by the method of the present invention is increased to 93.6%, which is more than 10% higher than that of SEAUNet.

[0129] For the expression of landform boundaries in high-noise environments, the method of the present invention achieved a pixel-level accuracy of 89.3% in the shadow areas of faults and slope breaks, which is 13.4% higher than FPUA-UNet. The number of boundary artifacts in specific shadow overlap samples was reduced by more than 40%. The average number of pixels requiring manual post-processing and repair in all test samples was reduced to 110 pixels / block (traditional U-Net requires 275 pixels / block).

[0130] In terms of inference speed, the onboard GPU real-time inference frame rate of the method of the present invention is stable at 8-12fps, which can meet the needs of AUV or ROV online autonomous identification, real-time path planning and multi-target terrain tracking, and significantly shorten the seabed surveying and mapping data processing cycle.

[0131] Training sample display:

[0132] Sediment sector samples: The original image has strong noise and the slope prior shows a band-like transition. The boundary of the sediment area segmented by the present invention fits the manual annotation, and the fault boundary is clear; pipeline area samples: The artificial structure is long and narrow, and the traditional method has many discontinuous pseudo-edges. The boundary branch of the present invention accurately detects all line segments and the endpoints are connected; samples with strong shadow interference: The segmentation results of the traditional method have blurred edges, and the model of the present invention significantly suppresses shadow pseudo-edges through the landform prior, and the contour is continuous.

[0133] The present invention explicitly introduces the seabed landform boundary prior map into the network structure, utilizes the multi-source physical information of normalized grayscale gradient, shadow direction and slope amplitude, and realizes the joint modeling of physical landform features and depth semantic features through the fusion of landform attention unit and multi-scale feature encoder. It can not only actively focus on the key slope breaks, faults, and complex return shadow areas in the sonar image, but also accurately suppress the false contours caused by noise and speckle interference, so that the model can obtain sub-meter segmentation resolution at the landform boundary, significantly improving the spatial accuracy of landform elements.

[0134] The present invention designs a landform mutual feedback gating module, which realizes the dynamic fusion of pixel-level weights of the modulated semantic feature map and the initial boundary feature map. Through mutual feedback learning of multi-scale landform responses, the network can adaptively adjust the attention of semantic category areas and boundary areas, taking into account both global category discrimination and fine-grained boundary recognition. It is suitable for the accurate extraction of thin layers of seabed sediments, debris flow fans, and weak boundary targets of artificial structures. In scenarios with small targets and multiple categories, the average mIoU index of the model is improved.

[0135] This method utilizes depthwise separable convolution, band-preserving convolution, and a lightweight model structure compression design, enabling the trained segmentation model to run in real time on an embedded platform for autonomous underwater vehicles at an inference speed of 8–12 fps. Furthermore, by combining boundary prior maps with a joint loss optimization strategy, high-quality training can be achieved with minimal pixel-level annotation or only coarse boundary annotation, significantly reducing the annotation cost and adaptability of submarine sonar data.

[0136] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for processing submarine sonar images based on semantic segmentation, characterized in that: The steps include: S1. Collecting raw seabed sonar image data for standardization, preprocessing the standardized sonar image blocks to obtain preprocessed sonar image blocks; S2. Calculate terrain slope features based on the preprocessed sonar image blocks, fuse the features, and generate a priori map of the seafloor topography boundary corresponding to the preprocessed sonar image blocks pixel by pixel; S3. Build an improved U-Net dual-branch semantic segmentation network, which includes a semantic branch and a boundary branch. The S3 includes the following steps: S31. Construct a multi-scale sonar feature encoder with input as a set of pre-processed sonar image patches and a priori atlas of seafloor landform boundaries At each scale, the multi-scale sonar feature encoder uses a parallel stack of depth-separable convolution and band-preserving convolution to generate the encoded feature map ; S32. At each scale, in the encoding feature map Then the landform attention unit is connected in series. The landform attention unit forms a landform feature vector by the global average mapping of the normalized grayscale gradient map and the normalized slope amplitude map. The landform feature vector is linearly transformed and normalized with the probability distribution constraint through the learnable mapping matrix to obtain the landform attention weight map. ; S33. Construct a semantic branch and modulate the topographic attention weight of the encoded feature map at each scale. The modulated semantic feature map is obtained by element-wise multiplication of each pixel in the encoded feature map with the topographic attention weight of the corresponding position. , used to highlight key areas containing sediment boundaries, fault lines, or artificial structures; S34. Construct a boundary branch. The boundary branch takes the corresponding seabed landform boundary prior map as input, performs a multi-scale dilated convolution operation to generate a preliminary boundary feature map, and performs a local contrast enhancement operation on the preliminary boundary feature map. The local contrast enhancement operation applies a large-scale average pooling to the boundary prior map and subtracts it from the dilated convolution result to obtain a locally enhanced boundary feature map of the boundary response. ; S4. Configure a geomorphic mutual feedback gating module at each scale skip layer. Embed the seabed geomorphic boundary prior map into the multi-scale skip structure of the improved U-Net dual-branch semantic segmentation network through the geomorphic mutual feedback gating module. Perform weighted fusion on the semantic feature maps output by the semantic branches to obtain a fused feature map. S5. Using preprocessed sonar image patches, a prior map of seafloor topography boundaries, and corresponding pixel-level category labels as training data, a modified U-Net two-branch semantic segmentation network is trained using a joint loss function. The trained segmentation model is then deployed on the autonomous underwater vehicle to form an onboard inference model. S6. Use the onboard inference model to perform online semantic segmentation inference on the pre-processed sonar image patches acquired in real time, and output the semantic category map and boundary vector map corresponding to the pre-processed sonar image patches.

2. The method for processing submarine sonar images based on semantic segmentation according to claim 1, characterized in that: Said S1 comprises the following steps: S11. Use seabed detection equipment to collect original seabed sonar image data to form a set of original images ,in, Indicates the Original seafloor sonar images, is the total number of collected images; S12. For the original image set Perform size adjustment and image segmentation according to the preset segment size Each original image Divide into several non-overlapping or partially overlapping sub-blocks to generate a set of standardized sonar image blocks; S13. performing an adaptive spatial filtering operation on the standardized sonar image block set, constructing a local response window, and applying a weighted denoising model in each window to obtain a denoised image block; S14. Perform texture preservation enhancement processing on each denoised image block and construct a set of structural response operators in multiple directions , where the set of structural response operators includes horizontal, vertical and two diagonal directions, corresponding to the direction angles , extract the gradient response features of the image block in each direction, compare the response strengths in different directions and select the maximum response direction for enhancement, thus generating an image block with enhanced texture structure; S15. Perform grayscale dynamic range equalization processing on the image block with enhanced texture structure, map the original pixel grayscale value to the equalized grayscale value, form a pre-processed sonar image block, and combine them to form a pre-processed sonar image block set. .

3. The method for processing submarine sonar images based on semantic segmentation according to claim 2, characterized in that: The S2 comprises the following steps: S21. Calculate the local grayscale gradient amplitude map of each pixel in the preprocessed sonar image block ,The local gray gradient amplitude map is used to characterize the gray level change degree of the ,preprocessed sonar image block in the horizontal and vertical directions; S22. Calculate the shadow direction response map of each pixel in the preprocessed sonar image block ,The shadow direction response map is used to characterize the sonar shadow saliency of pixel points in multiple directions; S23. For the historical digital elevation model data corresponding to the spatial position of the pre-processed sonar image block To calculate the slope, define the slope gradient component of each terrain grid cell as and , calculate the slope magnitude map ,The slope magnitude map is used to measure the size of the terrain slope at each pixel; S24. Normalize the local grayscale gradient amplitude map, the shadow direction response map, and the slope amplitude map to obtain normalized grayscale gradient maps. , Normalized shadow response diagram and normalized slope magnitude plots ; S25. Construct a pixel-level fusion function to perform weighted fusion of the normalized grayscale gradient map, the normalized shadow response map, and the normalized slope amplitude map according to a preset weight coefficient to generate a priori map of the seabed landform boundary. ; S26. Prior map of the seabed topography boundary of all pre-processed sonar image blocks Combining and constructing a priori atlas of seafloor landform boundaries .

4. The method for processing submarine sonar images based on semantic segmentation according to claim 3, characterized in that: The S4 comprises the following steps: S41. Construct a terrain mutual feedback gating module in each scale skip layer. The terrain mutual feedback gating module jointly analyzes the modulation semantic feature map and the boundary initial feature map, extracts features through independent learnable convolution kernels, and then adds them together. After Sigmoid normalization function processing, pixel-level mutual feedback coefficients are generated. ; S42. The modulation semantic feature map and the boundary initial feature map at each scale are calculated using the pixel-level mutual feedback coefficient Perform weighted fusion for weights to generate fusion feature maps .

5. The method for processing submarine sonar images based on semantic segmentation according to claim 1, wherein: The S5 comprises the following steps: S51. Construct a training dataset consisting of a preprocessed sonar image patch set, a priori atlas of corresponding seabed landform boundaries, and a pixel-by-pixel category label atlas. Together, they constitute Indicates the semantic category label of each pixel in the preprocessed sonar image block. The category label covers the semantic items of seabed sediments, bedrock structures, artificial structures and background areas. S52. Define multi-component joint loss function As the objective function of supervised training, the multi-component joint loss function is composed of the category cross entropy loss , Dice loss in boundary area Preserving loss with topological consistency Weighted calculation; S53. Using multi-component joint loss function End-to-end supervised training is performed on the constructed improved U-Net two-branch semantic segmentation network. During the training process, the parameter weights of the semantic branch and the boundary branch are optimized simultaneously until the improved U-Net two-branch semantic segmentation network converges on the validation set, obtaining a trained segmentation model. S54. Perform structural optimization and model compression on the trained segmentation model, replace redundant convolution modules with depthwise separable convolution, and fuse the boundary-guided weight map to form a lightweight network structure. Deploy the segmentation model to the onboard graphics processing unit of the autonomous underwater vehicle or remote-controlled underwater platform to form an onboard inference model.

6. The method for processing submarine sonar images based on semantic segmentation according to claim 5, characterized in that: The category cross entropy loss Final predicted pixel-wise category label map used to measure the output of the semantic branch and the actual pixel-by-pixel category label map The classification consistency between them on the entire image: ； in, represents the height of each pre-processed sonar image block, Indicates the width of each pre-processed sonar image block, Represents the total number of categories in the segmentation task, including seabed sediments, bedrock structures, artificial structures, and background categories. Indicates the horizontal coordinate index of the current pixel, Indicates the vertical coordinate index of the current pixel, represents the semantic category index, Indicates location The pixel at belongs to The true value label of the class, Represents the improved U-Net dual-branch semantic segmentation network at position Place opposite The predicted probability value of the class.

7. The method for processing submarine sonar images based on semantic segmentation according to claim 5, characterized in that: The boundary region Dice loss It is used to supervise the accuracy of the boundary branch in the landform boundary area, and obtain the seabed landform boundary prior map through the binary boundary mask map. Extract the boundary region: ； in, Represents the boundary prediction map output by the improved U-Net two-branch semantic segmentation network, Represents a binary boundary mask map, which is derived from the seabed landform boundary prior map Extracted according to the preset threshold, It represents the sum of the binary boundary mask map and the predicted boundary map after pixel-level multiplication. Represents the number of pixels belonging to the boundary area in the binary boundary mask image, Represents the sum of all pixel confidences in the predicted boundary map.

8. The method for processing submarine sonar images based on semantic segmentation according to claim 6, characterized in that: The topology consistency preservation loss It is used to constrain the consistency of the spatial topological relationship between the contour structure output by the improved U-Net two-branch semantic segmentation network and the true contour. The topological consistency preservation loss is obtained by respectively calculating the Euler number of each category area output by the improved U-Net two-branch semantic segmentation network and the Euler number of the corresponding category area in the true label map, and calculating the absolute value of the difference between the two Euler numbers and taking the average value.

9. The method for processing submarine sonar images based on semantic segmentation according to claim 6, characterized in that: The S6 comprises the following steps: S61. Real-time connection between the onboard inference model and the sonar data acquisition system of the seabed detection equipment; S62. Utilize the onboard inference model to batch or stream input the real-time acquired, standardized, and preprocessed sonar image patch sets, and perform the online semantic segmentation inference process. For each preprocessed sonar image patch, parallel inference outputs the corresponding semantic category map and boundary vector map. The semantic category map is the pixel-level category distribution result consistent with the spatial size of the input sonar image patch, and the boundary vector map is a multi-channel feature map that encodes the strength and direction of each pixel boundary response.

Images