A multi-source side-scan sonar joint semantic segmentation method and system oriented to heterogeneous labeling ranges

Abstract

The application provides a multi-source side scan sonar joint semantic segmentation method and system facing a heterogeneous annotation range, and relates to the technical field of underwater acoustic signal processing. A unified semantic space containing preset semantic categories is established, and an effective category set and an ineffective category set are determined according to the annotation protocol of each side scan sonar data set; in the training stage, pixel-by-pixel feature extraction is performed on the input side scan sonar image, and a pixel-by-pixel unnormalized category score vector on the unified semantic space is output; according to the effective category set of the data set to which the current image belongs, a preset negative attenuation bias value is added to the score belonging to the ineffective category set to obtain a soft masking score vector, and a probability normalized cross-entropy loss is calculated to update the network parameters; in the reasoning stage, the negative attenuation bias value is completely removed, and the pixel-by-pixel semantic segmentation graph is directly output on the complete unified semantic space based on the original unnormalized score vector.

Description

Technical Field

[0001] This invention relates to the field of underwater acoustic signal processing technology, and in particular to a multi-source side-scan sonar joint semantic segmentation method and system for heterogeneous annotation ranges. Background Technology

[0002] Side-scan sonar characterizes the underwater environment through spatially varying acoustic backscattering, enabling imaging of a wide range of underwater targets and seabed. In addition to direct target responses, side-scan sonar images also contain acoustic shadows, seabed textures, distance-dependent intensity attenuation, water column effects, and noise artifacts introduced by the sensing equipment. Therefore, semantic segmentation of acoustic images requires joint consideration of highlight morphology, shadow geometry, local context, and cross-track structural information.

[0003] Pixel-by-pixel classification using deep neural networks provides an effective means for semantic segmentation of sonar images. However, existing publicly available side-scan sonar datasets differ significantly from each other in sensor conditions, measurement geometry, target morphology, annotation accuracy, and semantic distinction definitions, resulting in heterogeneous annotation ranges among different datasets.

[0004] Each multi-source side-scan sonar dataset contains a two-dimensional side-scan sonar acoustic intensity image and annotation information associated with the acoustic intensity image. Based on the different annotation granularity and annotation protocol type, the side-scan sonar datasets are divided into two categories: The first category is a dataset containing pixel-level annotation masks, including a two-dimensional acoustic image and a mask image that corresponds to the image pixel by pixel. The mask image contains labeled target entity pixels with fine object contours (i.e., labeled targets) and background pixels where no target of interest appears. The second category is a dataset containing target detection box annotations, including a two-dimensional acoustic image and the target object location and category attributes marked by a rectangular bounding box. No fine category division is set in the area outside the detection box.

[0005] For example, one side-scan sonar dataset may only provide pixel-level masking for "shipwreck" targets, while defining other areas uniformly as "background"; another dataset may only provide target detection box annotations for "water pipes" or "mines"; and yet another dataset may specifically classify the substrate types of benthic habitats into multiple categories.

[0006] When training on multiple datasets, simply merging all labeled datasets can lead to incompatible labeling protocols, causing the same semantic category to be labeled as the target class in one dataset and as the background class in another. Existing techniques employing a "hard masking" strategy, which forcibly masks the scores of unlabeled invalid classes to extreme negative values ​​during training, completely cuts off the gradients of invalid classes in the network. During inference, when the masking is removed for full-space prediction, the unconstrained, unnormalized scores of invalid classes are prone to disordered drift, leading to misclassification collapse.

[0007] Therefore, how to achieve high-precision and robust joint semantic segmentation of heterogeneous annotation ranges while maintaining the integrity of the annotations of each multi-source dataset is a technical problem that urgently needs to be solved in the field of underwater acoustic image processing. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a method and system for joint semantic segmentation of multi-source side-scan sonar datasets with heterogeneous annotation ranges. It aims to solve the technical problems of joint training difficulties caused by incompatible annotation systems in multi-source side-scan sonar datasets, as well as the technical problems of unnormalized score drift and misclassification collapse that are easily caused by traditional hard masking mechanisms during the inference stage.

[0009] On the one hand, this invention provides a multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges, comprising the following steps: Step S1: Establish a unified semantic space, which contains N preset semantic categories, where N is an integer greater than or equal to 2; based on the annotation protocol of each side-scan sonar dataset, determine the set of valid categories and the set of invalid categories for each side-scan sonar dataset in the unified semantic space, specifically including: For a dataset containing pixel-level labeled masks, its labeled targets are mapped to the corresponding target categories in the unified semantic space, and its background pixels are mapped to the unified background category; the corresponding target categories and the unified background category together constitute the set of valid categories of the dataset in the unified semantic space; all other preset semantic categories contained in the unified semantic space besides the set of valid categories constitute the set of invalid categories of the dataset. For a dataset containing object detection box annotations, the position and attributes of each object detection box are read; the pixel region defined inside each detection box is mapped to the region mask of the corresponding object category in the unified semantic space; all pixel regions outside the detection box are set as ignored labels that do not participate in the loss calculation; when there are several detection boxes distributed in the same image, the pixels inside each detection box are spatially united; the preset object category corresponding to each detection box constitutes the effective category set of the dataset, and all other categories in the unified semantic space constitute the invalid category set of the dataset.

[0010] Step S2: Construct a shared side-scan sonar semantic segmentation network, perform prior processing on the side-scan sonar image to be processed to obtain the feature representation of the side-scan sonar image to be processed, and input it into the side-scan sonar semantic segmentation network to extract features pixel by pixel and output an N-dimensional unnormalized score vector per pixel in a unified semantic space. The side-scan sonar semantic segmentation network adopts a fully convolutional encoder-decoder architecture, specifically including: a feature extraction encoder, a multi-scale context aggregation module, a cascaded upsampling decoder, and a classification head connected in sequence; The feature extraction encoder consists of a multi-level cascaded two-dimensional convolutional layer, a batch normalization layer, a nonlinear activation layer, and a max pooling layer with a stride of 2. It performs spatial downsampling on the feature representation step by step and extracts multi-scale deep acoustic semantic maps. The multi-scale context aggregation module is connected to the end of the feature extraction encoder. It adopts a multi-receptive field dilated convolution or spatial pyramid pooling structure to capture the large-scale structural context of underwater acoustics and the association between dark shadows for multi-scale deep acoustic semantic maps, and outputs feature maps. The cascaded upsampling decoder gradually restores the spatial resolution of the feature map through multi-level bilinear interpolation upsampling, and cascades and fuses shallow high-resolution skip connection features from the feature extraction encoder to output a decoded and restored spatial high-dimensional feature map. The classification header is connected to the output of the cascaded upsampling decoder and contains several convolutional kernels with a size of [missing information]. The projection convolutional layer maps the high-dimensional feature map pixel by pixel, and outputs a pixel-wise unnormalized score vector with the same spatial dimension as the input image and the number of output channels equal to the total number of categories N in the unified semantic space.

[0011] The prior processing includes the following steps: Step A1: Extract physical features from the side-scan sonar image to be processed and generate a five-channel acoustic physical mapping representation with spatial dimensions aligned with the original image; The five-channel acoustic physical mapping representation specifically includes: Original intensity channel: Obtain the original intensity value of the side-scan sonar image to be processed, and linearly normalize it to the range of [0,1] according to the physical range of the image; Range-normalized intensity channel: Calculate the grayscale column mean μ at each distance grid column position along the cross-track range direction of the sonar transducer. r and standard deviation σ r The gain of each row of pixels is corrected column by column using a distance gain compensation formula. Local contrast channel: Construct a symmetrical smooth sliding window with a size of height × width centered on the current pixel, and calculate the local standard deviation of the acoustic intensity within the sliding window; Shadow Response Channel: Adaptively determine the low-energy truncation threshold for intensity shadows by combining the full-image signal-to-noise ratio distribution of the side-scan sonar image, perform acoustic shadow capture on dark areas with pixel intensity below the threshold, and output a binarized or soft-edge shadow prior probability response map. Reliability weight channel: The local signal-to-noise ratio is estimated based on the ratio of forward scattering intensity in the local neighborhood of the image to the reverberation energy of the background seabed. A pixel-by-pixel quality weight is constructed, and attenuation confidence weights are assigned to areas with large attenuation at long distances and areas with strong near-field distortion of water columns.

[0012] Step A2: Input the five-channel acoustic physical mapping representation into the convolutional projection layer, perform linear dimensionality reduction and feature interaction recombination in the channel dimension, and project a dense feature representation with 3 channels, which is the feature representation of the side-scan sonar image to be processed, and input it into the side-scan sonar semantic segmentation network. The convolutional projection layer is the physical front end for the input adaptation of the side-scan sonar semantic segmentation network, specifically including several two-dimensional convolutional kernels with a kernel size of 1×1 or 3×3, batch normalization operation, and activation function. Step S3: Jointly train the side-scan sonar semantic segmentation network for the multi-source heterogeneous annotation range; Step S3.1: Dataset Conditional Soft Masking Processing: Based on the effective class set and invalid class set of the side-scan sonar dataset to which the current input side-scan sonar image to be processed belongs, the component-wise conditional adjustment is performed on the pixel-wise unnormalized score vector output by the side-scan sonar semantic segmentation network. The component-wise conditional adjustment is as follows: for semantic categories belonging to the valid category set of the side-scan sonar dataset, the corresponding unnormalized score component in the pixel-wise unnormalized score vector remains unchanged; for semantic categories belonging to the invalid category set of the side-scan sonar dataset, a preset negative attenuation bias value is superimposed on the corresponding unnormalized score component. The negative attenuation bias value Δ is equal to the natural logarithm of the preset soft masking attenuation coefficient α, i.e., Δ=ln(α), where the soft masking attenuation coefficient α∈[0.01,0.5]. Step S3.2: Normalize the probability of the score vector after soft masking, and calculate the cross-entropy loss by combining the real supervision labels generated by mapping within the effective category set in the current input side-scan sonar image to be processed. Update the connection weights of each layer of the side-scan sonar semantic segmentation network according to the loss gradient backpropagation to complete the joint training of the multi-source heterogeneous side-scan sonar dataset; In the dataset containing object detection box annotations, pixels outside the detection box that are marked as ignored are automatically masked and the loss gradient is not backpropagated. Step S4: In the inference and deployment stage of multi-source side-scan sonar joint semantic segmentation, the side-scan sonar image to be segmented is input into the jointly trained side-scan sonar semantic segmentation network. Based on the complete unified semantic space with complete removal and no superimposed negative attenuation bias value, probability normalization is directly performed according to the original unnormalized score vector output by the classification head. The channel category index where the probability maximum value is located is taken to generate an unbiased pixel-by-pixel semantic segmentation map covering all N categories. This method also includes invalid class behavior evaluation, specifically: based on the unnormalized score vector of the original output of the side-scan sonar semantic segmentation network in the complete unified semantic space, the invalid class activation rate and invalid class score bias are calculated. The invalid class activation rate is the proportion of pixels whose highest invalid class unnormalized score exceeds the highest valid class unnormalized score among the total evaluated pixels; The invalid class score bias is the average of the difference between the highest invalid class unnormalized score and the highest valid class unnormalized score across all pixels in the image.

[0013] When a new semantic category is introduced into the new side-scan sonar dataset, the new semantic category is expanded into a unified semantic space, and the number of output channels of the classification head of the side-scan sonar semantic segmentation network is adjusted to the updated total number of categories. The calculation rule for superimposing the negative attenuation bias value in step S3 remains unchanged.

[0014] On the other hand, the present invention also provides a multi-source side-scan sonar joint semantic segmentation system for heterogeneous labeling range, which is used to implement the aforementioned multi-source side-scan sonar joint semantic segmentation method for heterogeneous labeling range, including: a label space definition module, a segmentation network module, a soft masking loss calculation module, a network training module, and an inference prediction module. The label space definition module establishes a unified semantic space and determines the effective category set and invalid category set for each side-scan sonar dataset according to the annotation protocol. The segmentation network module outputs an N-dimensional unnormalized score vector in a unified semantic space for each pixel of the input side-scan sonar image. The soft masking loss calculation module adds a negative decay bias value to the unnormalized score of invalid classes during the training phase and calculates the probability-normalized cross-entropy loss. The network training module calculates the gradient and updates the network parameters based on the cross-entropy loss. The inference prediction module removes the negative decay bias value during the inference stage and directly outputs a pixel-by-pixel semantic segmentation map on the complete unified semantic space based on the original unnormalized score vector.

[0015] The segmentation network module also includes an acoustic physical mapping preprocessing submodule, which is used to construct the original intensity channel, range-normalized intensity channel, local contrast channel, shadow response channel and reliability weight channel, and input them into the side-scan sonar semantic segmentation network after convolutional dimensionality reduction.

[0016] The aforementioned multi-source side-scan sonar joint semantic segmentation system for heterogeneous annotation ranges also includes a diagnostic evaluation module, used to calculate the invalid class activation rate and invalid class score bias based on the unnormalized category score vector.

[0017] Thirdly, this application proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges.

[0018] The beneficial effects of adopting the above technical solution are as follows: This invention provides a multi-source side-scan sonar joint semantic segmentation method and system for heterogeneous annotation ranges. Compared with the prior art, this invention has the following advantages: Suppressing invalid class score drift: During the training phase, negative decay bias values ​​are superimposed so that invalid classes are decayed while still retaining non-zero gradients. When invalid class scores are too high, corrective suppression is applied to them, which effectively avoids the disordered score drift and inference prediction collapse caused by gradient cut-off in traditional hard masking.

[0019] Decoupling of inference and dataset identity: The model does not need to input any dataset identification information during the inference stage and can directly perform pixel-by-pixel prediction in the complete semantic space, which improves the generalization ability to unknown sonar data.

[0020] Fusion of acoustic physical priors: By constructing a five-channel acoustic physical mapping representation, sonar gain, contrast, acoustic shadowing and signal-to-noise ratio features are explicitly fused to improve segmentation accuracy in complex acoustic backgrounds.

[0021] Easy to expand: When adding new datasets, only the unified semantic space needs to be expanded and the number of classification output channels needs to be adjusted. The algorithm calculation rules do not need to be changed, which has extremely high engineering practical value. Attached Figure Description

[0022] Figure 1 Overall flowchart of the multi-source side-scan sonar joint semantic segmentation method according to an embodiment of the present invention; Figure 2 Schematic diagram of the multi-source side-scan sonar joint semantic segmentation system according to an embodiment of the present invention; Figure 3 A schematic diagram showing the logical comparison between the soft masking process implemented in this invention and traditional methods in score adjustment. Detailed Implementation

[0023] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0024] Example 1: On the one hand, this invention provides a multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges, such as... Figure 1 As shown, it includes the following steps: Step S1: Establish a unified semantic space, which contains N preset semantic categories, where N is an integer greater than or equal to 2; based on the annotation protocol of each side-scan sonar dataset, determine the set of valid categories and the set of invalid categories for each side-scan sonar dataset in the unified semantic space, specifically including: For a dataset containing pixel-level labeled masks, its labeled targets are mapped to the corresponding target categories in the unified semantic space, and its background pixels are mapped to the unified background category; the corresponding target categories and the unified background category together constitute the set of valid categories of the dataset in the unified semantic space; all other preset semantic categories contained in the unified semantic space besides the set of valid categories constitute the set of invalid categories of the dataset. For a dataset containing object detection box annotations, the position and attributes of each object detection box are read; the pixel region defined inside each detection box is mapped to the region mask of the corresponding object category in the unified semantic space; all pixel regions outside the detection box are set as ignored labels that do not participate in the loss calculation; when there are several detection boxes distributed in the same image, the pixels inside each detection box are spatially united; the preset object category corresponding to each detection box constitutes the effective category set of the dataset, and all other categories in the unified semantic space constitute the invalid category set of the dataset.

[0025] In this embodiment, a unified semantic space is established containing 8 preset semantic categories (set as category 0 to category 7): Category 0 is a uniform background; Category 1 targets are shipwrecks; Category 2 is underwater pipes; Category 3 targets are mines; Categories 4 through 7 are rocky substrate, coarse sediments, fine sediments, and biogenic substrate, respectively.

[0026] Four representative publicly available side-scan sonar datasets were introduced for joint training. The mapping and effective class set of each dataset were determined as follows: The AI4Shipwrecks dataset provides pixel-level masks where the target is mapped to class 1 and the background to class 0. The valid class set contains classes 0 and 1; the invalid class set contains classes 2 through 7.

[0027] The SubPipe dataset provides object detection bounding box annotations. Pixels inside the box are mapped to category 2, while pixels outside the box are set as ignore labels that do not participate in the loss calculation. The valid category set includes category 0 and category 2.

[0028] The MCM dataset provides object detection bounding box annotations. Pixels inside the bounding box are mapped to category 3, while pixels outside the bounding box are set to ignore. The valid category set includes category 0 and category 3.

[0029] The BenthiCat dataset provides multi-class pixel masks for benthic habitats. The valid class set includes class 0 and classes 4 through 7.

[0030] Step S2: Construct a shared side-scan sonar semantic segmentation network, perform prior processing on the side-scan sonar image to be processed to obtain the feature representation of the side-scan sonar image to be processed, and input it into the side-scan sonar semantic segmentation network to extract features pixel by pixel and output an N-dimensional unnormalized score vector per pixel in a unified semantic space. The side-scan sonar semantic segmentation network adopts a fully convolutional encoder-decoder architecture, specifically including: a feature extraction encoder, a multi-scale context aggregation module, a cascaded upsampling decoder, and a classification head connected in sequence; The feature extraction encoder consists of a multi-level cascaded two-dimensional convolutional layer, a batch normalization layer, a nonlinear activation layer, and a max pooling layer with a stride of 2. It performs spatial downsampling on the feature representation step by step and extracts multi-scale deep acoustic semantic maps. The multi-scale context aggregation module is connected to the end of the feature extraction encoder. It adopts a multi-receptive field dilated convolution or spatial pyramid pooling structure to capture the large-scale structural context of underwater acoustics and the association between dark shadows for multi-scale deep acoustic semantic maps, and outputs feature maps. The cascaded upsampling decoder gradually restores the spatial resolution of the feature map through multi-level bilinear interpolation upsampling, and cascades and fuses shallow high-resolution skip connection features from the feature extraction encoder to output a decoded and restored spatial high-dimensional feature map. The classification header is connected to the output of the cascaded upsampling decoder and contains several convolutional kernels with a size of [missing information]. The projection convolutional layer maps the high-dimensional feature map pixel by pixel, and outputs a pixel-wise unnormalized score vector with the same spatial dimension as the input image and the number of output channels equal to the total number of categories N in the unified semantic space.

[0031] The prior processing includes the following steps: Step A1: Extract physical features from the side-scan sonar image to be processed and generate a five-channel acoustic physical mapping representation with spatial dimensions aligned with the original image; The five-channel acoustic physical mapping representation specifically includes: Original intensity channel: Obtain the original intensity value of the side-scan sonar image to be processed, and linearly normalize it to the range of [0,1] according to the physical range of the image, and save the original backscatter intensity distribution; Range-normalized intensity channel: Calculate the grayscale column mean μ at each distance grid column position along the cross-track range direction of the sonar transducer. r and standard deviation σ r The distance gain compensation formula is used to perform column-by-column gain correction on each row of pixels to eliminate the effects of geometric attenuation along beam propagation and transducer beam pointing offset. Local contrast channel: Construct a symmetrical smooth sliding window with a size of height × width (e.g., 7 × 7) centered on the current pixel, calculate the local standard deviation of acoustic intensity within the sliding window, and characterize the microscopic acoustic surface roughness and texture heterogeneity of underwater substrate and floating targets. Shadow Response Channel: Adaptively determine the low-energy truncation threshold for intensity shadows by combining the full-image signal-to-noise ratio distribution of the side-scan sonar image, perform acoustic shadow capture on dark areas with pixel intensity below the threshold, and output a binarized or soft-edge shadow prior probability response map. Reliability weight channel: The local signal-to-noise ratio is estimated based on the ratio of forward scattering intensity in the local neighborhood of the image to the reverberation energy of the background seabed. A pixel-by-pixel quality weight is constructed, and attenuation confidence weights are assigned to areas with large attenuation at long distances and areas with strong near-field distortion of water columns.

[0032] Step A2: Input the five-channel acoustic physical mapping representation into the convolutional projection layer, perform linear dimensionality reduction and feature interaction recombination in the channel dimension, and project a dense feature representation with 3 channels, which is the feature representation of the side-scan sonar image to be processed, and input it into the side-scan sonar semantic segmentation network; thus, while reusing the pre-trained network weights of the conventional encoder, the five-dimensional acoustic physical prior information is completely preserved. The convolutional projection layer is the physical front end for the input adaptation of the side-scan sonar semantic segmentation network, specifically including several two-dimensional convolutional kernels with a kernel size of 1×1 or 3×3, batch normalization operation, and activation function. In the five-channel acoustic physical mapping representation: The original intensity channel is the native grayscale intensity value of the side-scan sonar image; The range-normalized intensity channel is the intensity value after mean and standard deviation normalization correction along the gain distance direction; The local contrast channel is generated by calculating the local standard deviation of the image through a sliding window. The shadow response channel extracts the acoustic shadow region based on a preset intensity threshold using adaptive binarization. The reliability weight channel generates a pixel-by-pixel quality assessment weight map based on the local signal-to-noise ratio of the sonar image.

[0033] Step S3: Jointly train the side-scan sonar semantic segmentation network for the multi-source heterogeneous annotation range; Step S3.1: Dataset Conditional Soft Masking Processing: Based on the effective class set and invalid class set of the side-scan sonar dataset to which the current input side-scan sonar image to be processed belongs, the component-wise conditional adjustment is performed on the pixel-wise unnormalized score vector output by the side-scan sonar semantic segmentation network. The component-wise conditional adjustment is as follows: the side-scan sonar semantic segmentation network outputs a pixel-wise unnormalized score vector with N dimensional components for each pixel of the side-scan sonar image, and performs conditional adjustment on each component in the pixel vector to be adjusted: for semantic categories belonging to the valid category set of the side-scan sonar dataset, the corresponding unnormalized score component in the pixel-wise unnormalized score vector remains unchanged; for semantic categories belonging to the invalid category set of the side-scan sonar dataset, a preset negative attenuation bias value is superimposed on the corresponding unnormalized score component. The negative attenuation bias value Δ is equal to the natural logarithm of the preset soft masking attenuation coefficient α, i.e., Δ=ln(α), where α is a real number that satisfies 0<α≤1 (ln(α)≤0 at this time). Since no attenuation occurs when α=1 and degenerates into hard masking and cuts off the gradient when α→0, in the preferred implementation mode of the multi-source side-scan sonar joint semantic segmentation of the present invention, the engineering optimization range of the soft masking attenuation coefficient α is limited to 0.01 to 0.5.

[0034] Step S3.2: Normalize the probability of the score vector after soft masking, and calculate the cross-entropy loss by combining the real supervision labels generated by mapping within the effective category set in the current input side-scan sonar image to be processed. Update the connection weights of each layer of the side-scan sonar semantic segmentation network according to the loss gradient backpropagation to complete the joint training of the multi-source heterogeneous side-scan sonar dataset; In the dataset containing object detection box annotations, pixels outside the detection box that are marked as ignored are automatically masked and the loss gradient is not backpropagated. In this embodiment, after the network receives a sonar image as input, it outputs an 8-dimensional unnormalized score vector for each pixel. The training process is as follows: If the current pixel comes from the dataset AI4Shipwrecks, its valid categories are category 0 and category 1.

[0035] The soft masking attenuation coefficient is set to α=0.1, which corresponds to a negative attenuation bias of approximately -2.3.

[0036] For the unnormalized scores of categories 2 to 7 that belong to invalid categories, a negative attenuation bias of -2.3 is uniformly added; for categories 0 and 1 that belong to valid categories, the original scores are kept unchanged.

[0037] The adjusted score vector is probability normalized, and the cross-entropy loss is calculated to update the network parameters. Since invalid classes still participate in the normalization calculation after the bias is stacked, they can generate non-zero correction gradients when their scores rise abnormally, thus constraining the network.

[0038] Step S4: In the inference and deployment stage of multi-source side-scan sonar joint semantic segmentation, the side-scan sonar image to be segmented is input into the jointly trained side-scan sonar semantic segmentation network. Based on the complete unified semantic space with complete removal and no superimposed negative attenuation bias value, probability normalization is directly performed according to the original unnormalized score vector output by the classification head. The channel category index where the probability maximum value is located is taken to generate an unbiased pixel-by-pixel semantic segmentation map covering all N categories. In this embodiment, the side-scan sonar image to be segmented is directly input into the trained segmentation network; Completely remove conditional soft masking operations and do not add decay bias to scores for any category; The probability is normalized directly based on the original 8-dimensional unnormalized score output by the network, and the category with the highest probability is taken as the final segmentation result for the pixel.

[0039] This method also includes invalid class behavior evaluation, specifically: based on the unnormalized score vector of the original output of the side-scan sonar semantic segmentation network in the complete unified semantic space, the invalid class activation rate and invalid class score bias are calculated. The invalid class activation rate is the proportion of pixels whose highest invalid class unnormalized score exceeds the highest valid class unnormalized score among the total evaluated pixels; The invalid class score bias is the average of the difference between the highest invalid class unnormalized score and the highest valid class unnormalized score across all pixels in the image.

[0040] When a new semantic category is introduced into the new side-scan sonar dataset, the new semantic category is expanded into a unified semantic space, and the number of output channels of the classification head of the side-scan sonar semantic segmentation network is adjusted to the updated total number of categories. The calculation rule for superimposing the negative attenuation bias value in step S3 remains unchanged.

[0041] On the other hand, the present invention also provides a multi-source side-scan sonar joint semantic segmentation system for heterogeneous annotation ranges, used to implement the aforementioned multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges, such as... Figure 2 As shown, it includes: a label space definition module, a segmentation network module, a soft masking loss calculation module, a network training module, and an inference and prediction module; The label space definition module establishes a unified semantic space and determines the effective category set and invalid category set for each side-scan sonar dataset according to the annotation protocol. The segmentation network module outputs an N-dimensional unnormalized score vector in a unified semantic space for each pixel of the input side-scan sonar image. The soft masking loss calculation module adds a negative decay bias value to the unnormalized score of invalid classes during the training phase and calculates the probability-normalized cross-entropy loss. The network training module calculates the gradient and updates the network parameters based on the cross-entropy loss. The inference prediction module removes the negative decay bias value during the inference stage and directly outputs a pixel-by-pixel semantic segmentation map on the complete unified semantic space based on the original unnormalized score vector.

[0042] The segmentation network module also includes an acoustic physical mapping preprocessing submodule, which is used to construct the original intensity channel, range-normalized intensity channel, local contrast channel, shadow response channel and reliability weight channel, and input them into the side-scan sonar semantic segmentation network after convolutional dimensionality reduction.

[0043] The aforementioned multi-source side-scan sonar joint semantic segmentation system for heterogeneous annotation ranges also includes a diagnostic evaluation module, used to calculate the invalid class activation rate and invalid class score bias based on the unnormalized category score vector.

[0044] Example 2: To verify the technical effectiveness of this invention, the soft masking method of this invention was compared with the traditional hard masking method under the same multi-source side-scan sonar dataset combination. The evaluation metrics included mean intersection-over-union ratio (MIU), invalid class activation rate, and invalid class score bias, such as... Figure 3 As shown.

[0045] Test results show that: Improved segmentation accuracy: By using the soft masking method of this invention (when the attenuation coefficient is 0.1), the average intersection-union ratio of semantic segmentation is significantly improved compared with the traditional hard masking method.

[0046] Score drift is effectively suppressed: Traditional hard masking methods have a high activation rate of invalid classes during the inference phase, and the score bias of invalid classes is positive, indicating that there is a significant score drift phenomenon of invalid classes; while the method of this invention significantly reduces the activation rate of invalid classes to a low level, and keeps the score bias of invalid classes negative, indicating that the scores of valid classes are absolutely dominant, and successfully eliminates the risk of prediction collapse.

[0047] The datasets AI4Shipwrecks, SubPipe, MCM, and BenthiCat used in the embodiments are for illustrative purposes only. The soft masking joint training mechanism of the present invention is also applicable to any newly added side-scan sonar dataset.

[0048] Example 3: This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges.

[0049] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0050] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0051] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of the methods disclosed herein and their equivalents, then the intent of this disclosure also includes such modifications and variations.

Claims

1. A multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges, characterized in that, Includes the following steps: Step S1: Establish a unified semantic space, which contains N preset semantic categories, where N is an integer greater than or equal to 2; based on the annotation protocol of each side-scan sonar dataset, determine the set of valid categories and the set of invalid categories for each side-scan sonar dataset in the unified semantic space; Step S2: Construct a shared side-scan sonar semantic segmentation network, perform prior processing on the side-scan sonar image to be processed to obtain the feature representation of the side-scan sonar image to be processed, and input it into the side-scan sonar semantic segmentation network to extract features pixel by pixel and output an N-dimensional unnormalized score vector per pixel in a unified semantic space. Step S3: Jointly train the side-scan sonar semantic segmentation network for the multi-source heterogeneous annotation range; Step S3.1: Dataset Conditional Soft Masking Processing: Based on the effective class set and invalid class set of the side-scan sonar dataset to which the current input side-scan sonar image to be processed belongs, the component-wise conditional adjustment is performed on the pixel-wise unnormalized score vector output by the side-scan sonar semantic segmentation network. Step S3.2: Normalize the probability of the score vector after soft masking, and calculate the cross-entropy loss by combining the real supervision labels generated by mapping within the effective category set in the current input side-scan sonar image to be processed. Update the connection weights of each layer of the side-scan sonar semantic segmentation network according to the loss gradient backpropagation to complete the joint training of the multi-source heterogeneous side-scan sonar dataset; In the dataset containing object detection box annotations, pixels outside the detection box that are marked as ignored are automatically masked and the loss gradient is not backpropagated. Step S4: In the inference and deployment stage of multi-source side-scan sonar joint semantic segmentation, the side-scan sonar image to be segmented is input into the jointly trained side-scan sonar semantic segmentation network. Based on the complete unified semantic space with all negative attenuation bias values ​​removed and without any superimposed negative attenuation bias values, probability normalization is directly performed on the original pixel-by-pixel unnormalized score vector output by the classification head. The channel category index with the maximum probability is taken to generate an unbiased pixel-by-pixel semantic segmentation map covering all N categories.

2. The multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges according to claim 1, characterized in that, Step S1 specifically includes: For a dataset containing pixel-level labeled masks, its labeled targets are mapped to the corresponding target categories in the unified semantic space, and its background pixels are mapped to the unified background category; the corresponding target categories and the unified background category together constitute the set of valid categories of the dataset in the unified semantic space; all other preset semantic categories contained in the unified semantic space besides the set of valid categories constitute the set of invalid categories of the dataset. For a dataset containing object detection box annotations, the position and attributes of each object detection box are read; the pixel region defined inside each detection box is mapped to the region mask of the corresponding object category in the unified semantic space; all pixel regions outside the detection box are set as ignored labels that do not participate in the loss calculation; when there are several detection boxes distributed in the same image, the pixels inside each detection box are spatially united; the preset object category corresponding to each detection box constitutes the effective category set of the dataset, and all other categories in the unified semantic space constitute the invalid category set of the dataset.

3. The multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges according to claim 1, characterized in that, The side-scan sonar semantic segmentation network described in step S2 adopts a fully convolutional encoder-decoder architecture, which specifically includes: a feature extraction encoder, a multi-scale context aggregation module, a cascaded upsampling decoder, and a classification head connected in sequence. The feature extraction encoder consists of a multi-level cascaded two-dimensional convolutional layer, a batch normalization layer, a nonlinear activation layer, and a max pooling layer with a stride of 2. It performs spatial downsampling on the feature representation step by step and extracts multi-scale deep acoustic semantic maps. The multi-scale context aggregation module is connected to the end of the feature extraction encoder. It adopts a multi-receptive field dilated convolution or spatial pyramid pooling structure to capture the large-scale structural context of underwater acoustics and the association between dark shadows for multi-scale deep acoustic semantic maps, and outputs feature maps. The cascaded upsampling decoder gradually restores the spatial resolution of the feature map through multi-level bilinear interpolation upsampling, and cascades and fuses shallow high-resolution skip connection features from the feature extraction encoder to output a decoded and restored spatial high-dimensional feature map. The classification header is connected to the output of the cascaded upsampling decoder and contains several convolutional kernels with a size of [missing information]. The projection convolutional layer maps the high-dimensional feature map pixel by pixel, and outputs a pixel-wise unnormalized score vector with the same spatial dimension as the input image and the number of output channels equal to the total number of categories N in the unified semantic space.

4. The multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges according to claim 1, characterized in that, The prior processing described in step S2 includes the following steps: Step A1: Extract physical features from the side-scan sonar image to be processed and generate a five-channel acoustic physical mapping representation with spatial dimensions aligned with the original image; Step A2: Input the five-channel acoustic physical mapping representation into the convolutional projection layer, perform linear dimensionality reduction and feature interaction recombination in the channel dimension, and project a dense feature representation with 3 channels, which is the feature representation of the side-scan sonar image to be processed, and input it into the side-scan sonar semantic segmentation network.

5. The multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges according to claim 4, characterized in that, The five-channel acoustic physical mapping representation specifically includes: Original intensity channel: Obtain the original intensity value of the side-scan sonar image to be processed, and linearly normalize it to the range of [0,1] according to the physical range of the image; Range-normalized intensity channel: Calculate the grayscale column mean μ at each distance grid column position along the cross-track range direction of the sonar transducer. r and standard deviation σ r The gain of each row of pixels is corrected column by column using a distance gain compensation formula. Local contrast channel: Construct a symmetrical smooth sliding window with a size of height × width centered on the current pixel, and calculate the local standard deviation of the acoustic intensity within the sliding window; Shadow Response Channel: Adaptively determine the low-energy truncation threshold for intensity shadows by combining the full-image signal-to-noise ratio distribution of the side-scan sonar image, perform acoustic shadow capture on dark areas with pixel intensity below the threshold, and output a binarized or soft-edge shadow prior probability response map. Reliability weight channel: The local signal-to-noise ratio is estimated based on the ratio of forward scattering intensity in the local neighborhood of the image to the reverberation energy of the background seabed. A pixel-by-pixel quality weight is constructed, and attenuation confidence weights are assigned to areas with large attenuation at long distances and areas with strong near-field distortion of water columns.

6. The multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges according to claim 4, characterized in that, The convolutional projection layer is the physical front end for the input adaptation of the side-scan sonar semantic segmentation network. Specifically, it includes several two-dimensional convolutional kernels with a kernel size of 1×1 or 3×3, batch normalization operation, and activation function.

7. The multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges according to claim 1, characterized in that, The component-wise conditional adjustment in step S3.1 is as follows: for semantic categories belonging to the valid category set of the side-scan sonar dataset, keep the corresponding unnormalized score component in the pixel-wise unnormalized score vector unchanged; for semantic categories belonging to the invalid category set of the side-scan sonar dataset, add a preset negative attenuation bias value to the corresponding unnormalized score component. The negative attenuation bias value Δ is equal to the natural logarithm of the preset soft masking attenuation coefficient α, i.e., Δ=ln(α), where the soft masking attenuation coefficient α∈[0.01,0.5].

8. The multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges according to claim 1, characterized in that, This also includes evaluation of invalid class behaviors, specifically: based on the unnormalized score vector of the original output of the side-scan sonar semantic segmentation network in the complete unified semantic space, the invalid class activation rate and invalid class score bias are calculated. The invalid class activation rate is the proportion of pixels whose highest invalid class unnormalized score exceeds the highest valid class unnormalized score among the total evaluated pixels; The invalid class score bias is the average of the difference between the highest invalid class unnormalized score and the highest valid class unnormalized score across all pixels in the image. When a new semantic category is introduced into the new side-scan sonar dataset, the new semantic category is expanded into a unified semantic space, and the number of output channels of the classification head of the side-scan sonar semantic segmentation network is adjusted to the updated total number of categories. The calculation rule for superimposing the negative attenuation bias value in step S3 remains unchanged.

9. A multi-source side-scan sonar joint semantic segmentation system for heterogeneous annotation ranges, used to implement the multi-source side-scan sonar joint semantic segmentation method for heterogeneous annotation ranges as described in claim 1, characterized in that, include: The system includes a label space definition module, a segmentation network module, a soft masking loss calculation module, a network training module, and an inference and prediction module. The label space definition module establishes a unified semantic space and determines the effective category set and invalid category set for each side-scan sonar dataset according to the annotation protocol. The segmentation network module outputs an N-dimensional unnormalized score vector in a unified semantic space for each pixel of the input side-scan sonar image. The soft masking loss calculation module adds a negative decay bias value to the unnormalized score of invalid classes during the training phase and calculates the probability-normalized cross-entropy loss. The network training module calculates the gradient and updates the network parameters based on the cross-entropy loss. The inference prediction module removes the negative decay bias value during the inference stage and directly outputs a pixel-by-pixel semantic segmentation map in the complete unified semantic space based on the original unnormalized score vector. The segmentation network module also includes an acoustic physical mapping preprocessing submodule, which is used to construct the original intensity channel, range-normalized intensity channel, local contrast channel, shadow response channel and reliability weight channel, and input them into the side-scan sonar semantic segmentation network after convolutional dimensionality reduction.

10. A multi-source side-scan sonar joint semantic segmentation system for heterogeneous annotation ranges according to claim 9, characterized in that, It also includes a diagnostic assessment module for calculating the invalid class activation rate and invalid class score bias based on the unnormalized class score vector.

Images