Forward-looking sonar image semantic segmentation method based on multi-scale frequency domain

Abstract

The invention discloses a forward-looking sonar image semantic segmentation method based on a multi-scale frequency domain, and the method comprises the steps: constructing a forward-looking sonar image semantic segmentation network based on multi-scale frequency domain noise reduction, and training the forward-looking sonar image semantic segmentation network; performing semantic segmentation on the forward-looking sonar image by using the trained semantic segmentation network of the forward-looking sonar image to obtain a semantic segmentation result; the forward-looking sonar image semantic segmentation network comprises a basic feature extraction network, a context channel noise reduction module, a multi-scale frequency domain decoding module and a semantic segmentation output module. Wherein the basic feature extraction network term performs semantic feature extraction on an input forward-looking sonar image, the context channel noise reduction module is used for enhancing the extracted semantic features and corresponding context features, and the multi-scale frequency domain decoding module is used for decoding the enhanced features to obtain output features. And the semantic segmentation output module is used for classifying the output features and outputting a semantic segmentation result.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and underwater acoustic engineering, and in particular to a semantic segmentation method of forward-looking sonar images based on multi-scale frequency domain. Background Art

[0002] Forward-looking sonar (FLS) technology plays a key role in many fields such as marine resource exploration, underwater environment monitoring, marine engineering, military operations and national defense. Unlike optical detection methods that are severely affected by water quality conditions, acoustic detection methods show unique advantages in underwater environments. Some current sonar equipment can achieve a marine terrain detection range of 20 to 11,000 meters. Forward-looking sonar is widely used due to its high maturity and cost-effectiveness.

[0003] The semantic segmentation of forward-looking sonar images faces many challenges. Due to the complex underwater environment, sonar images are seriously interfered by environmental noise, including equipment noise, marine biological noise, reverberation noise and scattering noise, resulting in low image resolution, poor contrast, increased black and white spots, and difficulty in extracting effective information. At the same time, occlusion and shadow interference make target features blurred or missing, and segmentation boundaries unclear, which can easily cause shadows to be confused with other categories, affecting the accuracy of semantic information. In addition, the imaging principle of forward-looking sonar leads to similarity noise interference in the image, and a large number of similar features appear. As the resolution of the feature map decreases, the discrimination of features extracted at the same level decreases, further affecting the segmentation accuracy.

[0004] In recent years, traditional semantic segmentation methods, such as VT-HMRF-FCM, graph-based techniques, gaze-based segmentation methods, and the ilastik toolkit, have performed poorly in feature extraction and result prediction when processing forward-looking sonar images due to image complexity and segmentation accuracy requirements. Deep learning-based methods, including convolutional neural networks (CNNs) such as FCNs, UNet, Deeplabv3+, and Transformer-based methods such as Vision Transformer, Segmenter, Segformer, also have limitations when applied to forward-looking sonar images. CNNs face the challenge of extracting effective features when processing sonar images, while Transformers have poor performance due to sparse sonar data, high noise, high data volume requirements, large computational requirements, and difficulty in obtaining labeled data. Therefore, there is an urgent need for a new method to effectively integrate local and global features and enhance the model in high-noise environments. Summary of the invention

[0005] The purpose of the present invention is to provide a forward-looking sonar image semantic segmentation method based on multi-scale frequency domain to solve a series of problems faced by the semantic segmentation of forward-looking sonar images, to meet the needs of precise analysis in multiple fields, and to achieve more accurate and efficient segmentation effects.

[0006] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0007] A semantic segmentation method for forward-looking sonar images based on multi-scale frequency domain, comprising:

[0008] Construct a semantic segmentation network for forward-looking sonar images based on multi-scale frequency domain denoising, and train the semantic segmentation network for forward-looking sonar images;

[0009] The forward-looking sonar image is semantically segmented using a trained forward-looking sonar image semantic segmentation network to obtain a semantic segmentation result; the forward-looking sonar image semantic segmentation network includes a basic feature extraction network, a context channel denoising module, a multi-scale frequency domain decoding module and a semantic segmentation output module; wherein the basic feature extraction network is used to perform semantic feature extraction on the input forward-looking sonar image, the context channel denoising module is used to enhance the extracted semantic features and the corresponding context features, the multi-scale frequency domain decoding module is used to decode the enhanced features to obtain output features, and the semantic segmentation output module is used to classify the output features and output the semantic segmentation result.

[0010] Furthermore, the basic feature extraction network is used to extract semantic features and context features from the input forward-looking sonar image L; the basic feature extraction network includes four feature layers of different resolutions connected in sequence, each feature layer uses a PVTv2 encoder; after the forward-looking sonar image L is subjected to semantic feature and context extraction by the four feature layers, the feature map size output by the four feature layers from top to bottom is and The feature map output by each feature layer contains semantic features l i , i=1,2,3,4.

[0011] Furthermore, three context channel denoising modules are set, and the three context channel denoising modules are respectively connected to the first three feature layers of the basic feature extraction network to enhance the semantic features l1, l2, l3 extracted from the first three feature layers and the corresponding context features f1, f2, f3 to obtain the corresponding enhanced feature f1 C , The semantic feature l4 of the fourth feature layer does not pass through the feature enhancement module, and the semantic feature l4 is directly used as the corresponding enhanced feature Among them, the enhanced features As the context feature f3 of the third context channel denoising module, the enhanced feature As the context feature f2 of the third context channel denoising module, the enhanced feature As the context feature f1 of the third context channel denoising module;

[0012] Each context channel denoising module consists of three parts: feature extraction part, context channel attention block and global dependency building module.

[0013] Furthermore, the input feature of the feature extraction part is the semantic feature l output by the corresponding feature layer. i and context feature f i , the output feature is b i ; The feature extraction part includes the global maximum pooling layer, the feature dimension concatenation layer and the multi-layer perceptron; among them, the global maximum pooling layer is used for semantic features l i and context feature f i Each channel is processed and the maximum value is extracted as the feature representation of the channel; the semantic feature l i and context feature f i After passing through the global maximum pooling layer, the feature dimension splicing layer is used to splice according to the channel dimension, and finally passes through the multi-layer perceptron to obtain the deep feature b i .

[0014] Furthermore, the contextual channel attention block includes, from front to back, a LayerNorm layer, a contextual multi-head self-attention mechanism layer, a Dropout layer, a LayerNorm layer, a multi-layer perceptron, and a Dropout layer, where:

[0015] Deep Featuresb i After being processed by the LayerNorm layer, the query is processed using the multi-head self-attention mechanism MHSA key value Where R represents the real number space, B is H*W, H and W represent the height and width of the feature map entering the multi-head self-attention mechanism respectively; C i and C i+1 They are semantic features l i Number of channels and context features f i The number of channels; the contextual multi-head self-attention mechanism layer calculates the self-attention feature along the channel dimension based on the query Q, key K, and value V. The formula is:

[0016]

[0017] Among them, the superscript T represents the transposition operation, and Softmax is the normalized exponential function;

[0018] Self-attention feature F cmsaAfter the Dropout layer, and the deep feature b i The sum is the enhanced feature b ii , feature b ii After being processed by the LayerNorm layer, the multi-layer perceptron and the Dropout layer, it is combined with feature b again. ii Sum and get the enhanced deep feature b i C .

[0019] Furthermore, the global dependency module includes a feature splitting unit and a feature enhancement unit, wherein the feature splitting unit is used to enhance the deep feature b i C According to the semantic feature i and context feature f i The channel number relationship is split into features r in the channel dimension l and feature r f ; The feature enhancement unit is used to split the feature r l and feature r f Semantic features l i and context feature f i Multiply them to enhance the target channel features and attenuate similar noise channel features; at the same time, the learnable parameters α and β are introduced to determine the contribution of context features in the result prediction, and finally the filtered enhanced features f are obtained. i C , the formula is as follows:

[0020]

[0021] Furthermore, the multi-scale frequency domain decoding module corresponds to the basic feature extraction network, and includes four sequentially connected decoding blocks, each of which uses a MWC-VSS decoder; wherein the input of each decoding block is the output feature of the next layer of decoding blocks. i+1 and the corresponding enhanced features f output by the contextual channel denoising module i C , the output of the decoding block is the output feature o i ; For the underlying decoding block, the default input o is used during calculation i+1 is 0; the top-level decoding block finally obtains the output feature o1 and provides it to the semantic segmentation output module.

[0022] Furthermore, the MWC-VSS decoder includes a PatchExpand layer, a LayerNorm layer, a MWC-SS2D module and a DropPath layer connected in sequence, wherein: the PatchExpand layer is used to convert the feature o i+1 Restored to twice the original pixel size, and then combined with the enhanced feature fi C Sum and get the fusion feature g i , fusion feature g i After being processed by the LayerNorm layer, MWC-SS2D module and DropPath layer, it is combined with the fusion feature g i Sum and get the output feature o of the MWC-VSS decoder i ; The MWC-SS2D module is used to denoise the fused features and establish global dependencies.

[0023] Furthermore, the MWC-SS2D module has two branches, the first branch includes a linear layer, a multi-scale wavelet convolution attention module MWCA, a SiLU function, a SS2D module, and a LayerNorm layer; the second branch includes a linear layer, a SiLU function, and a linear layer;

[0024] Fusion feature g i After the LayerNorm layer processing, the feature t is obtained i And enter the MWC-SS2D module, feature t i In the first branch, after passing through the linear layer, it enters the multi-scale wavelet convolution attention module MWCA, in which different wavelet convolution layers are used to perform frequency domain denoising on the features, and after nonlinear transformation by the SiLU function, it is input into the SS2D module to establish global dependency, and then passes through the LayerNorm layer; feature t i In the second branch, after being processed by the linear layer and the SiLU function in turn, it is multiplied and fused with the output of the first branch, and finally processed by a linear layer to complete denoising and establish global dependencies.

[0025] Furthermore, the multi-scale wavelet convolutional attention module MWCA includes a convolutional attention module and four wavelet convolutional layers with different kernel sizes; the feature t in the first branch i After passing through the linear layer, it enters the convolutional attention module of MWCA to enhance its channel and spatial dimensions, and then enters four wavelet convolution layers to capture multi-scale local detail features. Finally, the output features of the module are obtained through the element-by-element addition fusion method.

[0026] Furthermore, the semantic segmentation output module includes a fully connected layer and a softmax function; the output feature o1 obtained by the multi-scale frequency domain decoding module is input into the fully connected layer to adjust the feature map size, and then each pixel of the feature map is classified through the softmax function, and finally the semantic segmentation result is obtained.

[0027] Furthermore, the loss function of the semantic segmentation network adopts the standard cross entropy loss function, which is expressed as:

[0028]

[0029] In the above formula, L represents the cross entropy loss function, N represents the pixel set of the forward-looking sonar image, that is, the product of the image height H and width W; W i Represents weight, which is used to deal with data imbalance problem; T i represents the true category label of pixel i; Represents the probability that the network predicts that pixel i belongs to a certain category.

[0030] Furthermore, multi-beam forward-looking sonar images are captured by sonar and resized. Then, the target categories contained in the forward-looking sonar images are manually interpreted and labeled to obtain a forward-looking sonar image dataset with sample labels and divide it into a training set and a test set.

[0031] A forward-looking sonar image semantic segmentation device comprises a processor, a memory and a computer program stored in the memory; the device is characterized in that when the processor executes the computer program, the forward-looking sonar image semantic segmentation method based on the multi-scale frequency domain is implemented.

[0032] A computer-readable storage medium having a computer program stored therein; wherein when the computer program is executed by a processor, the forward-looking sonar image semantic segmentation method based on multi-scale frequency domain is implemented.

[0033] Compared with the prior art, the present invention has the following technical features:

[0034] The network model constructed in the present invention proposes innovative solutions to various interferences faced by the semantic segmentation of forward-looking sonar images, such as occlusion shadows, similarity noise and environmental noise. By combining multi-scale wavelet convolution and visual state space modules, it effectively reduces interference, improves segmentation accuracy, and enhances target feature representation. In terms of feature extraction and fusion, it realizes multi-scale frequency domain feature extraction and global and local feature fusion, and optimizes computing resource utilization. It has excellent performance across multiple data sets, strong adaptability and generalization capabilities in complex environments, and can effectively cope with situations such as variable noise and target feature ambiguity. It reduces computational complexity, balances performance and resource requirements, and controls the number of parameters and memory usage while ensuring segmentation accuracy. It solves the limitations of existing technologies in processing semantic segmentation of forward-looking sonar images, provides reliable technical support for many fields such as marine resource exploration and underwater environment monitoring, promotes technological development in this field, and has important practical application value and social benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the process of the present invention;

[0036] Figure 2This is the structure diagram of the semantic segmentation network;

[0037] Figure 3 This is the semantic segmentation network model diagram;

[0038] Figure 4 Extract network structure diagram for basic features;

[0039] Figure 5 This is the structural diagram of the context channel denoising module;

[0040] Figure 6 This is the structural diagram of the feature extraction part of the context channel denoising module;

[0041] Figure 7 The structure diagram of the context channel attention block of the context channel denoising module;

[0042] Figure 8 Establish a global dependency structure diagram for the context channel denoising module;

[0043] Fig. 9 This is the structure diagram of the multi-scale frequency domain decoding module;

[0044] Fig.10 The MWC-VSS decoder structure diagram is a multi-scale frequency domain decoding module;

[0045] Fig.11 This is the block diagram of the MWC-SS2D module of the MWC-VSS decoder;

[0046] Fig.12 The MWCA module structure diagram of the MWC-SS2D module;

[0047] Fig.13 This is the structure diagram of the semantic segmentation output module;

[0048] Figure 14(a) , 14(b) 14(c) and 14(c) are the semantic segmentation effect diagrams of the forward-looking sonar image in this embodiment. DETAILED DESCRIPTION

[0049] The present invention provides a semantic segmentation method for forward-looking sonar images based on multi-scale frequency domain, which overcomes the defects of the prior art, improves segmentation accuracy, enhances model robustness, optimizes computing resource utilization, integrates multimodal information and solves practical application problems. Figure 1 , the method of the present invention comprises the following steps:

[0050] Construct a semantic segmentation network for forward-looking sonar images based on multi-scale frequency domain denoising, and train the semantic segmentation network for forward-looking sonar images;

[0051] Use the trained visual sonar image semantic segmentation network to perform semantic segmentation on the forward-looking sonar image to obtain the semantic segmentation result;

[0052] like Figure 2 and Figure 3 As shown, the forward-looking sonar image semantic segmentation network includes a basic feature extraction network, a context channel denoising module, a multi-scale frequency domain decoding module and a semantic segmentation output module; wherein the basic feature extraction network is used to perform semantic features on the input forward-looking sonar image, the context channel denoising module is used to enhance the extracted semantic features and the corresponding context features, the multi-scale frequency domain decoding module is used to decode the enhanced features to obtain output features, and the semantic segmentation output module is used to classify the output features and output the semantic segmentation results.

[0053] 1. Basic feature extraction network

[0054] like Figure 4 As shown in the figure, the basic feature extraction network is used to extract semantic features and context features from the input forward-looking sonar image L; the basic feature extraction network includes four feature layers of different resolutions connected in sequence, each of which uses the PVTv2 encoder; after the forward-looking sonar image L is subjected to semantic feature and context extraction by the four feature layers, the feature map size output by the four feature layers from top to bottom is and The feature map output by each feature layer contains semantic features l i , i=1,2,3,4; the feature map of semantic features is based on the feature extraction network, which captures the local and global semantic features of the sonar image layer by layer through hierarchical extraction. Its purpose is to retain the detail features to enhance the semantic understanding ability of low-resolution sonar images.

[0055] 2. Contextual channel denoising module

[0056] like Figure 3 As shown in FIG. 1 , three context channel denoising modules are set, and the three context channel denoising modules are respectively connected to the first three feature layers of the basic feature extraction network to enhance the semantic features l1, l2, l3 extracted by the first three feature layers and the corresponding context features f1, f2, f3 to obtain the corresponding enhanced feature f1 C , The semantic feature l4 of the fourth feature layer does not pass through the feature enhancement module, and the semantic feature l4 is directly used as the corresponding enhanced feature Participate in subsequent calculations. Among them, the enhanced features As the context feature f3 of the third context channel denoising module, the enhanced feature As the context feature f2 of the third context channel denoising module, the enhanced feature As the context feature f1 of the third context channel denoising module.

[0057] like Figure 5 As shown in the figure, each context channel denoising module includes three parts: feature extraction part, context channel attention block and global dependency establishment module, which is mainly used to solve the problem of similar noise interference in acoustic images; the context channel attention module is composed of five context channel attention blocks, and the number of heads of each block is 1, 2, 4, 8, and 16 respectively.

[0058] 2.1 Feature extraction part

[0059] like Figure 6 As shown in the figure, the input feature of the feature extraction part is the semantic feature l output by the corresponding feature layer i and context feature f i (i=1,2,3), the output feature is b i ; The feature extraction part includes the global maximum pooling layer, the feature dimension concatenation layer and the multi-layer perceptron; among them, the global maximum pooling layer is used for semantic features l i and context feature f i Each channel is processed and the maximum value is extracted as the feature representation of the channel; the semantic feature l i and context feature f i After passing through the global maximum pooling layer, the feature dimension splicing layer is used to splice according to the channel dimension, and finally passes through the multi-layer perceptron to obtain the deep feature b i (i=1,2,3).

[0060] 2.2 Contextual Channel Attention Block

[0061] like Figure 7 As shown, the input feature of the context channel attention block is the deep feature b i , the output feature is the enhanced deep feature b i C .

[0062] The contextual channel attention block is the basic unit of the contextual channel denoising module. It is based on the Channel-Transformer architecture and includes the LayerNorm layer (layer normalization layer), the contextual multi-head self-attention mechanism layer (including the multi-head self-attention mechanism MHSA), the Dropout layer, the LayerNorm layer, the multi-layer perceptron and the Dropout layer from front to back, where:

[0063] Deep Featuresb i After being processed by the LayerNorm layer, the query is processed using the multi-head self-attention mechanism MHSA key value Where R represents the real number space, B is H*W, H and W represent the height and width of the feature map entering the multi-head self-attention mechanism respectively; C i and C i+1 They are semantic features l i Number of channels and context features f i The contextual multi-head self-attention mechanism layer in this scheme calculates the self-attention feature along the channel dimension based on the query Q, key K, and value V. The formula is:

[0064]

[0065] Among them, the superscript T represents the transposition operation, and Softmax is the normalized exponential function.

[0066] Self-attention feature F cmsa After the Dropout layer, and the deep feature b i The sum is the enhanced feature b ii , feature b ii After being processed by the LayerNorm layer, the multi-layer perceptron and the Dropout layer, it is combined with feature b again. ii Sum and get the enhanced deep feature b i C . Multiple context channel attention blocks can be set up in cascade.

[0067] 2.3 Establish global dependency modules

[0068] The input of the global dependency module is to enhance the deep feature b i C , the output is the enhanced feature f i C .

[0069] The global dependency module includes a feature splitting unit and a feature enhancement unit, wherein the feature splitting unit is used to enhance the deep feature b i C According to the semantic feature i and context feature f i The channel number relationship is split into features r in the channel dimension l and feature r f ; The feature enhancement unit is used to split the feature r l and feature r f Semantic features l i and context feature f iMultiply them to enhance the target channel features and attenuate similar noise channel features; at the same time, introduce learnable parameters α and β (initial value is 1) to determine the contribution of context features in the result prediction, and finally obtain the filtered enhanced feature f i C , the formula is as follows:

[0070]

[0071] 3. Multi-scale frequency domain decoding module

[0072] like Fig. 9 As shown, the multi-scale frequency domain decoding module is used to enhance the feature f i C Decode and get the output feature o1.

[0073] like Figure 3 As shown in Figure 1, the multi-scale frequency domain decoding module corresponds to the basic feature extraction network, which includes four decoding blocks connected in sequence, each of which uses a MWC-VSS decoder; the input of each decoding block is the output feature o of the decoding block of the next layer (from bottom to top, the bottom layer is the fourth layer) i+1 and the corresponding enhanced features f output by the contextual channel denoising module i C , the output of the decoding block is the output feature o i It should be noted that for the decoding block of the bottom layer, since it has no next layer, the input o is used as the default in the calculation. i+1 is 0; the top-level decoding block finally obtains the output feature o1 and provides it to the semantic segmentation output module.

[0074] 3.1MWC-VSS decoder

[0075] The MWC-VSS decoder is used to decode the output features that combine multiple scales. i+1 The enhanced feature f i C , the output is o i .

[0076] like Fig.10 As shown in the figure, the MWC-VSS decoder in this scheme includes a PatchExpand layer, a LayerNorm layer, a MWC-SS2D module and a DropPath layer connected in sequence, wherein: the PatchExpand layer is used to convert the feature o i+1 Restored to twice the original pixel size, and then combined with the enhanced feature f i C Sum and get the fusion feature g i , which not only saves computing resources, but also integrates comprehensive information from different levels of features; fusion feature gi After being processed by the LayerNorm layer, MWC-SS2D module and DropPath layer, it is combined with the fusion feature g i Sum and get the output feature o of the MWC-VSS decoder i ; The MWC-SS2D module is used to denoise the fused features and establish global dependencies.

[0077] 3.1.1MWC-SS2D module

[0078] like Fig.11 As shown, the MWC-SS2D module in this scheme has two branches. The first branch consists of a linear layer (i.e., a multi-layer perceptron), a multi-scale wavelet convolution attention module MWCA, a SiLU function, a SS2D module, and a LayerNorm layer; the second branch consists of a linear layer, a SiLU function, and a linear layer.

[0079] Fusion feature g i After the LayerNorm layer processing, the feature t is obtained i And enter the MWC-SS2D module, feature t i In the first branch, after passing through the linear layer, it enters the multi-scale wavelet convolution attention module MWCA, in which different wavelet convolution layers are used to perform frequency domain denoising on the features, and after nonlinear transformation by the SiLU function, it is input into the SS2D module to establish global dependency, and then passes through the LayerNorm layer; feature t i In the second branch, after being processed by the linear layer and the SiLU function in turn, it is multiplied and fused with the output of the first branch, and finally processed by a linear layer to complete denoising and establish global dependencies.

[0080] 3.1.1.1 Multi-scale Wavelet Convolution Attention Module MWCA

[0081] like Fig.12 As shown in Figure 2, the multi-scale wavelet convolution attention module MWCA is used to perform frequency domain denoising on the feature augmentation; MWCA includes a convolution attention module and four wavelet convolution layers with different kernel sizes, and the four different wavelet convolution kernel sizes are set to 1×1, 3×3, 5×5 and 7×7 respectively; the feature t in the first branch i After passing through the linear layer, it enters the convolutional attention module (CBAM) of MWCA to enhance the features of its channel and spatial dimensions, and then enters four wavelet convolution layers to capture multi-scale local detail features. Finally, the output features of the module are obtained through the element-by-element addition fusion method to comprehensively enhance and integrate local information.

[0082] 4. Semantic Segmentation Output Module

[0083] The input of the semantic segmentation output module is the output feature o1, and the output is the semantic segmentation result.

[0084] The semantic segmentation output module includes a fully connected layer and a softmax function. First, the output feature o1 is input into the fully connected layer to adjust the feature map size. Then, each pixel of the feature map is classified through the softmax function, and finally the semantic segmentation result is obtained.

[0085] 5. Loss Function

[0086] The loss function of the semantic segmentation network uses the standard cross entropy loss function to measure the difference between the predicted probability distribution and the true probability distribution. The network is trained by minimizing the loss function to make the network prediction closer to the actual situation. The loss function is used to optimize the parameters of the forward-looking sonar image semantic segmentation network based on multi-scale frequency domain and visual state space. The mathematical expression of the cross entropy loss function is as follows:

[0087]

[0088] In the above formula, L represents the cross entropy loss function, N represents the pixel set of the forward-looking sonar image, that is, the product of the image height H and width W; W i Represents weight, which is used to deal with data imbalance problem; T i represents the true category label of pixel i; Represents the probability that the network predicts that pixel i belongs to a certain category.

[0089] 6. Dataset Construction

[0090] Multi-beam forward-looking sonar images are captured using sonar, and all images with a resolution greater than 512×1024 are scaled to 512×1024. Then, the target categories contained in the forward-looking sonar images are manually interpreted and labeled to obtain a forward-looking sonar image dataset with sample labels, which are divided into a training set and a test set according to a 3:1 ratio.

[0091] 7. Network Training

[0092] The constructed semantic segmentation network is trained using the training set and tested using the test set; during training, the network parameters are optimized using the cross entropy loss function, and the trained network model is saved for semantic segmentation of new forward-looking sonar images.

[0093] In one embodiment of the present invention, the semantic segmentation results of the forward-looking sonar image are obtained by using the trained network model. Figure 14(a) , 14(b) and 14(c); it can be seen that the designed scheme can accurately detect the target category in the forward-looking sonar image.

[0094] The present invention can effectively solve the various problems faced by the semantic segmentation of forward-looking sonar images and improve the segmentation accuracy and model performance. This method aims to solve the problem that forward-looking sonar images are affected by environmental noise, similar noise and occlusion shadows, and have low signal-to-noise ratio, low resolution and poor quality, which affect the accuracy of semantic segmentation. It uses multi-scale wavelet convolution to extract frequency domain features to reduce environmental noise and occlusion shadows, and uses CCDM to suppress similar noise interference and enhance target features. It has performed well in multiple data set comparison experiments, providing reliable technical support for applications in related fields.

[0095] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A semantic segmentation method for forward-looking sonar images based on multi-scale frequency domain, characterized in that: include: Construct a semantic segmentation network for forward-looking sonar images based on multi-scale frequency domain denoising, and train the semantic segmentation network for forward-looking sonar images; The forward-looking sonar image is semantically segmented using a trained forward-looking sonar image semantic segmentation network to obtain a semantic segmentation result; the forward-looking sonar image semantic segmentation network includes a basic feature extraction network, a context channel denoising module, a multi-scale frequency domain decoding module and a semantic segmentation output module; wherein the basic feature extraction network is used to perform semantic feature extraction on the input forward-looking sonar image, the context channel denoising module is used to enhance the extracted semantic features and the corresponding context features, the multi-scale frequency domain decoding module is used to decode the enhanced features to obtain output features, and the semantic segmentation output module is used to classify the output features and output the semantic segmentation result.

2. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 1, characterized in that: The basic feature extraction network is used to extract semantic features and context features from the input forward-looking sonar image L; the basic feature extraction network includes four feature layers of different resolutions connected in sequence, each feature layer uses a PVTv2 encoder; after the forward-looking sonar image L is subjected to semantic feature and context extraction by the four feature layers, the feature map size output by the four feature layers from top to bottom is and The feature map output by each feature layer contains semantic features l i , i=1,2,3,4.

3. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 1, characterized in that: Three context channel denoising modules are set up, and the three context channel denoising modules are respectively connected to the first three feature layers of the basic feature extraction network to enhance the semantic features l1, l2, l3 extracted by the first three feature layers and the corresponding context features f1, f2, f3 to obtain the corresponding enhanced feature f1 C , The semantic feature l4 of the fourth feature layer does not pass through the feature enhancement module, and the semantic feature l4 is directly used as the corresponding enhanced feature Among them, the enhanced features As the context feature f3 of the third context channel denoising module, the enhanced feature As the context feature f2 of the third context channel denoising module, the enhanced feature As the context feature f1 of the third context channel denoising module; Each context channel denoising module consists of three parts: feature extraction part, context channel attention block and global dependency building module.

4. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 3, characterized in that: The input feature of the feature extraction part is the semantic feature l output by the corresponding feature layer i and context feature f i , the output feature is b i ; The feature extraction part includes the global maximum pooling layer, the feature dimension concatenation layer and the multi-layer perceptron; among them, the global maximum pooling layer is used for semantic features l i and context feature f i Each channel of is processed and the maximum value is extracted as the feature representation of the channel; the semantic feature l i and context feature f i After the global maximum pooling layer, the feature dimension splicing layer is used to splice according to the channel dimension, and finally the multi-layer perceptron is used to obtain the deep feature b. i .

5. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 3, characterized in that: The context channel attention block includes LayerNorm layer, context multi-head self-attention mechanism layer, Dropout layer, LayerNorm layer, multi-layer perceptron and Dropout layer from front to back. in: Deep Featuresb i After being processed by the LayerNorm layer, the query is processed using the multi-head self-attention mechanism MHSA key value Where R represents the real number space, B is H*W, H and W represent the height and width of the feature map entering the multi-head self-attention mechanism respectively; C i and C i+1 They are semantic features l i Number of channels and context features f i The number of channels; the contextual multi-head self-attention mechanism layer calculates the self-attention feature along the channel dimension based on the query Q, key K, and value V. The formula is: Among them, the superscript T represents the transposition operation, and Softmax is the normalized exponential function; Self-attention feature F cmsa After the Dropout layer, and the deep feature b i The sum is the enhanced feature b ii , feature b ii After being processed by the LayerNorm layer, the multi-layer perceptron and the Dropout layer, it is combined with feature b again. ii Sum and get the enhanced deep feature b i C .

6. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 3, characterized in that: The global dependency module includes a feature splitting unit and a feature enhancement unit, wherein the feature splitting unit is used to enhance the deep feature b i C According to the semantic features i and context feature f i The channel number relationship is split into features r in the channel dimension l and feature r f ; The feature enhancement unit is used to split the feature r l and feature r f Semantic features l i and context feature f i Multiply them to enhance the target channel features and attenuate similar noise channel features; at the same time, the learnable parameters α and β are introduced to determine the contribution of context features in the result prediction, and finally the filtered enhanced features f are obtained. i C , the formula is as follows:

7. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 1, characterized in that: The multi-scale frequency domain decoding module corresponds to the basic feature extraction network and includes four sequentially connected decoding blocks, each of which uses an MWC-VSS decoder; wherein the input of each decoding block is the output feature o of the next layer of decoding blocks i+1 and the corresponding enhanced features f output by the contextual channel denoising module i C , the output of the decoding block is the output feature o i ; For the underlying decoding block, the default input o is used during calculation i+1 is 0; the top-level decoding block finally obtains the output feature o1 and provides it to the semantic segmentation output module.

8. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 1, characterized in that: The MWC-VSS decoder includes a PatchExpand layer, a LayerNorm layer, a MWC-SS2D module and a DropPath layer connected in sequence, wherein: the PatchExpand layer is used to convert the feature o i+1 Restored to twice the original pixel size, and then combined with the enhanced feature f i C Sum and get the fusion feature g i , fusion feature g i After being processed by the LayerNorm layer, MWC-SS2D module and DropPath layer, it is combined with the fusion feature g i Sum and get the output feature o of the MWC-VSS decoder i ; The MWC-SS2D module is used to denoise the fused features and establish global dependencies.

9. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 1, characterized in that: The MWC-SS2D module has two branches, the first branch is a linear layer, a multi-scale wavelet convolution attention module MWCA, a SiLU function, a SS2D module and a LayerNorm layer; the second branch is a linear layer, a SiLU function and a linear layer; Fusion feature g i After the LayerNorm layer processing, the feature t is obtained i And enter the MWC-SS2D module, feature t i In the first branch, after passing through the linear layer, it enters the multi-scale wavelet convolution attention module MWCA, in which different wavelet convolution layers are used to perform frequency domain denoising on the features, and after nonlinear transformation by the SiLU function, it is input into the SS2D module to establish global dependency, and then passes through the LayerNorm layer; feature t i In the second branch, after being processed by the linear layer and the SiLU function in turn, it is multiplied and fused with the output of the first branch, and finally processed by a linear layer to complete denoising and establish global dependencies.

10. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 1, characterized in that: The multi-scale wavelet convolutional attention module MWCA includes a convolutional attention module and four wavelet convolutional layers with different kernel sizes; the feature t in the first branch i After passing through the linear layer, it enters the convolutional attention module of MWCA to enhance its channel and spatial dimensions, and then enters four wavelet convolution layers to capture multi-scale local detail features. Finally, the output features of the module are obtained through the element-by-element addition fusion method.

Images