A multi-scale frequency domain-based forward-looking sonar image semantic segmentation method

Abstract

The application discloses a forward-looking sonar image semantic segmentation method based on a multi-scale frequency domain, which comprises the following steps: constructing a forward-looking sonar image semantic segmentation network based on multi-scale frequency domain noise reduction, and training the forward-looking sonar image semantic segmentation network; performing semantic segmentation on a forward-looking sonar image by using the trained forward-looking sonar image semantic segmentation network to obtain a semantic segmentation result; the forward-looking sonar image semantic segmentation network comprises a basic feature extraction network, a context channel noise reduction module, a multi-scale frequency domain decoding module and a semantic segmentation output module; wherein the basic feature extraction network is used for performing semantic feature extraction on an input forward-looking sonar image, the context channel noise reduction module is used for enhancing the extracted semantic features and corresponding context features, the multi-scale frequency domain decoding module is used for decoding the enhanced features to obtain output features, and the semantic segmentation output module is used for classifying the output features and outputting a semantic segmentation result.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and underwater acoustic engineering, and in particular to a semantic segmentation method for forward-looking sonar images based on a multi-scale frequency domain. Background Art

[0002] Forward-looking sonar (FLS) technology plays a key role in numerous fields, including marine resource exploration, underwater environmental monitoring, marine engineering, military operations, and national defense. Unlike optical detection methods, which are severely affected by water quality, acoustic detection methods exhibit unique advantages in underwater environments. Some current sonar devices can achieve an ocean terrain detection range of 20 to 11,000 meters. Forward-looking sonar is widely adopted due to its high maturity and cost-effectiveness.

[0003] Semantic segmentation of forward-looking sonar images faces numerous challenges. Due to the complex underwater environment, sonar images are severely affected by environmental noise, including equipment noise, marine life noise, reverberation noise, and scattering noise. This results in low image resolution, poor contrast, increased black and white spots, and difficulty extracting effective information. At the same time, occlusion and shadow interference blur or even eliminate target features, making segmentation boundaries unclear and easily confusing shadows with other categories, affecting the accuracy of semantic information. Furthermore, the principle of forward-looking sonar imaging results in similarity noise interference in the image, resulting in the appearance of a large number of similar features. Furthermore, as the resolution of the feature map decreases, the discrimination of features extracted at the same level decreases, further affecting segmentation accuracy.

[0004] In recent years, traditional semantic segmentation methods such as VT-HMRF-FCM, graph-based techniques, gaze-based segmentation methods, and the ilastik toolkit have performed poorly in feature extraction and prediction when processing forward-looking sonar imagery due to image complexity and segmentation accuracy requirements. Deep learning-based methods, including convolutional neural networks (CNNs) such as FCNs, UNet, and Deeplabv3+, and Transformer-based methods such as Vision Transformer, Segmenter, and Segformer, also have limitations when applied to forward-looking sonar imagery. CNNs face the challenge of extracting effective features when processing sonar imagery, while Transformers suffer from poor performance due to issues such as sparse and noisy sonar data, high data volume requirements, high computational demands, and difficulty in obtaining labeled data. Therefore, a new approach is urgently needed to effectively integrate local and global features and enhance the model's performance in high-noise environments. Summary of the Invention

[0005] The purpose of this invention is to provide a semantic segmentation method for forward-looking sonar images based on multi-scale frequency domain to solve a series of problems faced by the semantic segmentation of forward-looking sonar images, to meet the needs of precise analysis in multiple fields, and to achieve more accurate and efficient segmentation effects.

[0006] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0007] A semantic segmentation method for forward-looking sonar images based on multi-scale frequency domain, comprising:

[0008] Construct a semantic segmentation network for forward-looking sonar images based on multi-scale frequency domain denoising and train the network.

[0009] The forward-looking sonar image is semantically segmented using a trained forward-looking sonar image semantic segmentation network to obtain a semantic segmentation result; the forward-looking sonar image semantic segmentation network includes a basic feature extraction network, a context channel denoising module, a multi-scale frequency domain decoding module and a semantic segmentation output module; wherein the basic feature extraction network is used to perform semantic feature extraction on the input forward-looking sonar image, the context channel denoising module is used to enhance the extracted semantic features and the corresponding context features, the multi-scale frequency domain decoding module is used to decode the enhanced features to obtain output features, and the semantic segmentation output module is used to classify the output features and output the semantic segmentation result.

[0010] Furthermore, the basic feature extraction network is used to extract the input forward-looking sonar image. Extract semantic features and context features; the basic feature extraction network includes four feature layers of different resolutions connected in sequence, each of which uses the PVTv2 encoder; forward-looking sonar image After semantic features and context extraction through four feature layers, the feature map size output from the top to the bottom four feature layers is the forward-looking sonar image. of 、 、 and , the feature map output by each feature layer contains semantic features , .

[0011] Furthermore, three context channel denoising modules are set, and the three context channel denoising modules are respectively connected to the first three feature layers of the basic feature extraction network to extract the semantic features of the first three feature layers. and the corresponding context features Enhance and obtain the corresponding enhanced features ; The semantic features of the fourth feature layer Without the feature enhancement module, the semantic features Directly as the corresponding enhanced features Among them, the enhanced features Contextual features as the third context channel denoising module , enhanced features Contextual features as the second context channel denoising module , enhanced features Contextual features as the first contextual channel denoising module ;

[0012] Each context channel denoising module consists of three parts: feature extraction part, context channel attention block and global dependency establishment module.

[0013] Furthermore, the input features of the feature extraction part are the semantic features output by the corresponding feature layer. and contextual features , the output features are ; The feature extraction part includes the global maximum pooling layer, the feature dimension splicing layer and the multi-layer perceptron; Among them, the global maximum pooling layer is used for semantic features and contextual features Each channel is processed and the maximum value is extracted as the feature representation of the channel; semantic features and contextual features After the global maximum pooling layer, the feature dimension splicing layer is used to splice according to the channel dimension, and finally the multi-layer perceptron is used to obtain deep features. .

[0014] Furthermore, the contextual channel attention block includes, from front to back, a LayerNorm layer, a contextual multi-head self-attention mechanism layer, a Dropout layer, a LayerNorm layer, a multi-layer perceptron, and a Dropout layer, where:

[0015] Deep Features After being processed by the LayerNorm layer, the query is processed using the multi-head self-attention mechanism MHSA ,key ,value ,in R represents the real number space, B for H * W , H 、 W Respectively represent the height and width of the feature map entering the multi-head self-attention mechanism; and Semantic features Number of channels and context features The number of channels; the contextual multi-head self-attention mechanism layer is based on the query Q ,key K ,value V , self-attention feature calculation is performed along the channel dimension, the formula is:

[0016]

[0017] Among them, the superscript T represents the transpose operation, ‌Softmax ‌ is the normalized exponential function;

[0018] Self-attention features After the Dropout layer, and the deep features The enhanced features are obtained by summing ,feature After being processed by LayerNorm layer, multi-layer perceptron and Dropout layer, it is combined with feature Sum to get enhanced deep features .

[0019] Furthermore, the global dependency module includes a feature splitting unit and a feature enhancement unit, wherein the feature splitting unit is used to enhance the deep features. According to semantic features and contextual features The channel number relationship is split into features in the channel dimension and features ; The feature enhancement unit is used to split the features and features Semantic features and contextual features Multiply, enhance the target channel characteristics and attenuate similar noise channel characteristics; at the same time, introduce learnable parameters and Determine the contribution of contextual features in the result prediction and finally obtain the filtered enhanced features , the formula is as follows:

[0020] .

[0021] Furthermore, the multi-scale frequency domain decoding module corresponds to the basic feature extraction network and includes four sequentially connected decoding blocks, each of which uses an MWC-VSS decoder; wherein the input of each decoding block is the output feature of the next layer decoding block. And the corresponding enhanced features output by the context channel denoising module , the output of the decoding block is the output feature ; For the decoding block of the bottom layer, the default input is used during calculation is 0; the top-level decoding block finally obtains the output feature Provided to the semantic segmentation output module.

[0022] Furthermore, the MWC-VSS decoder includes a PatchExpand layer, a LayerNorm layer, a MWC-SS2D module and a DropPath layer connected in sequence, wherein: the PatchExpand layer is used to convert the feature Restore to twice the original pixel size and then enhance the features Sum and get the fusion feature , fusion features After being processed by LayerNorm layer, MWC-SS2D module and DropPath layer, it is combined with the fusion feature Sum and get the output characteristics of the MWC-VSS decoder ; The MWC-SS2D module is used to denoise the fused features and establish global dependencies.

[0023] Furthermore, the MWC-SS2D module has two branches. The first branch consists of a linear layer, a multi-scale wavelet convolution attention module MWCA, a SiLU function, an SS2D module, and a LayerNorm layer. The second branch consists of a linear layer, a SiLU function, and a linear layer.

[0024] Fusion Features After the LayerNorm layer processing, the features are obtained And enter the MWC-SS2D module, features In the first branch, after passing through the linear layer, it enters the multi-scale wavelet convolution attention module MWCA, in which different wavelet convolution layers are used to perform frequency domain denoising on the features, and after nonlinear transformation through the SiLU function, it is input into the SS2D module to establish global dependency, and then passes through the LayerNorm layer; In the second branch, after being processed by the linear layer and the SiLU function, it is multiplied and fused with the output of the first branch, and finally processed by a linear layer to complete denoising and establish global dependencies.

[0025] Furthermore, the multi-scale wavelet convolution attention module MWCA includes a convolutional attention module and four wavelet convolution layers with different kernel sizes; the features in the first branch After passing through the linear layer, it enters the convolutional attention module of MWCA to enhance its channel and spatial dimensions. Then it enters four wavelet convolution layers to capture multi-scale local detail features. Finally, the output features of the module are obtained through the element-wise addition fusion method.

[0026] Furthermore, the semantic segmentation output module includes a fully connected layer and a softmax function; the output features obtained by the multi-scale frequency domain decoding module are The input fully connected layer adjusts the size of the feature map, and then classifies each pixel of the feature map through the softmax function to finally obtain the semantic segmentation result.

[0027] Furthermore, the loss function of the semantic segmentation network adopts the standard cross entropy loss function, which is expressed as:

[0028]

[0029] In the above formula, represents the cross entropy loss function, Represents the pixel set of the forward-looking sonar image, that is, the image height H and width W The product of Represents weights, which are used to deal with data imbalance problems; Represents pixels The true category label of Represents the pixel predicted by the network The probability of belonging to a certain category.

[0030] Furthermore, multi-beam forward-looking sonar images are captured and resized using sonar. Then, the target categories contained in the forward-looking sonar images are manually interpreted and labeled to obtain a forward-looking sonar image dataset with sample labels and divide it into a training set and a test set.

[0031] A forward-looking sonar image semantic segmentation device comprises a processor, a memory, and a computer program stored in the memory; the device is characterized in that when the processor executes the computer program, the forward-looking sonar image semantic segmentation method based on the multi-scale frequency domain is implemented.

[0032] A computer-readable storage medium having a computer program stored therein; wherein when the computer program is executed by a processor, the forward-looking sonar image semantic segmentation method based on multi-scale frequency domain is implemented.

[0033] Compared with the prior art, the present invention has the following technical features:

[0034] The network model constructed in the present invention proposes innovative solutions to various interferences faced by the semantic segmentation of forward-looking sonar images, such as occlusion shadows, similarity noise and environmental noise. By combining multi-scale wavelet convolution and visual state space modules, it effectively reduces interference, improves segmentation accuracy, and enhances target feature representation; in terms of feature extraction and fusion, it realizes multi-scale frequency domain feature extraction and global and local feature fusion, and optimizes computing resource utilization; it performs excellently across multiple data sets, has strong adaptability and generalization capabilities in complex environments, and can effectively deal with situations such as variable noise and target feature ambiguity; it reduces computational complexity, balances performance and resource requirements, and controls the number of parameters and memory usage while ensuring segmentation accuracy; it solves the limitations of existing technologies in processing semantic segmentation of forward-looking sonar images, provides reliable technical support for many fields such as marine resource exploration and underwater environment monitoring, promotes technological development in this field, and has important practical application value and social benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the process of the present invention;

[0036] Figure 2 This is the structure diagram of the semantic segmentation network;

[0037] Figure 3 This is the semantic segmentation network model diagram;

[0038] Figure 4 Extract network structure diagram for basic features;

[0039] Figure 5 This is the structural diagram of the context channel denoising module;

[0040] Figure 6 This is the structural diagram of the feature extraction part of the context channel denoising module;

[0041] Figure 7 This is the structure diagram of the context channel attention block of the context channel denoising module;

[0042] Figure 8 Establish a global dependency structure diagram for the context channel denoising module;

[0043] Figure 9 This is the structural diagram of the multi-scale frequency domain decoding module;

[0044] Figure 10 This is the structure diagram of the MWC-VSS decoder of the multi-scale frequency domain decoding module;

[0045] Figure 11 This is the MWC-SS2D module structure diagram of the MWC-VSS decoder;

[0046] Figure 12This is the MWCA module structure diagram of the MWC-SS2D module;

[0047] Figure 13 This is the structure diagram of the semantic segmentation output module;

[0048] Figure 14(a) 、 14(b) 14(c) and 14(c) are the semantic segmentation effect diagrams of the forward-looking sonar image in this embodiment. DETAILED DESCRIPTION

[0049] The present invention provides a semantic segmentation method for forward-looking sonar images based on multi-scale frequency domain, which overcomes the shortcomings of the existing technology, improves segmentation accuracy, enhances model robustness, optimizes computing resource utilization, integrates multimodal information and solves practical application problems. Figure 1 , the method of the present invention comprises the following steps:

[0050] Construct a semantic segmentation network for forward-looking sonar images based on multi-scale frequency domain denoising and train the network.

[0051] Use the trained forward-looking sonar image semantic segmentation network to perform semantic segmentation on the forward-looking sonar image and obtain the semantic segmentation result;

[0052] like Figure 2 and Figure 3 As shown, the forward-looking sonar image semantic segmentation network includes a basic feature extraction network, a context channel denoising module, a multi-scale frequency domain decoding module and a semantic segmentation output module; wherein, the basic feature extraction network is used to perform semantic feature extraction on the input forward-looking sonar image, the context channel denoising module is used to enhance the extracted semantic features and the corresponding context features, the multi-scale frequency domain decoding module is used to decode the enhanced features to obtain output features, and the semantic segmentation output module is used to classify the output features and output the semantic segmentation results.

[0053] 1. Basic feature extraction network

[0054] like Figure 4 As shown, the basic feature extraction network is used to input the forward-looking sonar image Extract semantic features and context features; the basic feature extraction network includes four feature layers of different resolutions connected in sequence, each of which uses the PVTv2 encoder; forward-looking sonar image After semantic features and context extraction through four feature layers, the feature map size output from the top to the bottom four feature layers is the forward-looking sonar image. of 、 、 and , the feature map output by each feature layer contains semantic features , The feature map-based feature extraction network of semantic features captures the local and global semantic features of sonar images layer by layer through hierarchical extraction. Its purpose is to retain detailed features to enhance the semantic understanding ability of low-resolution sonar images.

[0055] 2. Contextual channel denoising module

[0056] like Figure 3 As shown, three context channel denoising modules are set, and the three context channel denoising modules are respectively connected to the first three feature layers of the basic feature extraction network to extract semantic features from the first three feature layers. and the corresponding context features Enhance and obtain the corresponding enhanced features ; The semantic features of the fourth feature layer Without the feature enhancement module, the semantic features Directly as the corresponding enhanced features Participate in subsequent calculations. Among them, the enhanced features Contextual features as the third context channel denoising module , enhanced features Contextual features as the second context channel denoising module , enhanced features Contextual features as the first contextual channel denoising module .

[0057] like Figure 5 As shown in the figure, each context channel denoising module consists of three parts: feature extraction, context channel attention block and global dependency establishment module, which are mainly used to solve the problem of similar noise interference in acoustic images; the context channel attention module is composed of five context channel attention blocks, and the number of heads of each block is 1, 2, 4, 8, and 16 respectively.

[0058] 2.1 Feature extraction

[0059] like Figure 6 As shown, the input features of the feature extraction part are the semantic features output by the corresponding feature layer and contextual features ( i =1,2,3), the output features are ; The feature extraction part includes the global maximum pooling layer, the feature dimension splicing layer and the multi-layer perceptron; Among them, the global maximum pooling layer is used for semantic features and contextual features Each channel is processed and the maximum value is extracted as the feature representation of the channel; semantic features and contextual features After the global maximum pooling layer, the feature dimension splicing layer is used to splice according to the channel dimension, and finally the multi-layer perceptron is used to obtain deep features. ( i =1,2,3).

[0060] 2.2 Contextual Channel Attention Block

[0061] like Figure 7 As shown, the input features of the context channel attention block are deep features , the output features are enhanced deep features .

[0062] The contextual channel attention block is the basic unit of the contextual channel denoising module. It is based on the Channel-Transformer architecture and includes, from front to back, the LayerNorm layer (layer normalization layer), the contextual multi-head self-attention mechanism layer (including the multi-head self-attention mechanism MHSA), the Dropout layer, the LayerNorm layer, the multi-layer perceptron, and the Dropout layer, where:

[0063] Deep Features After being processed by the LayerNorm layer, the query is processed using the multi-head self-attention mechanism MHSA ,key ,value ,in R represents the real number space, B for H * W , H 、 W Respectively represent the height and width of the feature map entering the multi-head self-attention mechanism; and Semantic features Number of channels and context features The contextual multi-head self-attention mechanism layer in this scheme is based on the query Q ,key K ,value V , self-attention feature calculation is performed along the channel dimension, the formula is:

[0064]

[0065] Among them, the superscript T represents the transpose operation, ‌Softmax ‌ is the normalized exponential function.

[0066] Self-attention features After the Dropout layer, and the deep features The enhanced features are obtained by summing ,feature After being processed by LayerNorm layer, multi-layer perceptron and Dropout layer, it is combined with feature Sum to get enhanced deep features . Multiple context channel attention blocks can be set up in cascade.

[0067] 2.3 Establishing global dependency modules

[0068] Establish the input of the global dependency module to enhance deep features , the output is the enhanced feature .

[0069] Establishing a global dependency module includes a feature splitting unit and a feature enhancement unit, wherein the feature splitting unit is used to enhance deep features According to semantic features and contextual features The channel number relationship is split into features in the channel dimension and features ; The feature enhancement unit is used to split the features and features Semantic features and contextual features Multiply, enhance the target channel characteristics and attenuate similar noise channel characteristics; at the same time, introduce learnable parameters and (The initial value is 1) Determine the contribution of context features in the result prediction, and finally obtain the enhanced features after filtering , the formula is as follows:

[0070]

[0071] 3. Multi-scale frequency domain decoding module

[0072] like Figure 9 As shown, the multi-scale frequency domain decoding module is used to enhance the features Decode to get output features .

[0073] like Figure 3 As shown in Figure 1, the multi-scale frequency domain decoding module corresponds to the basic feature extraction network and includes four sequentially connected decoding blocks. Each decoding block uses the MWC-VSS decoder. The input of each decoding block is the output feature of the next layer (from bottom to top, the bottom layer is the fourth layer). And the corresponding enhanced features output by the context channel denoising module , the output of the decoding block is the output feature It should be noted that for the decoding block of the bottom layer, since it has no next layer, the default input is used during calculation. is 0; the top-level decoding block finally obtains the output feature Provided to the semantic segmentation output module.

[0074] 3.1MWC-VSS decoder

[0075] MWC-VSS decoder is used to decode the output features combined with multi-scale Enhanced features after , the output is .

[0076] like Figure 10 As shown in the figure, the MWC-VSS decoder in this scheme includes the PatchExpand layer, LayerNorm layer, MWC-SS2D module and DropPath layer connected in sequence, where the PatchExpand layer is used to convert the feature Restore to twice the original pixel size and then enhance the features Sum and get the fusion feature , which not only saves computing resources but also integrates comprehensive information from different levels of features; fusion features After being processed by LayerNorm layer, MWC-SS2D module and DropPath layer, it is combined with the fusion feature Sum and get the output characteristics of MWC-VSS decoder ; The MWC-SS2D module is used to denoise the fused features and establish global dependencies.

[0077] 3.1.1 MWC-SS2D Module

[0078] like Figure 11 As shown in the figure, the MWC-SS2D module in this scheme has two branches. The first branch consists of a linear layer (i.e., a multi-layer perceptron), a multi-scale wavelet convolution attention module MWCA, a SiLU function, a SS2D module, and a LayerNorm layer; the second branch consists of a linear layer, a SiLU function, and a linear layer.

[0079] Fusion Features After the LayerNorm layer processing, the features are obtained And enter the MWC-SS2D module, features In the first branch, after passing through the linear layer, it enters the multi-scale wavelet convolution attention module MWCA, in which different wavelet convolution layers are used to perform frequency domain denoising on the features, and after nonlinear transformation through the SiLU function, it is input into the SS2D module to establish global dependency, and then passes through the LayerNorm layer; In the second branch, after being processed by the linear layer and the SiLU function, it is multiplied and fused with the output of the first branch, and finally processed by a linear layer to complete denoising and establish global dependencies.

[0080] 3.1.1.1 Multi-scale Wavelet Convolution Attention Module MWCA

[0081] like Figure 12 As shown in the figure, the multi-scale wavelet convolution attention module MWCA is used to perform frequency domain denoising on the feature augmentation; MWCA includes a convolution attention module and four wavelet convolution layers with different kernel sizes, and the four different wavelet convolution kernel sizes are set to 1×1, 3×3, 5×5 and 7×7 respectively; the features in the first branch After passing through the linear layer, it enters the convolutional attention module (CBAM) of MWCA to enhance its channel and spatial dimensions. It then enters four wavelet convolution layers to capture multi-scale local detail features. Finally, the output features of the module are obtained through the element-by-element addition fusion method to comprehensively enhance and integrate local information.

[0082] 4. Semantic Segmentation Output Module

[0083] The input of the semantic segmentation output module is the output feature , the output is the semantic segmentation result.

[0084] The semantic segmentation output module includes a fully connected layer and a softmax function; first, the output features The input fully connected layer adjusts the size of the feature map, and then classifies each pixel of the feature map through the softmax function to finally obtain the semantic segmentation result.

[0085] 5. Loss Function

[0086] The loss function of the semantic segmentation network uses the standard cross-entropy loss function to measure the difference between the predicted probability distribution and the true probability distribution. The network is trained by minimizing this loss function, making the network prediction closer to the actual situation. The loss function is used to optimize the parameters of the forward-looking sonar image semantic segmentation network based on multi-scale frequency domain and visual state space. The mathematical expression of the cross-entropy loss function is as follows:

[0087]

[0088] In the above formula, represents the cross entropy loss function, Represents the pixel set of the forward-looking sonar image, that is, the image height H and width W The product of Represents weights, which are used to deal with data imbalance problems; Represents pixels The true category label of Represents the pixel predicted by the network The probability of belonging to a certain category.

[0089] 6. Dataset Construction

[0090] Multi-beam forward-looking sonar images were captured using sonar. Images with a resolution greater than 512×1024 were scaled to 512×1024. The target categories contained in the forward-looking sonar images were manually interpreted and labeled to obtain a forward-looking sonar image dataset with sample labels. The dataset was then divided into a training set and a test set according to a 3:1 ratio.

[0091] 7. Network training

[0092] The constructed semantic segmentation network is trained using the training set and tested using the test set; during training, the network parameters are optimized using the cross entropy loss function, and the trained network model is saved for semantic segmentation of new forward-looking sonar images.

[0093] In one embodiment of the present invention, the semantic segmentation results of the forward-looking sonar image are obtained using the trained network model. Figure 14(a) 、 14(b) and 14(c); it can be seen that the designed scheme can accurately detect the target category in the forward-looking sonar image.

[0094] This method effectively addresses multiple challenges in semantic segmentation of forward-looking sonar images, improving segmentation accuracy and model performance. This method addresses the problem of low signal-to-noise ratio, low resolution, and poor quality in forward-looking sonar images, which are affected by environmental noise, similar noise, and occlusions. It utilizes multi-scale wavelet convolution to extract frequency-domain features to reduce environmental noise and occlusions, and CCDM to suppress similar noise interference and enhance target features. The method demonstrates excellent performance in comparative experiments across multiple datasets, providing reliable technical support for applications in related fields.

[0095] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A semantic segmentation method for forward-looking sonar images based on multi-scale frequency domain, characterized in that: include: Construct a semantic segmentation network for forward-looking sonar images based on multi-scale frequency domain denoising and train the network. The forward-looking sonar image is semantically segmented using a trained forward-looking sonar image semantic segmentation network to obtain a semantic segmentation result; the forward-looking sonar image semantic segmentation network includes a basic feature extraction network, a context channel denoising module, a multi-scale frequency domain decoding module and a semantic segmentation output module; wherein the basic feature extraction network is used to perform semantic feature extraction on the input forward-looking sonar image, the context channel denoising module is used to enhance the extracted semantic features and the corresponding context features, the multi-scale frequency domain decoding module is used to decode the enhanced features to obtain output features, and the semantic segmentation output module is used to classify the output features and output the semantic segmentation result; The multi-scale frequency domain decoding module corresponds to the basic feature extraction network and includes four sequentially connected decoding blocks, each of which uses an MWC-VSS decoder; wherein the input of each decoding block is the output feature of the next layer decoding block. And the corresponding enhanced features output by the context channel denoising module , the output of the decoding block is the output feature ; For the decoding block of the bottom layer, the default input is used during calculation is 0; the top-level decoding block finally obtains the output feature Provided to the semantic segmentation output module; The MWC-VSS decoder includes a PatchExpand layer, a LayerNorm layer, a MWC-SS2D module and a DropPath layer connected in sequence; the MWC-SS2D module has two branches, the first branch includes a linear layer, a multi-scale wavelet convolution attention module MWCA, a SiLU function, a SS2D module and a LayerNorm layer in sequence; the second branch includes a linear layer, a SiLU function and a linear layer in sequence.

2. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 1, characterized in that: The basic feature extraction network is used to extract the input forward-looking sonar image Extract semantic features and context features; the basic feature extraction network includes four feature layers of different resolutions connected in sequence, each of which uses the PVTv2 encoder; forward-looking sonar image After semantic features and context extraction through four feature layers, the feature map size output from the top to the bottom four feature layers is the forward-looking sonar image. of 、 、 and , the feature map output by each feature layer contains semantic features , .

3. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 1, characterized in that: Three context channel denoising modules are set up, and the three context channel denoising modules are connected to the first three feature layers of the basic feature extraction network respectively to extract the semantic features of the first three feature layers. and the corresponding context features Enhance and obtain the corresponding enhanced features ; The semantic features of the fourth feature layer Without the feature enhancement module, the semantic features Directly as the corresponding enhanced features Among them, the enhanced features Contextual features as the third context channel denoising module , enhanced features Contextual features as the second context channel denoising module , enhanced features Contextual features as the first contextual channel denoising module ; Each context channel denoising module consists of three parts: feature extraction part, context channel attention block and global dependency establishment module.

4. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 3, characterized in that: The input features of the feature extraction part are the semantic features output by the corresponding feature layer and contextual features , the output features are ; The feature extraction part includes the global maximum pooling layer, the feature dimension splicing layer and the multi-layer perceptron; Among them, the global maximum pooling layer is used for semantic features and contextual features Each channel is processed and the maximum value is extracted as the feature representation of the channel; semantic features and contextual features After the global maximum pooling layer, the feature dimension splicing layer is used to splice according to the channel dimension, and finally the multi-layer perceptron is used to obtain deep features. .

5. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 4, characterized in that: The context channel attention block includes LayerNorm layer, context multi-head self-attention mechanism layer, Dropout layer, LayerNorm layer, multi-layer perceptron and Dropout layer from front to back. in: Deep Features After being processed by the LayerNorm layer, the query is processed using the multi-head self-attention mechanism MHSA ,key ,value ,in R represents the real number space, B for H * W , H 、 W Respectively represent the height and width of the feature map entering the multi-head self-attention mechanism; and Semantic features Number of channels and context features The number of channels; the contextual multi-head self-attention mechanism layer is based on the query Q ,key K ,value V , self-attention feature calculation is performed along the channel dimension, the formula is: Among them, the superscript T represents the transpose operation, ‌Softmax ‌ is the normalized exponential function; Self-attention features After the Dropout layer, and the deep features The enhanced features are obtained by summing ,feature After being processed by LayerNorm layer, multi-layer perceptron and Dropout layer, it is combined with feature Sum to get enhanced deep features .

6. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 5, characterized in that: The global dependency module includes a feature splitting unit and a feature enhancement unit, wherein the feature splitting unit is used to enhance the deep features. According to semantic features and contextual features The channel number relationship is split into features in the channel dimension and features ; The feature enhancement unit is used to split the features and features Semantic features and contextual features Multiply, enhance the target channel characteristics and attenuate similar noise channel characteristics; at the same time, introduce learnable parameters and Determine the contribution of contextual features in the result prediction and finally obtain the filtered enhanced features , the formula is as follows: 。 7. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 1, characterized in that: The PatchExpand layer is used to expand features Restore to twice the original pixel size and then enhance the features Sum and get the fusion feature , fusion features After being processed by LayerNorm layer, MWC-SS2D module and DropPath layer, it is combined with the fusion feature Sum and get the output characteristics of the MWC-VSS decoder ; The MWC-SS2D module is used to fusion features Perform denoising and establish global dependencies.

8. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 7, characterized in that: The fusion features After the LayerNorm layer processing, the features are obtained And enter the MWC-SS2D module, features In the first branch, after passing through the linear layer, it enters the multi-scale wavelet convolution attention module MWCA, in which different wavelet convolution layers are used to perform frequency domain denoising on the features, and after nonlinear transformation through the SiLU function, it is input into the SS2D module to establish global dependency, and then passes through the LayerNorm layer; In the second branch, after being processed by the linear layer and the SiLU function, it is multiplied and fused with the output of the first branch, and finally processed by a linear layer to complete denoising and establish global dependencies.

9. The method for semantic segmentation of forward-looking sonar images based on multi-scale frequency domain according to claim 1, characterized in that: The multi-scale wavelet convolution attention module MWCA includes a convolution attention module and four wavelet convolution layers with different kernel sizes; the features in the first branch After passing through the linear layer, it enters the convolutional attention module of MWCA to enhance its channel and spatial dimensions. Then it enters four wavelet convolution layers to capture multi-scale local detail features. Finally, the output features of the module are obtained through the element-wise addition fusion method.

Images