An edge enhancement and spatial and frequency domain feature fusion multi-modal semantic segmentation method

Abstract

The present application relates to the technical field of computer vision, and especially relates to a multi-modal semantic segmentation method of edge enhancement and space-frequency domain feature fusion, which comprises two modal multi-scale edge enhancement and feature fusion, feature extraction, enhancement, inhibition and interaction of different frequency bands of two modal tensors, and fusion and interaction of space and frequency domain features. The present application extracts edges through Laplace operator, dynamically adjusts the contribution of different modalities and scales through multi-scale edge enhancement and different scale adaptive weight fusion and different modal edge feature fusion, and realizes edge feature emphasis. In view of the problem that the segmentation accuracy is limited due to the ignored frequency domain features, the present application adopts a two-step fusion method of space and frequency domains, adds frequency domain for fusion in the calculation of the space attention weight matrix, and finally fuses the space and frequency attention matrices, so as to realize the full fusion of space and frequency domain features and the cross-complementation of features between different modalities.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a multi-modal semantic segmentation method of edge enhancement and spatial and frequency domain feature fusion. BACKGROUND

[0002] In the field of computer vision, semantic segmentation, as a core task, aims to assign each pixel in an image to a corresponding semantic class through an algorithm. Currently, semantic segmentation technology based on a single visible light RGB modality has been widely researched and applied, but it has significant limitations in real-world scenarios. These include: 1. Sensitivity to lighting conditions. For example, in the context of autonomous driving, especially at night, visible light cameras are easily affected by strong light from oncoming vehicles, resulting in loss of image information; in low-light scenarios such as night scenes and mine scenes, target details are lost, outlines are blurred, and class differentiation is reduced. 2. Lack of depth, geometric information, and other dimensions of information. Only two-dimensional color and texture information can be provided, which cannot directly reflect the three-dimensional spatial position, distance, or scale relationship of the target. 3. When the target and background are highly similar in color and texture, the single RGB modality has difficulty distinguishing semantic classes based on color features. For example, in remote sensing images, it is difficult to distinguish bare land from uncultivated farmland; in military scenarios, the RGB features of camouflaged targets and backgrounds are similar, leading to segmentation failure.

[0003] Chinese Patent Publication No. CN119131375A discloses an image semantic segmentation method based on cross-dimension multi-spectral edge fusion. It takes RGB images and thermal infrared images as input, considers various attributes and interactions between low-level and high-level features in dynamic traffic scenarios, and solves problems such as information loss by fusing semantic and edge features, achieving accurate and robust semantic segmentation of RGB+thermal infrared images under complex lighting conditions. As can be seen, RGB+X multi-modal semantic segmentation technology has significant advantages over single RGB modality by fusing information from RGB modality and other modalities, where X can represent infrared images, SAR imaging, point clouds, depth maps, etc. However, current common deep learning multi-modal feature fusion semantic segmentation models are usually limited to a specific X modality, and generally have the shortcomings of lacking generalization ability for different X modalities, lacking support for in-depth interaction between modalities, lacking emphasis on key edge features, and focusing only on spatial information while ignoring frequency domain information, resulting in limited improvement compared to single modality semantic segmentation networks. SUMMARY

[0004] To this end, the application provides a multi-modal semantic segmentation method of edge enhancement and spatial and frequency domain feature fusion to overcome the problem that the design of the deep learning multi-modal feature fusion semantic segmentation model in the prior art is only limited to a certain specific mode, the generalization ability for different modes is poor, and the improvement compared with the single-modal semantic segmentation network is limited.

[0005] To achieve the above-mentioned purpose, the application provides a multi-modal semantic segmentation method of edge enhancement and spatial and frequency domain feature fusion, comprising,

[0006] The images of the two modes are respectively input into the backbone network with parallel double stems, and a plurality of stages are arranged in the parallel double stems of the backbone network, and any stage can perform feature tensor extraction on the input of the stage;

[0007] The feature tensors of the two modes output by each stage all enter the MEEFIM module arranged corresponding to the stage to perform multi-scale edge enhancement and feature fusion;

[0008] The feature tensors of multi-scale edge enhancement and feature fusion completed by each MEEFIM module are transmitted to the FSAM module arranged corresponding to each stage, and the FSAM module calculates the spatial cross attention matrix and the frequency domain branch attention matrix according to the feature tensors of the two modes to perform fusion and outputs the fusion tensor;

[0009] The output results of the FSAM modules of each stage are summarized by the decoder, and the semantic segmentation result is output;

[0010] The multi-scale edge enhancement of the MEEFIM module includes performing Laplacian operator edge feature extraction on the input feature tensor to obtain an edge feature tensor, and performing multi-scale edge feature enhancement on the original input feature tensor and the edge feature tensor.

[0011] Further, the feature tensors of multi-scale edge enhancement and feature fusion of the two modes output by the MEEFIM module are transmitted to the corresponding FSAM module, and also enter the next stage of the respective stem.

[0012] Further, the MEEFIM module performs Laplacian operator edge feature extraction on the feature tensors input by the two modes respectively to obtain the edge feature tensors E of the two modes respectively, and for each mode, the original input tensor x and the edge feature tensor E are subjected to multi-scale edge feature enhancement, and the scale is set as s, s=1, 2, 4, then,

[0013] ;

[0014] ;

[0015] In the formula, is up-sampling, For two-dimensional average pooling, H is the height of the feature map, W is the width of the feature map, x s is the processed input tensor, E s is the processed edge feature tensor;

[0016] For the scale that completes down-sampling and up-sampling, a convolution operation is performed to obtain a fractional tensor As, and then edge guided feature enhancement is performed to obtain an edge enhanced tensor Fs of the scale, and the edge enhanced tensors Fs of each scale form an edge enhanced tensor list, and the edge enhanced tensor lists of two modalities are obtained.

[0017] Further, the edge feature tensors of the two modalities obtained by the Laplacian operator are weighted and added to obtain the fusion edge feature tensors E combined of the two modalities after fusion, and then convolution operation channel transformation and softmax are performed to obtain the result tensor W s of the weight score of each scale.

[0018] Further, the multi-scale edge enhanced tensor list multi-scale-features of the two modalities and the weight score tensor W s are weighted and summed for each scale to obtain two multi-scale weighted sum edge enhanced tensors A 1attention and A 2attention , and the tensors A 1attention and A 2attention of the two modalities are spliced and convolved to obtain a multi-scale edge feature fusion result tensor A edge , which has the same shape as the input tensor.

[0019] Further, the outputs of the two modalities of the MEEFIM module are,

[0020] ;

[0021] ;

[0022] In the formula, , and , respectively represent two pairs of parameters with a sum of 1, which can be learned, and respectively represent the outputs of the two modalities through the MEEFIM module, and respectively represent the original inputs of the two modalities in the MEEFIM module.

[0023] Further, the FSAM module obtains three self-attention matrices Q spatial , K spatial , V spatial on the spatial domain through a linear layer from the feature tensors of the two modalities output by the MEEFIM module

[0024] In the FSAM module, the two modalities share one frequency domain branch. The two modality tensors are first transformed into tensors in the frequency domain through fast Fourier transform , and then pass through three threshold learnable low, medium and high frequency filters to obtain low, medium and high frequency components in the frequency domain , , , and then inverse fast Fourier transform is performed to obtain low, medium and high frequency components in the spatial domain , , , which represent the global structure, local structure and detail texture respectively.

[0025] Further, the difference matrix enhances or suppresses the feature difference Δ = |x1-x2| of the two modality feature tensors input by the FSAM module, and then performs light multi-scale pooling convolution, L2 normalization, ReLu activation, flattening, min-max scaling, and finally restores the dimension to obtain the difference matrix D. The difference matrix D is applied with a difference weight according to the frequency band, and the enhancement or suppression operation is performed to obtain

[0026] ;

[0027] Finally, one frequency band mixes the information of the two modalities, and outputs three tensors through channel dimension splicing and SE channel attention,

[0028] ;

[0029] In the formula, and are the enhanced or suppressed frequency band components of the two modalities, respectively, is a tensor splicing operation, is a classic channel attention mechanism.

[0030] Further, the FSAM module fuses the features in the frequency domain and the spatial domain through two consecutive operations. The attention weight matrix is composed of the medium frequency and the high frequency in the spatial domain and the frequency domain. The frequency domain part is fused with the medium frequency component output by the SE channel attention as the query vector, and the high frequency component is supplemented as the key vector to supplement its information, and the spatial and frequency domain fusion attention score matrix is calculated.

[0031] Again through cross attention, each modality is multiplied by the value vector of the other modality to obtain the spatial cross attention matrix SFFA of the fused frequency domain feature,

[0032] ;

[0033] The frequency domain branch takes the low-frequency component as the query vector Q, the high-frequency component as the key vector K, and the medium-frequency component as the value vector V to perform attention calculation, and obtains the frequency domain branch attention matrix FDA,

[0034] The spatial cross attention matrix SFFA and the frequency domain branch attention matrix FDA are fused through the lightweight DCG in the FSAM module to obtain two tensors and ,

[0035] ;

[0036] wherein DCG is the input of the two modalities after global average pooling and full connection layer, is the spatial cross attention matrix of the input DCG, is the frequency domain branch attention matrix of the input DCG;

[0037] The two tensors and are obtained by splicing and softmax function to obtain two weights participating in multi-modal hybrid operation and ,

[0038] .

[0039] Further, the DCG output of each modality in the FSAM is further subjected to convolution fusion operation to output a multi-modal feature fusion, frequency domain and spatial domain feature fusion tensor ,

[0040] ;

[0041] The FSAM module of each stage outputs such a tensor, and the decoder collects the four output tensors to output the semantic segmentation prediction result.

[0042] Compared with the prior art, the beneficial effects of the present application are that by adopting the parallel double-trunk trunk network, the image of two modalities can enter and extract feature tensors respectively, and multiple stages are arranged in the trunk network to insert the MEEFIM module for multi-level fusion processing operation, thereby improving the feature fusion effect, and the method can perfectly fuse and predict semantic segmentation for various different modal images, improves the generalization ability of semantic segmentation prediction of different modalities, enables in-depth interaction between modalities, and finally improves the accuracy of multi-modal semantic segmentation prediction.

[0043] Further, the advantages of the MEEFIM module are that the parameter-free edge detection uses the Laplace operator with fixed parameters for edge extraction, which reduces the parameter amount and calculation amount while ensuring the effect; multi-scale edge enhancement performs edge-guided feature enhancement at different scales; and adaptive weight fusion dynamically adjusts the contribution of different modalities and scales through learnable parameters.

[0044] Further, after the feature tensors of the two modalities pass through the MEEFIM module, they enter the FSAM module for feature extraction and fusion in the spatial and frequency domains; the innovation and advantage of the FSAM module are that the frequency domain features are effectively extracted and fully fused, the FSAM module performs fast Fourier filtering on the input feature tensors of the two modalities, dynamically adjusts the filter threshold and uses difference tensors to perform difference enhancement of low and high frequencies and difference suppression of medium frequencies, and outputs the low, medium and high frequency feature tensors of the feature fusion of the two modalities; the spatial domain features are fully fused through the cross-attention mechanism, the two modalities realize cross-attention calculation through the exchange of value matrices, and the feature fusion is fully performed; the fusion of the spatial and frequency domains is completed through the two steps, the frequency domain information is added when calculating the spatial attention weight, and finally the spatial attention tensor and the frequency attention tensor are fused through the lightweight fusion layer to fully fuse the spatial and frequency domain features.

[0045] It is of great significance to improve the generalization ability, precision and accuracy of semantic segmentation, especially multi-modal semantic segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The flowchart of the multi-modal semantic segmentation method of edge enhancement and spatial and frequency domain feature fusion of the present embodiment;

[0047] Figure 2 The overall architecture diagram of the multi-modal semantic segmentation method of edge enhancement and spatial and frequency domain feature fusion of the present embodiment;

[0048] Figure 3 The architecture diagram of the MEEFIM module of the present embodiment;

[0049] Figure 4Architecture diagram of the FSAM module of the present embodiment;

[0050] Figure 5 Prediction comparison diagram on the PIE-RGB-SAR dataset of the present embodiment;

[0051] Figure 6 Prediction comparison diagram on the MSRS dataset of the present embodiment;

[0052] Figure 7 Prediction comparison diagram on the PST900 dataset of the present embodiment. DETAILED DESCRIPTION

[0053] In order to make the objects and advantages of the present application clearer, the present application will be further described below in conjunction with embodiments. It should be understood that the specific embodiments described herein merely serve to explain the present application and should not be used to limit the present application.

[0054] The preferred embodiments of the present application will be described below with reference to the accompanying drawings. It should be understood by those skilled in the art that the embodiments merely serve to explain the technical principles of the present application and are not used to limit the protection scope of the present application.

[0055] It should be noted that, in the description of the present application, the terms "upper", "lower", "left", "right", "inner", "outer" and the like indicating the direction or positional relationship are based on the direction or positional relationship shown in the drawings, which is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0056] In addition, it should also be noted that, in the description of the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be the communication inside two elements. Those skilled in the art can understand the specific meaning of the above terms in the present application according to the specific circumstances.

[0057] Please refer to Figure 1 and Figure 2As shown, the embodiment proposes a multi-modal semantic segmentation method of edge enhancement and spatial-frequency feature fusion, namely EEFSFNet, Edge-Enhanced Frequency-Spatial Fused Net, which adopts a double-parallel four-stage backbone network to process the information of two modalities respectively, realizes the RGB+X multi-modal semantic segmentation technology, and has a significant advantage compared with the single RGB modality by fusing the information of the RGB modality and other modalities, where X can represent infrared images, SAR imaging, point clouds, depth maps, etc. The backbone adopts a UniFormer backbone network, which has the characteristics of lightweight and fast, and is also compatible with mainstream attention backbone networks such as SwinTransFormer, SegFormer, and mainstream convolutional neural networks such as ConvNeXt.

[0058] After each stage of the backbone ends, the output feature tensors of the two modalities jointly enter a multi-scale edge-enhanced feature interaction module, namely MEEFIM module, Multi-Scale Edge-Enhanced Feature Interaction Module, for multi-scale edge enhancement and feature interaction.

[0059] After the two modalities of the feature tensor pass through the MEEFIM module, they enter a frequency-spatial adaptive fusion module, namely FSAM module, Frequency-Spatial adaptive Moudule, for feature extraction and fusion in the spatial and frequency domains.

[0060] Specifically, the multi-modal semantic segmentation method of edge enhancement and spatial-frequency feature fusion of the embodiment includes,

[0061] Step S100, input the two modalities of the input image into the backbone network with a parallel double backbone of UniFormer, and extract the information of the two modalities of the input image;

[0062] Step S200, the backbone network is set with several stages, and the output feature tensors of the two modalities in different stages enter the corresponding MEEFIM module set in the stage for multi-scale edge enhancement and feature fusion, and the output of the MEEFIM module is transmitted to the FSAM module set in the corresponding stage, also as the input of the next stage of the backbone network;

[0063] Step S300, the FSAM module performs feature extraction, enhancement, suppression and interaction of different frequency bands on the input two modalities of the tensor, completes the fusion and interaction of the spatial and frequency domain features through two steps, and outputs a fusion tensor to the decoder, and the decoder finally outputs the semantic segmentation result through the FSAM module results of each stage.

[0064] In step S200, the feature tensors of the two modalities pass through one stage of the respective backbone and then enter the MEEFIM module for feature fusion. In the module, multi-scale edge enhancement and edge feature fusion interaction are performed, and finally the edge-enhanced and feature-interacted feature tensors of the two modalities are output. The two output results enter the next stage of the respective backbone and also enter the FSAM module of the stage.

[0065] Please continue to refer to Figure 3 As shown in FIG. 6, which is an architecture diagram of the MEEFIM module of the embodiment. Laplacian operator edge feature extraction is performed on the feature tensors input by the two modalities respectively to obtain the edge feature tensors of the two modalities respectively. For each modality, multi-scale edge feature enhancement is performed on the original input tensor x and the edge feature tensor E , and the scale is set as s, s = 1, 2, 4, then

[0066] ;

[0067] ;

[0068] In the formula, is up-sampling, is two-dimensional average pooling, H is the height of the feature map (Height), and W is the width of the feature map (Weight), is the processed input tensor, is the processed edge feature tensor.

[0069] Convolution operation is performed on the tensors after down-sampling and up-sampling to obtain a score tensor As,

[0070] ;

[0071] In the formula, is a series of convolution and activation operations, including a two-dimensional convolution with a convolution kernel size of 1, a ReLu activation function, another two-dimensional convolution with a convolution kernel size of 1, and a sigmoid function;

[0072] Then edge-guided feature enhancement is performed to obtain the edge-enhanced tensor F s of the scale.

[0073] ;

[0074] In the formula, represents the tensor is a four-dimensional tensor with a shape of B, C, H, and W;

[0075] The three scales finally compose a list of edge enhanced tensors, and both modalities output a list of edge enhanced tensors after the multi-scale edge enhancement step, multi-scale-features,

[0076] ;

[0077] which is denoted as is a list of five-dimensional tensors with shape 3, B, C, H, W.

[0078] Specifically, the edge feature tensors of the two modalities obtained by the Laplacian operator are weighted and added to obtain

[0079] ;

[0080] wherein E1 and E2 are the edge feature tensors of the two modalities after down-sampling and up-sampling, and α1 and α2 are two learnable parameters that add up to 1, and E combined is the fusion edge feature tensor after the fusion of the two modalities.

[0081] Then, a convolution operation, a channel transformation and a softmax function calculation are performed to obtain a result tensor W s ,

[0082] ;

[0083] When the number of scales is three , i.e. 3, is a series of operations, including global average pooling, a convolution operation with a convolution kernel of 1, a ReLu activation function, a channel-adjusting convolution operation with a convolution kernel of 1, adjusting the channel to ;

[0084] Specifically, the multi-scale edge enhanced tensor list multi-scale-features of the two modalities and the weight score tensor W s are weighted and summed to obtain two multi-scale weighted sum edge enhanced tensors A 1attention and A 2attention , which have the same shape as the input tensor, and the tensors of the two modalities are further spliced and convolved to obtain a multi-scale edge feature fusion result A edge , which has the same shape as the input tensor.

[0085] The final output of the two modalities of the MEEFIM module is A 1attention and A 2attention multiplied by the original input of each modality, and then added to A edgeThe original input of each mode is multiplied by the original input of each mode, and the original input of each mode is connected by jumping, to obtain the final output of each mode,

[0086] ;

[0087] ;

[0088] In the formula, and , respectively represent two pairs of parameters with sum of 1 and learnable, , respectively represent the outputs of two modes through the MEEFIM layer.

[0089] Specifically, the two modal outputs of the MEEFIM module enter the FSAM module for extraction and interaction of frequency domain features and enhancement and fusion of spatial domain features, and the fusion and interaction of spatial domain and frequency domain features are completed in two steps, each of the two modes has a spatial domain branch, and the input tensor obtains three self-attention matrices Q spatial , K spatial , V spatial on the spatial domain through a linear layer.

[0090] ;

[0091] ;

[0092] ;

[0093] In the formula, is the dimension number of each head, The code is implemented by the linear layer function of pytorch, is a single-modal input tensor adjusted to a sequence shape, i.e., a (B, H W, C) shape, the reshape operation reshapes the tensor to ( ), and the permute operation changes the dimension order of the tensor, finally obtaining the query matrix Q spatial , the key matrix K spatial , and the value matrix V spatial on the spatial domain.

[0094] Please continue to refer to Figure 4 , which is an architecture diagram of the FSAM module of the embodiment, the FSAM frequency-space adaptive fusion module shares a frequency domain branch for two modes, and the two modal tensors first pass through a fast Fourier transform to obtain the tensors in the frequency domain,

[0095] ;

[0096] In the formula, x is the single-modal tensor input to the FSAM layer, with shape (B,C,H,W), and FFT2 represents the two-dimensional Fast Fourier Transform. This means moving the low-frequency components of the output spectrum to the center, while distributing the high-frequency components around the perimeter. The specific operations utilize functions encapsulated in the PyTorch library. The output spectrum is shown below. .

[0097] By using three threshold-learnable low, mid, and high-frequency filters, the frequency components of the low, mid, and high frequencies can be obtained.

[0098] ;

[0099] ;

[0100] ;

[0101] In the formula, , These are the low, medium, and high frequency filters mentioned above. , , These are the low, mid, and high frequency components output after filtering, respectively.

[0102] Then, an inverse fast Fourier transform is performed to obtain the low, mid, and high frequency components in the spatial domain, which respectively represent the global structure, local structure, and detailed texture.

[0103] ;

[0104] ;

[0105] ;

[0106] In the formula, Compared to the previous Conversely, restoring the frequency domain tensor with "low frequencies in the center" to the original frequency domain distribution with "low frequencies in the corners" is possible. This is the inverse two-dimensional fast Fourier transform, and the specific implementation here also uses functions encapsulated in the PyTorch library.

[0107] To enhance or suppress features using the difference matrix, first calculate the difference between two features Δ = |x1 - x2|, where x1 and x2 are tensors of the two modalities input to the FSAM layer. This difference is then processed through lightweight multi-scale pooling convolution, followed by L2 normalization, ReLU activation, flattening, and min-max scaling. Finally, the dimensionality is restored to obtain the difference matrix D.

[0108] ;

[0109] In the formula, is a multi-scale pooling operation, is a two-dimensional standardization, is an activation function, and Scale is a max-min value standardization, and the specific operation is is first performed Multi-scale pooling, that is, the pooling operation with a pooling kernel of 3 and the global pooling operation, concatenates the pooling results and changes the channels through the convolution operation to obtain the output result, and then performs the max-min value standardization through ReLu activation function, and the formula is

[0110] ;

[0111] In the formula, diff is the output of the activation function, diff_min is the minimum value of diff in each channel, and diff_max is the maximum value.

[0112] Differential weights are applied according to the frequency band, and enhancement or suppression operations are performed,

[0113] ;

[0114] Finally, one frequency band mixes the information of two modalities through channel dimension concatenation and SE channel attention output,

[0115] ;

[0116] In the formula, and are the enhanced or suppressed frequency band components of the RGB modality and the X modality respectively, is a tensor concatenation operation, and the used is the tensor concatenation function in the library, The layer is a classic channel attention mechanism, that is, the two modalities of the concatenated tensor are first adjusted to have a height and a width of 1 through global average pooling, and then the shape is adjusted to pass through two fully connected layers, the channel weights of the two modalities are adjusted jointly, and then the channel-adjusted concatenated tensor is obtained by multiplying the input of the channel attention layer, is the frequency band component of the multi-modal mixing and channel attention enhancement; a total of three tensors are output, which are the mixed two modalities, enhanced or suppressed low, medium and high frequency band tensors, and can be respectively represented as , , .

[0117] Specifically, the FSAM module fuses the features in the frequency domain and the spatial domain through two consecutive operations, and the attention weight matrix is composed of the medium frequency and the high frequency in the spatial domain and the frequency domain. The frequency domain part is the fused medium frequency component output by the SE channel attention As the query vector, the high-frequency component is fused As the key vector to supplement its information, the final result formula is,

[0118] ;

[0119] In the formula, is the number of input channels divided by the number of multi-head, and the calculated The spatial-frequency fusion attention score matrix.

[0120] Then through cross attention, each modality is multiplied by the value vector of the other modality to obtain the spatial cross attention matrix SFFA of the fused frequency domain features,

[0121] ;

[0122] In the formula, is the value vector of the other modality.

[0123] The frequency domain branch finally takes the low-frequency component as the query vector Q, the high-frequency component as the key vector K, and the medium-frequency component as the value vector V to perform attention calculation, and obtains the frequency domain branch attention matrix FDA,

[0124] ;

[0125] In the formula, Conv is an adjustment convolution with a convolution kernel of 1, and C is the number of channels of the frequency band tensor.

[0126] Subsequently, the SFFA matrix and the attention matrix FDA of the frequency domain branch are fused through the lightweight DCG in the FSAM module, which is the second fusion between the two modalities between the spatial domain and the frequency domain; wherein, DCG is the input of two modalities through global average pooling and fully connected layer, to obtain two tensors and The specific operation is as follows formula,

[0127] ;

[0128] In the formula, is the full connection layer operation, and the method is the method packaged by the pytorch library.

[0129] The two tensors and are spliced and the two weights participating in the multi-modal hybrid operation are obtained through the softmax function

[0130] ;

[0131] ;

[0132] The output is the multiplication of the above two weights and the input of the corresponding modal, and then the weighted addition is added by the spatial input jump connection,

[0133]

[0134] wherein, is a learnable parameter;

[0135] The DCG output of the two modalities is subjected to a convolution fusion operation, outputting a multi-modal feature fusion, frequency domain and spatial domain feature fusion tensor, and each stage of the FSAM module outputs such a tensor, and the decoder collects the four output tensors to finally output the semantic segmentation prediction result,

[0136]

[0137] wherein, , k = 1, 2, 3, 4, i.e. the output of the FSAM module of each of the four stages of the backbone network, the UperNet decoder semantic segmentation head adopts the decoder semantic segmentation head originally used by the UniFormer model without modification, and the result is the semantic segmentation prediction result.

[0138] Based on the above method, test experiments are carried out, and the experimental environment configuration is shown in Table 1. The hyperparameter configuration is slightly different according to different data sets. The optimizer is selected as the momentum stochastic gradient descent method, and the initial learning rate is usually set to le -4 ; the learning rate is dynamically adjusted by the cosine annealing strategy, which decays along the cosine curve from the initial value, and the learning rate decay value is 0.001 at the end of training; the number of image batches used for each batch training is selected from 2 to 8 according to the image size of the data set; the number of training rounds is uniformly 300 rounds.

[0139] Table 1: Experimental environment configuration table.

[0140]

[0141] This embodiment carries out experiments on three different scene multi-modal data sets in total, the scenes involve satellite remote sensing, autonomous driving and simulated mine disaster scenes, and two X modalities are involved, including SAR synthetic aperture radar imaging and thermal imager images.

[0142] The backbone used in the experiment of the EEFSFNet of this embodiment is UniFormer, and the small pre-training weight upernet_global_small.pth is used. The models CMX, ASANet and EGFNet are used as comparison,

[0143] ​​Wherein the backbone adopted by CMX is SwinTransFormer, and the pre-training weight used is:

[0144] swin small patch4 window7 224 22k.pth,

[0145] The backbone adopted by ASANet is ConvNextV2, and the pre-training weight is:

[0146] convnext-v2-tiny_fcmae-in21k-pre_3rdparty_in1k-384px_20230104-d8579f84.pth.

[0147] The backbone adopted by EGFNet is ResNet, and the pre-training weight used is:

[0148] resnet152-b121ed2d.pth.

[0149] The experimental results are shown in Tables 2, 3 and 4,

[0150] Table 2: Experimental results table on PIE-RGB-SAR dataset.

[0151]

[0152] Table 3: Experimental results table on MSRS dataset.

[0153]

[0154] Table 4: Experimental results table on PST900 dataset.

[0155]

[0156] Please refer to Figures 5 to 7It can be seen from Tables 2, 3 and 4 that the average intersection over union mIOU of the semantic segmentation core index of the embodiment is better than that of the current mainstream general multi-modal semantic segmentation model CMX on the three data sets. As can be seen from Table 2, the performance on the remote sensing RGB-SAR multi-modal data set PIE-RGB-SAR is better than that of the semantic segmentation model ASANet specially proposed for RGB-SAR multi-modal. As can be seen from Tables 3 and 4, the performance on the RGB-Thermal multi-modal data set MSRS and PST900 is better than that of the model EGFNet specially proposed for RGB-Thermal. And as can be seen from the experimental results in Tables 2 and 4, for the road class or hand drill class of the data set, which is a subtle or small target, compared with other models that do not emphasize edge features, the application greatly improves the segmentation effect by emphasizing edge details.

[0157] The EEFSFNet provided by the embodiment aims to improve the multi-modal semantic segmentation generalization ability of a single model for RGB modalities and different X modalities, and to enhance the feature emphasis and extraction of edge features and frequency domain features of the model to improve the segmentation precision and accuracy. In view of the problem that some important edge features are not emphasized, the method uses a Laplacian operator to extract edges, and dynamically adjusts the contributions of different modalities and scales through multi-scale edge enhancement, different scale adaptive weight fusion and different modality edge feature fusion, to realize edge feature emphasis. In view of the problem that the segmentation precision is limited due to the neglect of frequency domain features, the method uses a space-frequency two-step fusion method to add frequency domain fusion in the calculation of the space attention weight matrix and finally fuse the space and frequency attention matrices, to realize the full intermingling of space and frequency features and the cross-complementation of features between different modalities. Experiments prove that the application has excellent performance on multi-modal semantic segmentation data sets of RGB and different X modalities, and the algorithm proposed by the application improves the average intersection over union by 1% to 6% compared with mainstream multi-modal models, significantly improving the precision and accuracy and generalization ability of the model for multi-modal semantic segmentation.

[0158] The above experiments can show that the model proposed by the application is a multi-modal semantic segmentation model with strong generalization ability that can be applied to various X modalities and various scenes.

[0159] So far, the technical solutions of the application have been described in combination with the preferred embodiments shown in the drawings, but those skilled in the art can easily understand that the protection scope of the application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to related technical features without departing from the principles of the application, and the technical solutions after the changes or replacements will all fall within the protection scope of the application.

[0160] The above merely illustrates the preferred embodiments of the present application, and is not used to limit the present application; for those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multimodal semantic segmentation method that fuses edge enhancement and spatial-frequency features, characterized in that, include, The images of the two modalities are respectively input into a parallel dual-backbone backbone network. Several stages are set in the parallel dual-backbone backbone network, and feature tensors can be extracted from the input of any stage. The feature tensors of the two modalities output from each stage are fed into the MEEFIM module set for that stage for multi-scale edge enhancement and feature fusion. The feature tensors of the MEEFIM modules that complete multi-scale edge enhancement and feature fusion are passed to the FSAM modules set for each stage. The FSAM modules calculate the spatial cross attention matrix and the frequency branch attention matrix based on the two modal feature tensors and fuse them to output the fused tensor. The decoder summarizes the outputs of the FSAM modules at each stage and outputs the semantic segmentation result. The multi-scale edge enhancement of the MEEFIM module includes extracting edge features from the input feature tensor using the Laplacian operator to obtain an edge feature tensor, and then performing multi-scale edge feature enhancement using the original input feature tensor and the edge feature tensor.

2. The multimodal semantic segmentation method based on edge enhancement and spatial-frequency feature fusion according to claim 1, characterized in that, The feature tensors of the two modalities output by the MEEFIM module, which complete multi-scale edge enhancement and feature fusion, are passed to the corresponding FSAM modules and then enter the next stage of their respective trunks.

3. The multimodal semantic segmentation method based on edge enhancement and spatial-frequency feature fusion according to claim 2, characterized in that, The MEEFIM module performs Laplacian edge feature extraction on the feature tensors input to each of the two modalities, obtaining edge feature tensors E for each modality. For each modality, its original input tensor x and edge feature tensor E are enhanced with multi-scale edge features, with the scale set to s, where s = 1, 2, 4. Then, we have... ； ； In the formula, For upsampling, This is a two-dimensional average pooling method, where H is the height of the feature map, W is the width of the feature map, and x... s It is the processed input tensor, E s The processed edge feature tensor; For each scale that has completed downsampling and upsampling, a convolution operation is performed to obtain a fractional tensor As. Then, edge-guided feature enhancement is performed to obtain the edge enhancement tensor Fs for that scale. The edge enhancement tensors Fs of each scale form an edge enhancement tensor list, resulting in the two modal edge enhancement tensor lists multi-scale-features.

4. The multimodal semantic segmentation method based on edge enhancement and spatial-frequency feature fusion according to claim 3, characterized in that, The edge feature tensors obtained from the two modes through the Laplacian operator are weighted and summed to obtain the fused edge feature tensor E after fusing the two modes. combined Then, convolution operations, channel transformation, and softmax are performed to obtain the result tensor W with weight scores for each scale. s .

5. The multimodal semantic segmentation method based on edge enhancement and spatial-frequency feature fusion according to claim 4, characterized in that, The list of multi-scale edge enhancement tensors for two modalities, multi-scale-features, is combined with the weight fraction tensor W. s Each scale corresponds to a weighted summation, resulting in two multi-scale weighted summation edge enhancement tensors A. 1attention With A 2attention and the tensor A of the two modes 1attention With A 2attention Perform concatenation and convolution operations to obtain the multi-scale edge feature fusion result tensor A. edge The shape is consistent with the shape of the input tensor.

6. The multimodal semantic segmentation method based on edge enhancement and spatial-frequency feature fusion according to claim 5, characterized in that, The outputs of the two modes of the MEEFIM module are: ； ； In the formula, , and , Let represent two pairs of learnable parameters that sum to 1. and These represent the outputs of the two modes after passing through the MEEFIM module. and These represent the original inputs of the two modes in the MEEFIM module.

7. The multimodal semantic segmentation method based on edge enhancement and spatial-frequency feature fusion according to claim 6, characterized in that, The FSAM module processes the feature tensors of the two modes output by the MEEFIM module through a linear layer to obtain three self-attention matrices Q in the spatial domain. spatial K spatial V spatial ; In the FSAM module, two modes share a frequency domain branch. The tensors of the two modes are first obtained in the frequency domain through Fast Fourier Transform. By using three threshold-learnable low, mid, and high frequency filters, the frequency components of the low, mid, and high frequencies can be obtained. , , Then, a fast inverse Fourier transform is performed to obtain the low, mid, and high frequency components in the spatial domain. , , These represent the global structure, local structure, and detail texture, respectively.

8. The multimodal semantic segmentation method based on edge enhancement and spatial-frequency feature fusion according to claim 7, characterized in that, The feature difference Δ = |x1 - x2| between the two modal feature tensors input to the FSAM module is calculated by enhancing or suppressing the difference matrix. This difference is then subjected to lightweight multi-scale pooling convolution, followed by L2 normalization, ReLU activation, flattening, and min-max scaling. Finally, the dimensionality is restored to obtain the difference matrix D. Differential weights are then applied according to frequency bands, and enhancement or suppression operations are performed. ； Ultimately, information from two modes is mixed in one frequency band, and three outputs are generated through channel-level concatenation and SE channel attention. The tensor is a hybrid tensor of two modes, which has been enhanced or suppressed in the low, medium, and high frequency bands. ； In the formula, and The frequency band components that are enhanced or suppressed are obtained for each of the two modes. This is a tensor splicing operation. This is a classic channel attention mechanism.

9. The multimodal semantic segmentation method based on edge enhancement and spatial-frequency feature fusion according to claim 8, characterized in that, The FSAM module fuses features from the frequency and spatial domains through two consecutive operations. The attention weight matrix is ​​composed of intermediate and high frequencies from both the spatial and frequency domains. The frequency domain portion is fused with the intermediate frequency component through the attention output of the SE channel. As a query vector, high-frequency components are fused. As a supplement to the information of the key vector, the calculated spatial-frequency domain fusion attention score matrix is ​​obtained. ; Then, through cross-attention, each modality is multiplied by the value vector of the other modality to obtain the spatial cross-attention matrix SFFA that fuses frequency domain features. ； The frequency domain branch uses the low-frequency component as the query vector Q, the high-frequency component as the key vector K, and the mid-frequency component as the value vector V to perform attention calculation, resulting in the frequency domain branch attention matrix FDA. The spatial cross-attention matrix SFFA and the frequency domain branch attention matrix FDA are fused using a lightweight DCG within the FSAM module to obtain two tensors. and , ； In this process, DCG is formed by the inputs of two modalities passing through global average pooling and fully connected layers. Given the spatial cross-attention matrix of the DCG input, The frequency domain branch attention matrix is ​​the input DCG; two tensors and The two weights participating in the multimodal mixing operation are obtained by concatenation and the softmax function. and , 。 10. The multimodal semantic segmentation method based on edge enhancement and spatial-frequency feature fusion according to claim 9, characterized in that, The DCG outputs of the two modes within the FSAM are then subjected to a convolutional fusion operation to output a tensor that fuses multimodal features and frequency-spatial features. , ； Each stage of the FSAM module outputs a tensor like this, and the decoder aggregates these four output tensors to output the semantic segmentation prediction result.

Images