Remote sensing image segmentation method based on Transsubnet edge information enhancement and multi-dimensional feature perception

Abstract

The invention belongs to the field of deep learning technology and remote sensing image segmentation, and particularly relates to a remote sensing image segmentation method based on Transsubnet edge information enhancement and multi-dimensional feature perception, and the method comprises the steps: S1, preparing a data set; s2, constructing remote sensing picture text description; s3, constructing and training a remote sensing image segmentation model; and S4, storing and testing the model. The invention designs a multi-modal feature extraction method based on parallelism of a sampling branch and a text feature extraction branch under edge feature compensation. The residual error mixing axial attention module is used for forming a transformer structure; and a text-picture multi-dimensional feature fusion enhancement module and a decoder part are embedded. According to the method, the ground feature identification capability can be improved through a text and picture multi-modal feature enhancement strategy, the segmentation boundary and small target object feature information is enhanced, more fine-grained features are reserved, the cross-regional long-distance dependency relationship is better captured, the common gradient disappearance problem in a deep network is relieved, and the method is suitable for large-scale popularization and application. And meanwhile, the small sample data set segmentation effect is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning and remote sensing image segmentation, and particularly relates to a remote sensing image segmentation method based on edge information reinforcement and multi-dimensional feature perception of Transunet. BACKGROUND

[0002] With the development of remote sensing technology, remote sensing images can be used for forest coverage detection, coastline change monitoring, and other work, helping government agencies and research institutions to monitor environmental changes in real time; can quickly analyze the changes in disaster areas, providing accurate data support for disaster assessment and emergency response; can provide accurate map data for urban planning departments to improve the scientific nature and efficiency of planning. However, traditional methods are difficult to efficiently and accurately extract target information in images. Therefore, an accurate and efficient remote sensing image segmentation algorithm can classify each pixel in the remote sensing image, thereby achieving accurate identification of surface objects, which has important practical significance for various industries.

[0003] Chinese Patent Publication No. CN117876679A, entitled "Remote Sensing Image Scene Segmentation Method Based on Convolutional Neural Network", constructs an image segmentation network based on an improved TransUNet network; initializes the image segmentation network, sets the training parameters, and uses the training set to train the image segmentation network; inputs the test set into the image segmentation network to predict the scene target segmentation result. A foreground optimization method is used on the jump connection to associate the geographical space scene related context, improve the foreground feature recognition ability, and solve the foreground background imbalance problem. Embed a point flow module when cascading up-sampling, select the background prominent area point and target boundary point as the key point to propagate the context information, solve the semantic gap problem, get high semantic high resolution feature, reduce the background introduced noise while maintaining the efficiency. However, this method only considers the picture mode as the input of the network, and does not consider that TransUNet adopts a structure based on Transformer, which requires high memory and computing resources in the calculation process, so it may be difficult to deploy in hardware resource limited scenarios (such as edge devices or low configuration environment). In addition, the performance of the Transformer model is usually highly dependent on the size of the training data, thereby limiting the application effect on small data sets.

[0004] In remote sensing image segmentation, U-Net is a classic and widely used network model. It realizes pixel-level segmentation through an encoder-decoder structure, but since U-Net is mainly based on convolution operations, the receptive field of the convolution kernel is limited, so it performs poorly in capturing long-range context information in images; in remote sensing images, features such as buildings and roads may span a large range, and U-Net may not be able to efficiently capture these global dependencies; although U-Net combines features from different layers through skip connections, the fusion effect is limited when dealing with complex remote sensing scenes (such as objects of different scales), which can easily lead to loss of detail information or blurred boundaries. TransUNet is a hybrid model that combines Transformer and U-Net, aiming to improve performance in image segmentation tasks. It takes advantage of the strong global modeling ability of Transformer and the fine-grained feature extraction ability of U-Net, but due to the introduction of Transformer modules, TransUNet consumes a lot of computing resources and memory, making it unsuitable for running in low-resource environments; and the performance of Transformer-based models is positively correlated with the size of the data, meaning that Transformer-based models will be limited in small data scenarios. Therefore, the present application designs a multi-modal feature extraction method based on edge feature compensation down-sampling branch and text feature extraction branch in parallel instead of the original CNN encoding method in Transunet; the transformer structure composed of residual hybrid axial attention modules replaces the multi-head attention mechanism in the original Transunet; and the embedded text-picture multi-dimensional feature fusion enhancement module. The present application can enhance the segmentation boundary and small target object feature information, better capture the long-range dependencies across regions, and improve the segmentation effect of small sample datasets. SUMMARY

[0005] (I) Technical problems solved

[0006] To address the shortcomings of the prior art and achieve accurate segmentation of remote sensing images, the present application proposes a remote sensing image segmentation method based on Transunet edge information reinforcement and multi-dimensional feature perception, which solves the problems raised in the background art.

[0007] (II) Technical solutions

[0008] To achieve the above purpose, the present application specifically adopts the following technical solutions:

[0009] The remote sensing image segmentation method based on Transunet edge information reinforcement and multi-dimensional feature perception includes the following steps:

[0010] S1, preparing a data set: obtaining a public remote sensing image data set to be segmented, preprocessing the images in the data set, and dividing the images into a training set and a test set according to a required proportion;

[0011] S2, constructing a remote sensing picture text description: performing text description of each image in the training set in step S1 to be segmented;

[0012] S3, constructing and training a remote sensing image segmentation model: inputting the images in step S1 and the text description information in step S2 into a U-shaped segmentation network model based on a double-branch encoder, a residual hybrid axial attention module, a text-picture multi-dimensional feature fusion enhancement module, and a decoder; designing a reasonable training strategy according to the characteristics of remote sensing images, selecting a suitable loss function, training the remote sensing image segmentation network based on the edge information reinforcement and multi-dimensional feature perception of Transunet using the training set obtained in step S1, and calculating the loss;

[0013] S4, model saving and testing: replacing the optimal parameters into the segmentation network model according to the optimal parameters obtained in the training phase of the above segmentation network model, testing the network performance using the test set, and quantitatively evaluating the optimal segmentation model using evaluation indexes.

[0014] Further, in step S1, a public remote sensing image data set is first selected, and then the selected data set is input into ENVI software for radiation correction; after preprocessing, the high-resolution pictures in the data set are batched and cropped into 1067 images with a size of 512x512 pixels and labels; 70% of the experimental data set is used as a training set, and 30% is used as a test set.

[0015] Further, in step S2, the distribution number and density of the building, vehicle, and miscellaneous classes are described by using a template word filling method for each training set image; a rectangular coordinate system is established with the center point of each training set image as the origin to describe the quadrants occupied by impervious surfaces, trees, and low shrubs.

[0016] Further, in S3, the double-branch encoder includes a multi-modal feature extraction method in parallel with the edge feature compensation down-sampling branch and the text feature extraction branch, which replaces the CNN down-sampling method in the original Transunet;

[0017] The edge feature compensation down-sampling branch reduces the resolution of the remote sensing picture by half step by step, doubles the channel number step by step, performs feature extraction and refinement of the remote sensing picture, reduces the loss of edge information, and enhances the detail retention capability;

[0018] The text feature extraction branch is used for supplementing the remote sensing picture information, and the text-picture multi-modal feature enhancement strategy is used to improve the ground object discrimination capability.

[0019] Further, in the S3, the text-picture multi-dimensional feature fusion enhancement module is used for aligning and fusing the text features and the image features, and meanwhile, performing feature enhancement on the fused features in the spatial and channel dimensions.

[0020] Further, in the S3, the residual mixed axial attention module is used to replace the multi-head attention mechanism in the original Transunet.

[0021] The residual mixed axial attention module is composed of a multi-scale multi-head interactive gated axial attention mechanism and a multi-head relative position encoding axial attention mechanism. The multi-scale multi-head interactive gated axial attention mechanism captures the global dependency relationship after the strip convolution enhancement through the bidirectional cross attention in the horizontal and vertical directions, and meanwhile, the position encoding weight is adaptively adjusted by using the gating unit, so that the segmentation robustness under small sample data is improved. The relative position encoding axial attention mechanism can effectively enhance the feature capturing ability of the model in the segmentation task.

[0022] Further, in the S3, a suitable loss function is selected as a mixed loss function composed of a cross-entropy loss function L cross , a logarithmic cosine Dice loss function L lc-dice and a Jaccard loss function L IOU .

[0023] The cross-entropy loss function L cross is used to measure the matching degree of the model prediction and the target label.

[0024] The logarithmic cosine Dice loss function L lc-dice is used to manage the class imbalance problem and improve the precision and recall.

[0025] The Jaccard loss function L IOU is used to directly optimize the evaluation index of the Jaccard index, so as to improve the positioning or segmentation accuracy of the model.

[0026] Further, in the step S4, the optimal segmentation model is quantitatively evaluated by using evaluation indexes including mIou coefficients, mPrecision coefficients and mRecall coefficients.

[0027] (Three) beneficial effects

[0028] Compared with the prior art, the present application provides a remote sensing image segmentation method based on Transunet edge information reinforcement and multi-dimensional feature perception, and has the following beneficial effects:

[0029] The application designs a multi-modal feature extraction method based on an edge feature compensation down-sampling branch and a text feature extraction branch in parallel instead of the original CNN coding mode in Transunet. The edge information loss problem in the CNN down-sampling process is effectively compensated, the edge detection accuracy and detail retention ability are enhanced while maintaining efficient feature extraction; at the same time, the small target object feature information is enhanced, more fine-grained features are retained, and the cross-region long-distance dependency relationship is better captured; the text feature extraction branch is used for supplementing the remote sensing picture information, and the ground object discrimination ability is improved through the text and picture multi-modal feature enhancement strategy.

[0030] The application designs a residual mixed axial attention module to replace the original multi-head attention mechanism in the Transunet network, and the relative position encoding axial attention and the gated axial attention in the mixed axial attention are calculated in the horizontal direction and the vertical direction respectively, which reduces the calculation complexity of directly applying global attention to the whole image; after adding the gate, the position information can be learned more accurately to improve the small sample dataset segmentation accuracy; the residual structure helps to retain information flow and avoid information loss in the transmission process.

[0031] The application embeds a text-picture multi-dimensional feature fusion enhancement module. The text features and image features are aligned and fused during the feature transmission process, and the fused features are enhanced in the spatial and channel dimensions, and the image has stronger perception ability for detail features, and the relative position encoding information is implicitly added. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The flowchart of the remote sensing image segmentation method based on the Transunet edge information reinforcement and multi-dimensional feature perception proposed by the application;

[0033] Figure 2 The result example graph of the training set image for the segmentation class text description proposed by the application;

[0034] Figure 3 The overall network architecture diagram of the remote sensing image segmentation method based on the Transunet edge information reinforcement and multi-dimensional feature perception proposed by the application;

[0035] Figure 4 The structure diagram of the edge feature compensation module 2 proposed by the application;

[0036] Figure 5 The structure diagram of the edge feature compensation module 3 proposed by the application;

[0037] Figure 6 The structure diagram of the edge feature compensation module 4 proposed by the application;

[0038] Figure 7 Structure diagram of the text-picture multi-dimensional feature fusion enhancement module 1 proposed in the application;

[0039] Figure 8 Structure diagram of the text-picture multi-dimensional feature fusion enhancement module 2 proposed in the application;

[0040] Figure 9 Structure diagram of the text-picture multi-dimensional feature fusion enhancement module 3 proposed in the application;

[0041] Figure 10 Architecture diagram of the residual mixed axial attention module in the encoder part proposed in the application;

[0042] Figure 11 Architecture diagram of the relative position encoding axial attention machine module in the residual mixed axial attention module proposed in the application;

[0043] Figure 12 Structure diagram when the relative position encoding axial attention machine processes features;

[0044] Figure 13 Architecture diagram of the interactive gating axial attention machine module in the residual mixed axial attention module proposed in the application;

[0045] Figure 14 Structure diagram when the interactive gating axial attention machine processes features;

[0046] Figure 15 Comparison diagram of evaluation indexes of the remote sensing image segmentation effect of the remote sensing image segmentation method based on the Transunet edge information reinforcement and multi-dimensional feature perception proposed in the application and other open source methods. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0048] EMBODIMENT

[0049] As shown in the drawings, Figure 1 the application discloses a remote sensing image segmentation method based on Transunet edge information reinforcement and multi-dimensional feature perception, and the steps include:

[0050] S1, prepare the dataset, obtain the public remote sensing image dataset Postdam, the Postdam dataset is an airborne image dataset provided by the International Society for Photogrammetry and Remote Sensing (ISPRS) for the test project of urban classification and three-dimensional building reconstruction, the covered area is Potsdam, Germany, which is a typical historical city with large building groups, narrow streets and dense settlement structures, the dataset has been manually classified into 6 most common land cover categories: buildings, vehicles, clutter, roads, trees and low shrubs, and then the selected dataset is input into the ENVI software for radiation correction, after preprocessing, the 6000x6000 high-resolution pictures in the Postdam dataset are batched into 1067 512x512 pixel size images and labels, 70% of the experimental dataset is used as the training set, and 30% is used for the test set. In this example, the feature is represented as CxHxW, C is the feature dimension, and HxW is the feature resolution.

[0051] S2, construct a remote sensing picture description, each training set image in step S1 is described with a text description of the class to be segmented, each training set image is described by using a template to fill in the number of buildings, vehicles and clutter categories, a rectangular coordinate system is established with the center point of each training set image as the origin, and the quadrants occupied by impervious surfaces, trees and low shrubs are described. The template for filling in the words is "Buildings:__location(s);Cars:__unit(s);Clutter:__location(s).Impervious surfaces occupy the__quadrant(s) / No roads present;Trees occupy the__quadrant(s) / No trees present;Low vegeation occupies the__quadrant(s) / No low shrubs present." As shown in Figure 2 the example shows how to describe the presence or absence of the number of buildings, vehicles and clutter categories in the training set image, and the template filling method for describing the quadrants occupied by roads, trees and low shrubs.

[0052] S3, construct and train a remote sensing image segmentation model, input the images in step S1 and the text description information in step S2 into a U-shaped segmentation network model based on a double-branch encoder, a residual hybrid axial attention module, a text-picture multi-dimensional feature fusion enhancement module and a decoder, design a reasonable training strategy according to the characteristics of remote sensing images, select a suitable loss function, and use the training set obtained in step S1 to train the remote sensing image segmentation network based on the edge information enhancement and multi-dimensional feature perception of Transunet and calculate the loss, as shown in Figure 3As shown in the overall network architecture diagram of the remote sensing image segmentation method based on Transunet edge information reinforcement and multi-dimensional feature perception proposed in the application. The double-branch encoder includes a multi-modal feature extraction method in parallel with the edge feature compensation down-sampling branch and the text feature extraction branch instead of the CNN down-sampling method in the original Transunet. As shown in Figure 4 、 Figure 5 and Figure 6 , the edge feature compensation down-sampling branch is composed of a convolution layer 1, an edge feature compensation module 1, an edge feature compensation module 2, and an edge feature compensation module 3. In this embodiment, the convolution layer 1 (with a convolution kernel size of 7x7 and a step of 2) and the edge feature compensation module 1, the edge feature compensation module 2, and the edge feature compensation module 3 (A) part together complete four down-sampling operations and edge feature compensation, that is, the resolution size is halved in turn, while the channel number is increased accordingly; the (B) part of the edge feature compensation module 1, the edge feature compensation module 2, and the edge feature compensation module 3 is used to further refine the features and compensate the edge features. The edge feature compensation effect is embodied in the edge feature adaptive extraction part in the residual connection of the edge feature compensation module. The edge feature adaptive extraction part selects bilateral filtering to replace the Gaussian filtering in the original Canny operator; on the basis of the traditional 3x3 sobel operator and gradient template, 45° and 135° gradient templates are added, that is and Using non-maximum suppression, the gradient values in four directions are compared, and only the pixel points with local maximum values in the gradient direction are retained. This step traverses all pixel points, suppresses non-maximum points, and eliminates edge false detection; then, an adaptive threshold determination method based on an improved crow search algorithm is proposed to automatically determine the double thresholds according to the characteristics of each image. Among them, three improvements are made to the crow search algorithm: ①Introduce a segmented nonlinear dynamic perception probability adjustment strategy to replace the original fixed perception probability, which can be represented as:

[0053]

[0054] Where AP is the perception probability; iter is the iteration number.

[0055] ②Introduce the Levy flight strategy to improve the single search mechanism of the algorithm, which can be represented as:

[0056]

[0057] Where r a is a random number in the interval (0, 1); γ follows a normal distribution; n is the dimension; a is the step factor, which is 0.01, β is a constant, which is 1.5; Γ is the standard gamma function.

[0058] ③In order to avoid the blindness of the crow in the search, the information of the global optimal solution is introduced, so that the crow can be guided by the global optimal solution of the parent in the search, which can be expressed as:

[0059] x i,iter =x i,iter +r i ×fl i,iter ×(m i,iter -x i,iter )×r i ×fl i,iter ×(P best -x i , iter )

[0060] Wherein, P best is the global optimal solution.

[0061] In summary, the output feature sizes of the convolutional layer 1, the edge feature compensation module 1, the edge feature compensation module 2 and the edge feature compensation module 3 can be represented as: 64x256x256, 256x128x128, 512x64x64, 1024x32x32 respectively; denoted as: F image1 , F image2 , F image3 , F image4 .

[0062] The text feature extraction branch of the embodiment is composed of a CLIP text feature encoder and a feature dimension matching layer. The output feature of the text through the CLIP text feature encoder can be represented as: 6xC text ; the feature dimension matching layer matches the feature dimension of the output feature of the CLIP text feature encoder with the feature dimensions of F image1 , F image2 , F image3 , F image4 in turn, and the feature size of the output feature after the feature dimension matching layer can be represented as: 6x64, 6x256, 6x512, 6x1024; denoted as: F text1 , F text2 , F text3 , F text4 .

[0063] The text-picture multi-dimensional feature fusion enhancement module of the embodiment is used to align and fuse text features and image features, and to perform spatial and channel dimension feature enhancement on the fused features. The present application uses four text-picture multi-dimensional feature fusion enhancement modules, such as Figure 7As shown, the input features of the text-image multi-dimensional feature fusion enhancement module 1 are the output features F of convolutional layer 1. image1 The output feature F after the feature dimension matching layer text1 After performing batch matrix multiplication, a matching score map with a feature size of 6×64×64 is obtained. Then, the matching score map is combined with the output feature F of convolutional layer 1. image1 After performing a concat operation, dimensionality reduction is achieved through 1×1 convolution, which then serves as the input feature for subsequent multi-dimensional feature enhancement. Multi-dimensional feature enhancement is applied to the input features... Instead of performing 3×3 soft pooling on the features (F'1), which are then subjected to global max pooling and global average pooling respectively, resulting in four 64×1×1 features, the input is fed into a shared multilayer perceptron that first increases and then decreases the dimensionality. The resulting four feature maps are then element-wise summed and followed by Sigmoid activation. Simultaneously, the 3×3 soft pooled features (F'1) are... ’ 1) After channel-based global max pooling and global average pooling, two features represented as 1×254×254 are obtained. Input features After channel-based global max pooling and global average pooling, two features of 1×256×256 are obtained. These features are concatenated pairwise and then reduced in dimensionality by 7×7 and 5×5 convolutions before being summed. Sigmoid activation is then applied, and the weights of these two activations are respectively added to the input features. Multiplication yields the enhanced multidimensional feature F1, denoted as:

[0064]

[0065] `concat(*)` represents the feature concatenation operation, `σ` represents the Sigmoid activation operation, and `MLP(*)` represents the shared multilayer perceptron operation of first increasing dimensionality and then decreasing dimensionality. 7×7 (*) represents a 7×7 convolutional dimensionality reduction operation, f 5×5 (*) represents a 5×5 convolution dimensionality reduction operation.

[0066] like Figure 8 As shown, the input features of the text-image multi-dimensional feature fusion enhancement module 2 are the current stage image features (the output features F of the edge feature compensation module 1). image2 The feature F is output after the feature dimension matching layer. text2 The output feature F1 of the text-image multi-dimensional feature fusion enhancement module 1 is then processed through multi-scale convolution downsampling to reduce its feature resolution to match that of feature F. image2 The same, so we get feature F. 1down For feature F image2 With feature F1down The adaptive weights a and 1-a are set respectively. Then the batch matrix multiplication is performed together to obtain a matching score map with a feature size of 6x256x256. The subsequent process of the text-picture multi-dimensional feature fusion enhancement module 2 is exactly the same as that of the text-picture multi-dimensional feature fusion enhancement module 1, and finally the multi-dimensional feature enhanced feature F2 is obtained. The overall process of the text-picture multi-dimensional feature fusion enhancement module 3 is exactly the same as that of the text-picture multi-dimensional feature fusion enhancement module 2, and finally the multi-dimensional feature enhanced feature F3 is obtained.

[0067]

[0068] As shown in Figure 9 , the process of obtaining the multi-dimensional feature enhanced input feature in the text-picture multi-dimensional feature fusion enhancement module 4 is exactly the same as that of the text-picture multi-dimensional feature fusion enhancement module 3. The core goal of adding a new intermediate branch in the multi-dimensional feature enhancement process compared with the other three text-picture multi-dimensional feature fusion enhancement modules is to efficiently capture and fuse the context information from different spatial scales in the deep neural network. In order to optimize the calculation efficiency and promote the cross-channel information interaction, the input feature is first processed through a 1x1 convolutional layer to compress the channel dimension of the input feature to 256, generating a more compact intermediate feature representation X1. Next, parallel multi-scale feature extraction is performed, and the intermediate feature X1 is simultaneously input into four independent convolutional branches. The first branch uses a standard 3x3 convolution, mainly focusing on local detail information. The other three branches use 3x3 dilated convolution with dilation rates of 3, 6 and 12 respectively, to ensure that the outputs of all four parallel branches maintain a channel dimension of 256. The features extracted by the 3x3 convolution feature extraction branch are first added element by element with the intermediate feature X1 generated at the beginning of the module, as the input feature of the 3x3 convolution feature extraction branch with a dilation rate of 3. Then the features extracted by the 3x3 convolution feature extraction branch with a dilation rate of 3 are added pixel by pixel with the input feature of the 3x3 convolution feature extraction branch with a dilation rate of 3, as the input feature of the 3x3 convolution feature extraction branch with a dilation rate of 6, and so on. Subsequently, the four feature maps with a channel number of 256 are concatenated along the channel dimension to form an aggregated feature map with a channel number of 1024 containing multi-scale information. Finally, in order to integrate these features from different scales and paths, the original input feature is directly added between them for sufficient information interaction. The concatenated feature map with a channel number of 2048 will pass through a final 1x1 convolutional layer to restore to the input feature dimension, and the intermediate branch output feature is denoted as F middle . The subsequent process of the text-picture multi-dimensional feature fusion enhancement module 4 is exactly the same as that of the text-picture multi-dimensional feature fusion enhancement module 3, and finally the multi-dimensional feature enhanced feature F4 is obtained.

[0069]

[0070] The input feature of the residual mixed axial attention module in this embodiment is the final feature obtained by the multi-modal feature extraction method in parallel with the edge feature compensation downsampling branch and the text feature extraction branch, that is, the input feature is F4. As shown in Figure 10 each residual mixed axial attention module is composed of an interactive gated axial attention mechanism module and two relative position encoding axial attention mechanism modules, and the input feature and the output feature after passing through the mixed axial attention module are combined through a skip connection to form a residual mixed axial attention module. The transformer structure is composed of four residual mixed axial attention modules in series, and the input feature F4 passes through the transformer structure to obtain the final output feature of the encoder part, which has the same resolution, dimension and feature F4, represented as 1024x32x32.

[0071] As shown in Figure 11 the relative position encoding axial attention mechanism module in the residual mixed axial attention module in the encoder part of this embodiment is composed of a normalization layer, a multi-head relative position encoding axial attention mechanism and a multi-layer perception. The multi-head relative position encoding axial attention mechanism is specifically selected as 8 heads, that is, the feature is evenly decomposed into 8 features with the same dimension, and then the feature dimension is restored through the concat operation after the representation learning of the 8 relative position encoding width axial attention mechanisms and the representation learning of the 8 relative position encoding height axial attention mechanisms. The relative position encoding axial attention mechanism, that is, the introduction of relative position information in the axial attention mechanism, can effectively enhance the feature capturing ability of the model in the segmentation task. For the image segmentation task, the relative position information introduced from the query angle helps to distinguish the adjacent target and background; by adding the relative position information in the key, the model can provide more clues about the position relationship of each pixel with other pixels.

[0072] As shown in Figure 12 the position encoding along the width axis direction and the height axis direction when the relative position encoding axial attention mechanism processes the feature can be represented as:

[0073] n is the specific relative position encoding axial attention mechanism in the multi-head relative position encoding axial attention mechanism, q ij , k iw , v ih is the query matrix, T is the matrix transpose, R Q , R K , RV It is a relative position encoding.

[0074] In this embodiment, the multi-head relative position encoding axial attention mechanism for processing features can be represented as follows:

[0075]

[0076] Norm(*) represents the normalization layer operation, PMA(*) represents the feature learning result of the multi-head relative position encoding axial attention mechanism, and concat(*) represents the feature concatenation operation.

[0077] like Figure 13 As shown, the multi-scale multi-head interactive gating axial attention mechanism module in this embodiment consists of a normalization layer, a multi-scale multi-head interactive gating axial attention mechanism, and a multilayer perceptron. Each branch of the multi-scale interactive gating axial attention mechanism (two parallel branches, calculating attention along the horizontal and vertical axes respectively) consists of three one-dimensional convolutions with different kernel sizes. These convolutions encode multi-scale contextual information along one spatial dimension, leveraging the power of convolution to capture multi-scale feature representations, aiming to better segment lesion regions with various individual sizes and shapes. Specifically, the multi-scale multi-head interactive gating axial attention mechanism is chosen to have 8 heads, meaning the features are decomposed into 8 features with the same dimensions. After representation learning through 8 interactive gating axial attention mechanisms, the feature dimensions are restored through a concat operation. This method is based on effective axial attention with added relative position encoding, and further incorporates gating units. When data samples are limited and the network struggles to accurately capture positional biases, the gating units adaptively adjust the relative position encoding, assigning it a smaller weight, thereby weakening the impact of inaccurate positional information on the model's segmentation accuracy. Furthermore, global information is better captured by calculating bidirectional cross-attention between two parallel gated axial attentions.

[0078] 1x3, 1x5, and 1x7 convolutional kernels capture local contextual information of different ranges in the width dimension, serving as input features for the multi-scale interactive gated axial attention mechanism along the width direction; 3x1, 5x1, and 7x1 convolutional kernels capture local contextual information of different ranges in the height dimension, serving as input features for the multi-scale interactive gated axial attention mechanism along the height direction. This multi-scale local information enhances the ability of subsequent width-axis attention to model long-range dependencies. Providing preprocessed features that have encoded local multi-scale information in the width and height directions for axial attention allows for more intelligent and effective modeling of global dependencies along that axis. The multi-scale bar convolutional encoding in the multi-scale multi-head interactive gated axial attention mechanism can be represented as follows:

[0079] f w = f 1×1 (concat(f 1×3 (Norm(f)),f 1×5 (Norm(f)),f 1×7 (Norm(f)), Norm(d)))

[0080] f h = f 1×1 (concat(f 3×1 (Norm(f)),f 5×1 (Norm(f)),f 7×1 (Norm(f)), Norm(f)))

[0081] f w , f h are output features of input features Norm(f) along the width axis direction and the height axis direction, respectively, as input features of the interactive gated axial attention mechanism along the width axis direction and the height axis direction, Norm(*) represents a normalization layer operation, and concat(*) represents a feature concatenation operation.

[0082] As shown in Figure 14 , the position encoding along the width axis direction and the height axis direction when the interactive gated axial attention mechanism processes the features can be represented as

[0083] wherein n is the specific interactive gated axial attention mechanism in the multi-head interactive gated axial attention mechanism, q ij , k iw , and v ih are query matrices, wherein the superscripts f h , f w represent the sources of the query matrices, T is a matrix transpose, R Q , R K , and R V are relative position encodings, G Q , G K , and G V are adaptive learning gate units.

[0084] wherein the multi-scale multi-head interactive gated axial attention mechanism can be represented as

[0085]

[0086] CGMA(*) is the multi-head interactive gated axial attention mechanism feature learning result, and concat(*) represents the feature concatenation operation.

[0087] Taking the residual hybrid axial attention mechanism module 1 as an example, the input feature F4 first enters the interactive gated axial attention mechanism module in the first residual hybrid axial attention mechanism module. The feature learning of this module can be expressed as:

[0088] F w =f 1×1 (concat(f 1×3 (Norm(F4)),f 1×5 (Norm(F4)),f 1×7 (Norm(F4)),Norm(F4)))

[0089] F h =f 1×1 (concat(f 3×1 (Norm(F4)),f 5×1 (NormF4),f 7×1 (Norm(F4)),Norm(F4)))

[0090] F 41 =F4+(CGMA(F h ,F w ,F w )+CGMA(F w ,F h ,F h ))

[0091] F 42 =F 41 +MLP(Norm(F 41 ))

[0092] F w 、F h are the output features of the input feature Norm(F) along the width axis direction and the height axis direction, respectively, which are the input features of the interactive gated axial attention mechanism along the width axis direction and the height axis direction in this embodiment. Norm(*) represents the normalization layer operation, CGMA(*) is the multi-head interactive gated axial attention mechanism feature learning result, and Norm(F4) y 、Norm(F4) x The subscripts x and y in Norm(F4) represent the query matrix from different axial directions, and MLP(*) represents the multi-layer perception operation.

[0093] Secondly, it enters two relative position encoding axial attention mechanism modules, and the feature learning can be expressed as,

[0094] F 43 =F 42 +(PMA(Norm(F 42 ) x Norm(F) 42 ) x Norm(F) 42 ) x )+PMA(Norm(F 42 ) y Norm(F) 42 ) y Norm(F) 42 ) y ))

[0095] F 44 =F 43 +MLP(Norm(F 43 ))

[0096] F 45 =F 44 +(PMA(Norm(F 44 ) x Norm(F) 44 ) x Norm(F) 44 ) x )+PMA(Norm(F 44 ) y Norm(F) 44 ) y Norm(F) 44 ) y ))

[0097] F 46 =F 45 +MLP(Norm(F 45 ))

[0098] Norm(*) represents the normalization layer operation, PMA(*) represents the feature learning result of the multi-head relative position encoding axial attention mechanism, and MLP(*) represents the multilayer perceptron operation. F 43 F represents the feature learning result of the first relative position encoding axial attention mechanism module. 45 The feature learning representation of the second relative position encoding axial attention mechanism module is used.

[0099] The output of the second relative position encoding axial attention mechanism module is added to the input of the hybrid axial attention module to obtain the final output of hybrid axial attention module 1, denoted as F. 47 =F4+F 46The mixed axial attention module 2, the mixed axial attention module 3, and the mixed axial attention module 4 have the same processing procedure as the mixed axial attention module 1.

[0100] In the S3, a suitable loss function is a cross-entropy loss function L cross , a log-cosh Dice loss function L lc-dice , and an IoU loss function L IOU The mixed loss function is represented as:

[0101] Loss = aL cross + bL lc-dice + (1-a-b)L IOU

[0102] The cross-entropy loss function L cross is used to measure the matching degree between the model prediction and the target label, and is represented as:

[0103]

[0104] where N represents the number of samples in the remote sensing data set, S represents the number of categories, y i represents the label of the i-th sample in the data set, and P i represents the training prediction result of the i-th sample in the data set.

[0105] The log-cosh Dice loss function L lc-dice is used to manage the class imbalance problem and improve the precision and recall rate, and is represented as:

[0106] L lc-dice = log(cosh(L dice ))

[0107]

[0108] where X represents the set of predicted remote sensing image pixels, Y represents the set of labeled remote sensing image pixels, |X| represents the number of predicted remote sensing image pixels, and |Y| represents the number of labeled remote sensing image pixels.

[0109] The IoU loss function L IOU is used to directly optimize the evaluation index of IoU, so as to improve the positioning or segmentation accuracy of the model, and is represented as:

[0110]

[0111] where M represents the predicted segmentation mask, and N represents the real segmentation mask.

[0112] S4, Model Saving and Testing: Based on the segmentation network model described above, obtain the optimal parameters from the training phase, replace the optimal parameters with those parameters in the segmentation network model, test the network performance using the test set, and use evaluation metrics to quantitatively evaluate the optimal segmentation model.

[0113] In step S4, evaluation metrics are used to quantitatively evaluate the optimal segmentation model. These metrics include the mIou coefficient, mPrecision coefficient, and mRecall coefficient.

[0114] The formula for the mIou coefficient is:

[0115]

[0116] The formula for the mPrecision coefficient is:

[0117]

[0118] The formula for the mRecall coefficient is:

[0119]

[0120] TP represents true positives, FN represents false negatives, FP represents false positives, and k represents the number of classes to be segmented in the dataset.

[0121] like Figure 15 The figure shows the experimental performance comparison results of the proposed method with U-Net, U-Net++, TransUnet, ECANet, and DeepLabv3+ segmentation methods on the PostDam remote sensing dataset. It can be seen that the proposed method has higher mIou, mPrecision, and mRecall parameters, which further demonstrates that the proposed method has achieved the expected results.

[0122] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A remote sensing image segmentation method based on Transunet edge information enhancement and multi-dimensional feature perception, characterized in that, Includes the following steps: S1, Prepare the dataset: Obtain the publicly available remote sensing image dataset to be segmented, preprocess the images in the dataset, and divide them into training and test sets according to the required proportions; S2, Constructing Remote Sensing Images. This article describes: Each training set image in step S1 is given a text description of the category to be segmented. S3, Construct and train the remote sensing image segmentation model: Input the image from step S1 and the text description information from step S2 into a U-shaped segmentation network model based on a dual-branch encoder, residual hybrid axial attention module, text-image multi-dimensional feature fusion enhancement module and decoder for training; Design a reasonable training strategy according to the characteristics of remote sensing images, select an appropriate loss function, and use the training set obtained in step S1 to train the remote sensing image segmentation network based on Transunet edge information enhancement and multi-dimensional feature perception and calculate the loss; S4, Model Saving and Testing: Based on the segmentation network model described above, obtain the optimal parameters during the training phase, replace the optimal parameters into the segmentation network model, test the network performance using the test set, and use evaluation metrics to quantitatively evaluate the optimal segmentation model.

2. The remote sensing image segmentation method based on Transunet edge information enhancement and multi-dimensional feature perception according to claim 1, characterized in that: In step S1, the publicly available Potsdam remote sensing image dataset is first selected. Preprocessing involves inputting the selected dataset into ENVI software for radiometric correction. After preprocessing, the high-resolution images in the dataset are batch-cropped into 1067 images of 512×512 pixels each, along with labels. 70% of the experimental dataset is used as the training set, and 30% is used as the test set.

3. The remote sensing image segmentation method based on Transunet edge information enhancement and multi-dimensional feature perception according to claim 1, characterized in that: In step S2, each training set image will be described by using a template-filling method to describe the distribution of buildings, vehicles and debris; a rectangular coordinate system will be established with the center point of each training set image as the origin to describe the quadrants occupied by impermeable ground, trees and low shrubs.

4. The remote sensing image segmentation method based on Transunet edge information enhancement and multi-dimensional feature perception according to claim 1, characterized in that: In S3, the dual-branch encoder includes a multimodal feature extraction method that combines an edge feature compensation downsampling branch with a text feature extraction branch, replacing the CNN downsampling method in the original Transunet. The edge feature compensation downsampling branch halves the resolution of the remote sensing image step by step and doubles the number of channels step by step to extract and refine the features of the remote sensing image, while reducing the loss of edge information and enhancing the ability to preserve details. The text feature extraction branch is used to supplement remote sensing image information and improves the ability to identify ground features through text and image multimodal feature enhancement strategies.

5. The remote sensing image segmentation method based on Transunet edge information enhancement and multi-dimensional feature perception according to claim 1, characterized in that: In S3, the text-image multi-dimensional feature fusion enhancement module is used to align and fuse text features and image features, and at the same time enhance the fused features in terms of spatial and channel dimensions.

6. The remote sensing image segmentation method based on Transunet edge information enhancement and multi-dimensional feature perception according to claim 1, characterized in that: In S3, the transformer structure composed of residual hybrid axial attention modules replaces the multi-head attention mechanism in the original Transunet. The residual hybrid axial attention module is composed of a multi-scale multi-head interactive gating axial attention mechanism and a multi-head relative position encoding axial attention mechanism. The multi-scale multi-head interactive gating axial attention mechanism captures the global dependencies enhanced by strip convolution through bidirectional cross attention in the horizontal and vertical directions, while using gating units to adaptively adjust the position encoding weights to improve the segmentation robustness under small sample data. The relative position encoding axial attention mechanism introduces relative position information into the axial attention mechanism, which can effectively enhance the feature capture ability of the model in segmentation tasks.

7. The remote sensing image segmentation method based on Transunet edge information enhancement and multi-dimensional feature perception according to claim 1, characterized in that: In S3, a suitable loss function is selected as the cross-entropy loss function L. cross Log-cosine Dice loss function L lc-dice Cross-comparison loss function L IOU The resulting hybrid loss function; The cross-entropy loss function L cross This is used to measure the degree of matching between the model prediction and the target label; The log-cosine Dice loss function L lc-dice It is used to manage class imbalance and improve precision and recall. The cross-union ratio loss function L IOU This is used to directly optimize the intersection-union ratio (IUGR) evaluation metric, thereby improving the model's localization or segmentation accuracy.

8. The remote sensing image segmentation method based on Transunet edge information enhancement and multi-dimensional feature perception according to claim 1, characterized in that: In step S4, the optimal segmentation model is quantitatively evaluated using evaluation metrics, including the mIou coefficient, mPrecision coefficient, and mRecall coefficient.

Images