Remote sensing image semantic segmentation method, device and equipment based on hybrid feature extraction

Abstract

The application relates to a remote sensing image semantic segmentation method, device and equipment based on mixed feature extraction. The method comprises the following steps: inputting an acquired remote sensing image to be segmented into a backbone network to obtain output features of each stage; the backbone network is a context clustering network CoCs; inputting the output features of the fourth stage into a semantic feature enhancement extraction module to filter and extract key semantic information by adopting a Transform module; inputting the output features of the second stage into an enhanced feature extraction module to perform extraction by adopting an inverted bottleneck module based on a depth separable convolution; inputting the output features of the third stage, semantic enhanced features and high-resolution enhanced features into a feature fusion module to perform feature fusion and obtain fused features; and inputting the fused features into a segmentation head to obtain a semantic segmentation result of the remote sensing image to be segmented. The method can realize accurate segmentation of the remote sensing image and balance between the calculation cost and the performance effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a remote sensing image semantic segmentation method, device and equipment based on hybrid feature extraction. BACKGROUND

[0002] With the continuous development of remote means such as aircraft and satellites to obtain information about the features of the earth's surface, through remote sensing technology, information about the earth's surface in multiple aspects such as land, sea, atmosphere, and biology can be observed, and analysis and processing of them can be achieved. Remote sensing technology has been widely used in the fields of agriculture, forestry, water resource management, urban planning, environmental protection, weather forecasting, and land resource investigation. How to utilize these increasingly rich and perfect remote sensing datasets for intelligent observation of the earth is a great opportunity and challenge in the field of remote sensing. Remote sensing semantic segmentation uses images obtained by remote sensing technology, and classifies and labels them at the pixel level through computer vision and deep learning algorithms, so that each pixel point in the image is labeled as the corresponding category, thereby achieving semantic understanding and recognition of the image. The main requirement is that the network must be able to capture the overall view of the remote sensing scene while retaining the details and semantic information of the remote sensing image. In remote sensing applications, semantic segmentation can help quickly and accurately extract various ground object information such as buildings, roads, rivers, and forests from images, thereby enabling fine-grained analysis and research of the features of the earth's surface, and can also assist in decision-making in the fields of agriculture, forestry, and water resource management, which is of great significance for the utilization of remote sensing datasets.

[0003] Traditional semantic segmentation methods usually use region-based methods, such as using region growing or clustering algorithms to divide pixels into different regions. These methods can achieve segmentation of objects in images to some extent, but cannot accurately label object boundaries, and the processing effect on complex scenes is not ideal. In recent years, deep learning methods have become the mainstream method in the field of semantic segmentation. By using deep learning models such as convolutional neural networks (CNN), image features can be effectively extracted and accurate pixel-level classification can be performed. Among them, fully convolutional networks (FCN), encoder-decoder networks (Encoder-Decoder), dilated convolution networks (Dilated ConvNet), and segmentation networks (SegNet) are widely used in semantic segmentation tasks. Recently, ViTs have introduced pure self-attention-based Transformers into the visual field and achieved new SOTA performance on various visual tasks, such as SegFormer. Although successful, ViTs related to global attention require powerful computing resources and are not friendly to some mobile devices. SUMMARY

[0004] Therefore, it is necessary to provide a remote sensing image semantic segmentation method, device and equipment based on mixed feature extraction to solve the above technical problems.

[0005] A remote sensing image semantic segmentation method based on mixed feature extraction, the method comprises:

[0006] Input the obtained remote sensing image to be segmented into a backbone network to obtain output features of each stage; the backbone network is a context clustering network CoCs.

[0007] The output features of the fourth stage are input into a semantic feature enhancement extraction module to filter and extract key semantic information by using a Transformer module to obtain semantic enhancement features.

[0008] The output features of the second stage are input into an enhanced feature extraction module to extract by using an inverted bottleneck module based on a depth separable convolution to obtain high-resolution enhanced features.

[0009] The output features of the third stage, the semantic enhancement features and the high-resolution enhanced features are input into a feature fusion module for feature fusion to obtain fusion features.

[0010] The fusion features are input into a segmentation head to obtain a semantic segmentation result of the remote sensing image to be segmented.

[0011] In one of the embodiments, the semantic feature enhancement extraction module is composed of a plurality of stacked Transformer modules.

[0012] In one of the embodiments, the enhanced feature extraction module is composed of a plurality of inverted bottleneck modules based on depth separable convolution.

[0013] In one of the embodiments, the feature fusion module includes two 1x1 convolution layers and two up-sampling modules.

[0014] The output features of the third stage, the semantic enhancement features and the high-resolution enhanced features are input into a feature fusion module for feature fusion to obtain fusion features, including:

[0015] The semantic enhancement features are changed in dimension by using a first 1x1 convolution layer, and the obtained features are up-sampled by using a first up-sampling module to obtain intermediate semantic enhancement features.

[0016] The high-resolution enhanced features are changed in dimension by using a second 1x1 convolution layer to obtain intermediate high-resolution enhanced features.

[0017] The output features of the third stage are up-sampled by using a second up-sampling module to obtain up-sampled features.

[0018] The intermediate semantic enhancement feature, the intermediate high-resolution enhancement feature, and the up-sampling feature are multiplied to obtain a first fusion feature.

[0019] The first fusion feature, the intermediate semantic enhancement feature, and the intermediate high-resolution enhancement feature are added and fused to obtain a fusion feature.

[0020] In one of the embodiments, the segmentation head includes two convolution modules; the convolution modules include convolution layers, batch normalization processing layers, and ReLu activation functions.

[0021] The fusion feature is input into the segmentation head to obtain a semantic segmentation result of the to-be-segmented remote sensing image, including:

[0022] The fusion feature is input into the first convolution module of the segmentation head to obtain a feature with a reduced channel dimension.

[0023] The feature with the reduced channel dimension is input into the second convolution module of the segmentation head to obtain the semantic segmentation result of the to-be-segmented remote sensing image.

[0024] In one of the embodiments, the context clustering network CoCs includes four stages, each of which includes a third down-sampling module and a plurality of clustering blocks;

[0025] The number of clustering blocks in each stage is distributed as 3, 4, 5, and 2; and the channel dimension of each stage is 32, 64, 196, and 320, respectively.

[0026] A remote sensing image semantic segmentation device based on mixed feature extraction, the device includes:

[0027] A multi-layer feature extraction module is configured to input a to-be-segmented remote sensing image obtained to a backbone network to obtain output features of each stage; the backbone network is a context clustering network CoCs.

[0028] A semantic feature and detail enhancement feature extraction module is configured to input the output feature of the fourth stage to a semantic feature enhancement extraction module to filter and extract key semantic information by using a Transformer module to obtain a semantic enhancement feature; and input the output feature of the second stage to an enhancement feature extraction module to extract by using an inverted bottleneck module based on a depth separable convolution to obtain a high-resolution enhancement feature.

[0029] A mixed feature fusion module is configured to input the output feature of the third stage, the semantic enhancement feature, and the high-resolution enhancement feature to a feature fusion module to perform feature fusion to obtain a fusion feature.

[0030] The semantic segmentation module is configured to input the fusion feature into a segmentation head to obtain a semantic segmentation result of the remote sensing image to be segmented.

[0031] In one of the embodiments, the semantic feature enhancement extraction module in the semantic feature and detail enhancement feature extraction module is stacked by a plurality of Transformer modules.

[0032] In one of the embodiments, the enhancement feature extraction module in the semantic feature and detail enhancement feature extraction module is stacked by a plurality of inverted bottleneck modules based on depth separable convolution.

[0033] A computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the following steps:

[0034] The obtained remote sensing image to be segmented is input into a backbone network to obtain output features of each stage; the backbone network is a context clustering network CoCs.

[0035] The output feature of the fourth stage is input into a semantic feature enhancement extraction module to filter and extract key semantic information by using a Transformer module, to obtain semantic enhancement features.

[0036] The output feature of the second stage is input into an enhancement feature extraction module to extract by using an inverted bottleneck module based on depth separable convolution, to obtain high-resolution enhancement features.

[0037] The output feature of the third stage, the semantic enhancement features and the high-resolution enhancement features are input into a feature fusion module to perform feature fusion, to obtain fusion features.

[0038] The fusion features are input into a segmentation head to obtain a semantic segmentation result of the remote sensing image to be segmented.

[0039] The remote sensing image semantic segmentation method, device and equipment based on mixed feature extraction, the method comprises: inputting an obtained remote sensing image to be segmented into a backbone network to obtain output features of each stage; the backbone network is a context clustering network CoCs; inputting the output features of the fourth stage into a semantic feature enhancement extraction module to filter and extract key semantic information by using a Transform module to obtain semantic enhancement features; inputting the output features of the second stage into an enhanced feature extraction module to extract by using an inverted bottleneck module based on a depth separable convolution to obtain high-resolution enhanced features; inputting the output features of the third stage, the semantic enhancement features and the high-resolution enhanced features into a feature fusion module to perform feature fusion to obtain fusion features; and inputting the fusion features into a segmentation head to obtain a semantic segmentation result of the remote sensing image to be segmented. The method can realize accurate segmentation of remote sensing images, and balance between calculation cost and performance effect. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A flowchart of a remote sensing image semantic segmentation method based on mixed feature extraction in an embodiment;

[0041] Figure 2 A network structure diagram of a remote sensing image semantic segmentation method based on mixed feature extraction in another embodiment;

[0042] Figure 3 A structure diagram of a semantic feature enhancement extraction module in another embodiment;

[0043] Figure 4 A structure diagram of an enhanced feature extraction module in another embodiment;

[0044] Figure 5 A structure block diagram of a remote sensing image semantic segmentation device based on mixed feature extraction in an embodiment;

[0045] Figure 6 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0046] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0047] The remote sensing image semantic segmentation network based on mixed feature extraction is abbreviated as HFE-Net.

[0048] In semantic segmentation, several key elements are crucial to achieve high-performance results, including capturing contextual information with a large receptive field, achieving precise segmentation with a high-resolution feature, precise prediction with the fusion of details and semantic features, and improving the attention mechanism of feature representation. The most advanced models usually combine several or even all of these elements to achieve superior performance.

[0049] In one embodiment, as shown in Figure 1 a remote sensing image semantic segmentation method based on hybrid feature extraction is provided, which comprises the following steps:

[0050] Step 100: input the acquired remote sensing image to be segmented into a backbone network to obtain output features of each stage; the backbone network is a context clustering network CoCs.

[0051] Specifically, the backbone network is a context clustering network CoCs. In traditional semantic segmentation, the classic method is to divide pixels into different regions by region growing or clustering algorithm. These methods can achieve segmentation of objects in the image to some extent, but cannot accurately mark the object boundary. With the development of deep learning, models based on deep learning gradually occupy the field of semantic segmentation, but these developments also indirectly indicate that clustering algorithms have an inherent advantage in image region division, especially in the field of image segmentation. The context clustering network CoCs regards the image as a series of unordered points, each point including original features and position information, and performs hierarchical grouping and feature extraction on deep features through a simplified clustering algorithm. The context clustering network CoCs provides a new perspective for images and visual representation. The clustering algorithm induces all pixel points in a picture into several clusters through feature similarity, and performs feature aggregation and feature update within the cluster. Since the total number of pixel points in a cluster is not fixed, it is possible that the total number of pixel points in a cluster is zero, so a pixel point will interact with other pixel points in an irregular and unstable receptive field. This feature meets the receptive field requirement in image segmentation.

[0052] The entire backbone network is divided into four stages, each stage containing a downsampling and N clustering blocks. The downsampling module is used to reduce the resolution of the feature map and increase the dimension of the feature map.

[0053] Step 102: input the output features of the fourth stage into a semantic feature enhancement extraction module to filter and extract key semantic information using a Transformer module to obtain semantic enhancement features.

[0054] Specifically, the remote sensing image to be segmented is processed by the backbone network. In order to obtain enhanced spatial details and global semantic information in the dense prediction task, an enhanced feature extraction module based on convolution and ViT is used to extract features from the features output by the backbone network, and the extracted features are fused into the original features through a feature fusion module.

[0055] In particular, for the network output in the low-layer high-resolution stage (the second stage of the backbone network), based on the fact that the features at this scale contain rich spatial details and texture information, N inverted bottleneck modules based on convolution are used to capture more accurate segmentation details.

[0056] Step 104: input the output features of the second stage into the enhanced feature extraction module to extract using the inverted bottleneck module based on deep separable convolution, to obtain high-resolution enhanced features.

[0057] In particular, for the network output in the high-layer low-resolution stage (the fourth stage of the backbone network), based on the fact that the features at this scale contain rich global semantic information, M ViT modules based on Transformer are used to capture semantic feature information.

[0058] Step 106: input the output features of the third stage, the semantic enhanced features and the high-resolution enhanced features into the feature fusion module for feature fusion, to obtain fused features.

[0059] Specifically, in order to realize the fusion of these features, a feature fusion module is used to fuse the details and global features in the feature extraction module with the corresponding original scale features to enhance the representation.

[0060] Step 108: input the fused features into the segmentation head to obtain the semantic segmentation result of the remote sensing image to be segmented.

[0061] The remote sensing image semantic segmentation network based on mixed feature extraction is composed of the backbone network, the semantic feature enhancement extraction module, the enhanced feature extraction module, the feature fusion module and the segmentation head. The structure of the remote sensing image semantic segmentation network based on mixed feature extraction is shown in Figure 2

[0062] ​In the remote sensing image semantic segmentation method based on mixed feature extraction, the method comprises the following steps: inputting an obtained remote sensing image to be segmented into a backbone network to obtain output features of each stage; the backbone network is a context clustering network CoCs; inputting the output features of the fourth stage into a semantic feature enhancement extraction module to filter and extract key semantic information by using a Transformer module to obtain semantic enhancement features; inputting the output features of the second stage into an enhanced feature extraction module to extract by using an inverted bottleneck module based on a depth separable convolution to obtain high-resolution enhanced features; inputting the output features of the third stage, the semantic enhancement features and the high-resolution enhanced features into a feature fusion module to perform feature fusion to obtain fusion features; and inputting the fusion features into a segmentation head to obtain a semantic segmentation result of the remote sensing image to be segmented. The method can realize accurate segmentation of the remote sensing image, and balance between the calculation cost and the performance effect.

[0063] In one embodiment, as shown in Figure 3 , the semantic feature enhancement extraction module in step 102 is stacked by several Transformer modules.

[0064] Specifically, as preferred, the semantic feature enhancement extraction module is stacked by three Transformer modules.

[0065] The Transformer module mainly comprises a normalization layer, a multi-head self-attention mechanism, an MLP module and a residual connector.

[0066] In one embodiment, as shown in Figure 4 , the enhanced feature extraction module in step 104 is stacked by several inverted bottleneck modules based on a depth separable convolution.

[0067] Specifically, the enhanced feature extraction module is given a set of feature points , and the backbone network generates a feature map , respectively represent the output features of the first stage, the second stage, the third stage and the fourth stage of the backbone network, wherein has a large receptive field and contains rich semantic features, so we use M Transformer modules to filter and extract key semantic information. has high-resolution features, and The loss of spatial detail information caused by downsampling is crucial for more accurate segmentation results, such as more accurate house edge information and more continuous long-distance river segmentation. However, overemphasis on high-resolution features inevitably leads to an increase in computational cost. Therefore, we use N inverted bottleneck modules based on depth separable convolution to extract enhanced high-resolution features while controlling the computational cost. The formula of the above method is as follows:

[0068] ;

[0069] ;

[0070] wherein, is the output feature of the second stage of the backbone network, is the output feature of the fourth stage of the backbone network, is the output of the enhanced feature extraction module, is the output of the semantic feature enhancement extraction module, is the enhanced feature extraction module, is the semantic feature enhancement extraction module.

[0071] In one embodiment, the feature fusion module includes two 1x1 convolution layers and two up-sampling modules. Step 106 includes: using the first 1x1 convolution layer to change the dimension of the semantic enhancement feature, and using the first up-sampling module to up-sample the obtained feature to obtain an intermediate semantic enhancement feature; using the second 1x1 convolution layer to change the dimension of the high-resolution enhancement feature to obtain an intermediate high-resolution enhancement feature; using the second up-sampling module to up-sample the output feature of the third stage to obtain an up-sampling feature; multiplying the intermediate semantic enhancement feature, the intermediate high-resolution enhancement feature, and the up-sampling feature to obtain a first fusion feature; adding and fusing the first fusion feature, the intermediate semantic enhancement feature, and the intermediate high-resolution enhancement feature to obtain a fusion feature.

[0072] Specifically, after obtaining the enhanced features, since the features are extracted by different feature extractors, the application designs a feature fusion module to fuse these features. First, given the channel dimension of as the standard, use convolution layers to change the dimensions of and to the same as , and use resolution as the benchmark to up-sample and to the same size as . Then , and are multiplied, and rich semantic information and spatial detail information are used to filter The formula expression of the method is as follows:

[0073] ;

[0074] ;

[0075] ;

[0076] wherein, is the fusion feature, is the convolution result of the semantic enhancement feature, is the intermediate high-resolution enhancement feature, is convolution, is up-sampling.

[0077] In one of the embodiments, the segmentation head includes two convolution modules; the convolution module includes a convolution layer, a batch normalization processing layer and a ReLu activation function; and the step 108 includes: inputting the fusion feature into the first convolution module of the segmentation head to obtain a feature with a reduced channel dimension; and inputting the feature with the reduced channel dimension into the second convolution module of the segmentation head to obtain a semantic segmentation result of the remote sensing image to be segmented.

[0078] Specifically, after the feature fusion, the fusion feature captures rich spatial information and semantic information, which is the basis of the segmentation performance. On this basis, a simple segmentation head is added. The segmentation head is composed of two 1x1 layers, and the output result is obtained through smooth channel dimension reduction.

[0079] In one of the embodiments, the context clustering network CoCs in the step 100 includes four stages, each stage including a third down-sampling module and a plurality of clustering blocks; the number of clustering blocks in each stage is distributed as 3, 4, 5 and 2; and the channel dimension of each stage is respectively: 32, 64, 196 and 320.

[0080] Specifically, the backbone network is based on the context clustering network CoCs, and the basic component unit is a clustering block. One clustering block includes two modules, wherein the first module is a region division module, which realizes the region division of the feature map by using a feature similarity matrix, and the second module is a feature aggregation and updating module, which performs the updating calculation of the pixel point features in the divided region.

[0081] The region division module is given a set of feature points wherein n is equal to the total number of pixel points, and d is equal to the feature dimension of the pixel. First, P is projected to To perform similarity calculations, c center points are uniformly extracted in space, and the center features are calculated by averaging the values ​​of their k neighboring points. Then, the similarity is calculated... The cosine similarity matrix is ​​obtained by combining the values ​​of the center point set. Each point is assigned to the most similar center, resulting in c clusters. Each cluster may have a different number of pixels. This similarity-based clustering method, when correctly partitioned, is equivalent to giving each pixel a suitable receptive field, which may be very suitable for semantic segmentation tasks that require dense prediction.

[0082] The feature aggregation and update module performs feature aggregation in each region after region segmentation. Assuming a region has m pixels and a similarity of m / s with the center, the aggregation function is... We map these pixels to a value space. Where w is the number of dimensions. Aggregate features The formula is expressed as follows:

[0083] ;

[0084] ;

[0085] in, As an aggregation feature, As cluster center, Value space The first in i A vector, and It is a learnable scaling scalar. ( ) represents the sigmoid activation function.

[0086] To control the size, the aggregated features are normalized to... C Then, aggregate features. Each point in the region is adaptively assigned based on similarity. In doing so, these points can communicate with each other and share features from all points in the region, for each point... The update method is as follows, where FC is a fully connected layer:

[0087] ;

[0088] according to By calculating the pixels in each region, we obtain a calculation method for a cluster block. The entire backbone network is constructed through downsampling and stacking of cluster blocks.

[0089] It should be understood that, although Figure 1The steps in the flowchart are shown in sequence according to the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order limitation for the execution of the steps, and the steps can be executed in other orders. Moreover, Figure 1 At least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be alternately executed with other steps or at least part of the sub-steps or stages of other steps.

[0090] In a verification embodiment, the data set is selected as the GID-5 data set. The GID-5 data set is a large-scale high-resolution remote sensing image land cover data set based on the Gaofen-2 satellite of China, including five land categories of buildings, farmland, forest, grassland and water area, a total of 150 scene pixel-level labeled Gaofen-2 satellite remote sensing data. The GID-5 data set is widely distributed in land cover information, and the pictures are obtained from more than 60 different cities in the country, and the overall image covers a geographical area of more than 50,000 square kilometers. On the other hand, GID-5 is close to the real ground object distribution characteristics, including a large number of samples in different seasons and different lightings in the same area, which have extremely rich diversity in spectrum, texture and structure, and are very close to the real ground object distribution characteristics. In order to facilitate the experiment, the GID-5 data set is divided into 27,300 1024 1024 resolution pictures, of which 21,840 are training sets and 5,460 are validation sets.

[0091] The experimental environment is shown in Table 1.

[0092] Table 1 Overview of experimental environment

[0093]

[0094] In order to verify the effectiveness of the method of the present application, an exploration verification is carried out on the DIG-5 data set. The model is evaluated from the parameters (Params), the calculation of floating point numbers (FLOPs), the average intersection over union (mIoU) and the delay (Lantency) and other indicators. Compared with the experimental model, we selected the classic model STDC and DeepLabV3+ in the segmentation field, and the lightweight model SeaFormer related to the method of the present application. The experimental results are shown in Table 2.

[0095] Table 2 Overview of experimental results

[0096]

[0097] According to experience, the experimental results in Table 2 prove the effectiveness of HFE-Net in remote sensing semantic segmentation. HFE-Net exceeds DeepLabV3+ with 12.5M parameters by about 0.3% mIoU with 8.3M parameters, reaching 88.6% mIoU. HFE-Net also outperforms STDC and Seaformer on the GID-5 dataset in the comparative network experiment with the same number of parameters, indicating that HFE-Net has found a balance between computational cost and performance.

[0098] The method of the present application uses a large receptive field to capture contextual information, a high-resolution feature to achieve accurate segmentation, a fusion of details and semantic features to accurately predict, and an improved attention mechanism for feature representation. The four guiding principles are achieved by clustering area division to achieve reasonable division of the receptive field, using convolution-based CBlock to obtain enhanced high-resolution features, using Transformer-based TBlock to obtain enhanced global semantic information, and then using a feature fusion module and dimension smoothing to achieve accurate prediction. The segmentation head realizes the construction of our HFE-Net for hybrid feature extraction in the field of remote sensing semantic segmentation. The feasibility and effectiveness of HFE-Net are verified in the experiment, and a balance between computational cost and performance is achieved, with the ability not inferior to classic networks and emerging networks.

[0099] In one embodiment, as shown in Figure 5 a hybrid feature extraction-based remote sensing image semantic segmentation device is provided, comprising: a multi-layer feature extraction module, a semantic feature and detail enhancement feature extraction module, a hybrid feature fusion module, and a semantic segmentation module, wherein:

[0100] The multi-layer feature extraction module is configured to input the acquired remote sensing image to be segmented into a backbone network to obtain output features at each stage. The backbone network is a context clustering network CoCs.

[0101] The semantic feature and detail enhancement feature extraction module is configured to input the output feature of the fourth stage into the semantic feature enhancement extraction module to filter and extract key semantic information using a Transformer module to obtain semantic enhancement features. The output feature of the second stage is input into the enhancement feature extraction module to extract using an inverted bottleneck module based on depth separable convolution to obtain high-resolution enhancement features.

[0102] The hybrid feature fusion module is configured to input the output feature of the third stage, the semantic enhancement feature, and the high-resolution enhancement feature into the feature fusion module for feature fusion to obtain fused features.

[0103] The semantic segmentation module is configured to input the fused feature into a segmentation head to obtain a semantic segmentation result of the remote sensing image to be segmented.

[0104] In one of the embodiments, the semantic feature enhancement extraction module in the semantic feature and detail enhancement feature extraction module is stacked by a plurality of Transformer modules.

[0105] In one of the embodiments, the enhancement feature extraction module in the semantic feature and detail enhancement feature extraction module is stacked by a plurality of inverted bottleneck modules based on depth separable convolution.

[0106] In one of the embodiments, the feature fusion module includes two 1x1 convolution layers and two up-sampling modules; the mixed feature fusion module is further configured to change the dimension of the semantic enhancement feature by using the first 1x1 convolution layer, up-sample the obtained feature by using the first up-sampling module to obtain an intermediate semantic enhancement feature; change the dimension of the high-resolution enhancement feature by using the second 1x1 convolution layer to obtain an intermediate high-resolution enhancement feature; up-sample the output feature of the third stage by using the second up-sampling module to obtain an up-sampled feature; multiply the intermediate semantic enhancement feature, the intermediate high-resolution enhancement feature and the up-sampled feature to obtain a first fused feature; add and fuse the first fused feature, the intermediate semantic enhancement feature and the intermediate high-resolution enhancement feature to obtain the fused feature.

[0107] In one of the embodiments, the segmentation head includes two convolution modules; the convolution module includes a convolution layer, a batch normalization processing layer and a ReLu activation function; the semantic segmentation module is further configured to input the fused feature into the first convolution module of the segmentation head to obtain a feature with reduced channel dimension; input the feature with reduced channel dimension into the second convolution module of the segmentation head to obtain the semantic segmentation result of the remote sensing image to be segmented.

[0108] In one of the embodiments, the context clustering network CoCs in the multi-layer feature extraction module includes four stages, each of which includes a third down-sampling module and a plurality of clustering blocks; the number of clustering blocks in each stage is distributed as 3, 4, 5 and 2; and the channel dimension of each stage is 32, 64, 196 and 320 respectively.

[0109] Specific limitations of the remote sensing image semantic segmentation device based on hybrid feature extraction can be seen from the limitations of the remote sensing image semantic segmentation method based on hybrid feature extraction in the above, which will not be repeated here. Each module in the remote sensing image semantic segmentation device based on hybrid feature extraction can be realized by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor calls and executes the operations of each module.

[0110] In one embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram of the computer device can be as shown in Figure 6 The computer device includes a processor 601, a memory 602, a network interface 603, a display screen 604 and an input device 605 connected through a system bus. The processor 601 of the computer device is configured to provide computing and control capabilities. The memory 602 of the computer device includes a non-volatile storage medium 6022 and an internal memory 6021. The non-volatile storage medium 6022 stores an operating system and a computer program. The internal memory 6021 provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface 603 of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor 601 to implement a bio-medicine knowledge extraction method. The display screen 604 of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device 605 of the computer device can be a touch layer overlaid on the display screen, or can be a key, trackball or touchpad arranged on the shell of the computer device, or can be an external keyboard, touchpad or mouse, etc.

[0111] Those skilled in the art can understand that Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0112] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method in the method embodiment.

[0113] Each technical feature of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0114] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of protection of the present application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A semantic segmentation method for remote sensing images based on hybrid feature extraction, characterized in that, The method includes: The acquired remote sensing image to be segmented is input into the backbone network to obtain the output features at each stage; the backbone network is a context clustering network CoCs. The output features of the fourth stage are input into the semantic feature enhancement and extraction module, where the Transformer module filters and extracts key semantic information to obtain semantic enhancement features; the semantic feature enhancement and extraction module is composed of several stacked Transformer modules. The output features from the second stage are input into the enhanced feature extraction module, where they are extracted using an inverted bottleneck module based on depthwise separable convolution to obtain high-resolution enhanced features. The enhanced feature extraction module is composed of several stacked inverted bottleneck modules based on depthwise separable convolution. The output features of the third stage, the semantic enhancement features, and the high-resolution enhancement features are input into the feature fusion module for feature fusion to obtain fused features; The fused features are input into the segmentation head to obtain the semantic segmentation result of the remote sensing image to be segmented.

2. The method according to claim 1, characterized in that, The feature fusion module includes two 1×1 convolutional layers and two upsampling modules; The output features from the third stage, the semantic enhancement features, and the high-resolution enhancement features are input into the feature fusion module for feature fusion to obtain fused features, including: The semantic enhancement features are subjected to dimensionality transformation using the first 1×1 convolutional layer, and the resulting features are upsampled using the first upsampling module to obtain intermediate semantic enhancement features. The high-resolution enhancement feature is transformed into an intermediate high-resolution enhancement feature by using a second 1×1 convolutional layer to change its dimension. The output features of the third stage are upsampled using the second upsampling module to obtain upsampled features; The intermediate semantic enhancement feature, the intermediate high-resolution enhancement feature, and the upsampled feature are multiplied together to obtain the first fusion feature; The first fusion feature, the intermediate semantic enhancement feature, and the intermediate high-resolution enhancement feature are added and fused together to obtain the fusion feature.

3. The method according to claim 1, characterized in that, The segmentation head includes two convolutional modules; each convolutional module includes a convolutional layer, a batch normalization layer, and a ReLU activation function. The fused features are input into the segmentation head to obtain the semantic segmentation result of the remote sensing image to be segmented, including: The fused features are input into the first convolutional module of the segmentation head to obtain features with reduced channel dimensions; The features with reduced channel dimensions are input into the second convolutional module of the segmentation head to obtain the semantic segmentation result of the remote sensing image to be segmented.

4. The method according to claim 1, characterized in that, Contextual clustering networks (CoCs) consist of four stages, each of which includes a third downsampling module and several clustering blocks. The number of cluster blocks in each stage is distributed as 3, 4, 5, and 2; the channel dimensions in each stage are 32, 64, 196, and 320, respectively.

5. A semantic segmentation device for remote sensing images based on hybrid feature extraction, characterized in that, The device includes: A multi-layer feature extraction module is used to input the acquired remote sensing image to be segmented into the backbone network to obtain the output features at each stage; the backbone network is a context clustering network CoCs. The semantic feature and detail enhancement feature extraction modules are used to input the output features of the fourth stage into the semantic feature enhancement extraction module, where the Transformer module filters and extracts key semantic information to obtain semantic enhancement features; and to input the output features of the second stage into the enhancement feature extraction module, where an inverted bottleneck module based on depthwise separable convolution is used for extraction to obtain high-resolution enhancement features; the semantic feature enhancement extraction module is composed of several stacked Transformer modules; the enhancement feature extraction module is composed of several stacked inverted bottleneck modules based on depthwise separable convolution. The hybrid feature fusion module is used to input the output features of the third stage, the semantic enhancement features, and the high-resolution enhancement features into the feature fusion module for feature fusion to obtain fused features; The semantic segmentation module is used to input the fused features into the segmentation head to obtain the semantic segmentation result of the remote sensing image to be segmented.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

Images