Deep learning based remote sensing image semantic segmentation method
By extracting abstract semantic and edge information features from remote sensing images using deep learning methods and combining them with multi-scale cascaded dilated convolution modules for semantic segmentation, the problem of blurred segmentation boundaries in remote sensing images is solved, and segmentation accuracy is improved.
Patent Information
- Application Number
- CN202510485705.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-17
- Publication Date
- 2025-12-12
- Estimated Expiration
- 2045-04-17
AI Technical Summary
Existing remote sensing image semantic segmentation methods suffer from blurred segmentation boundaries, resulting in low segmentation accuracy.
By using deep learning-based methods to extract features from remote sensing images, abstract semantic information features and edge information features are obtained. Combined with key features in the horizontal and vertical directions, semantic segmentation is performed using multi-scale cascaded dilated convolution modules to improve segmentation accuracy.
It enables more accurate capture of edge details in remote sensing images, improving the accuracy of semantic segmentation.
Smart Images
Figure CN120451530B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, in particular to a remote sensing image semantic segmentation method based on deep learning. BACKGROUND
[0002] Remote sensing semantic segmentation assigns each pixel in a remote sensing image to a semantic class and uses different colors to label the segmentation map, thereby realizing the identification and extraction of different objects and regions on the ground. Remote sensing semantic segmentation has a wide range of applications in many fields, such as urban planning, land use, precision agriculture, environmental monitoring, disaster assessment, etc. Therefore, accurate semantic segmentation of remote sensing images is a key factor in determining the success and impact of earth science work and its applications.
[0003] At present, although the traditional semantic segmentation methods (such as the semantic segmentation method based on full convolution network, the semantic segmentation method based on full connection fusion path, etc.) can effectively capture the semantic information in the remote sensing image, there is a technical problem that the segmentation boundary is blurred, resulting in low segmentation accuracy of the remote sensing image. Therefore, there is an urgent need in the industry for a semantic segmentation method that can accurately extract the boundary of the remote sensing image to improve the segmentation accuracy. SUMMARY
[0004] The main purpose of the present application is to provide a remote sensing image semantic segmentation method based on deep learning, which aims to solve the technical problem that the existing remote sensing image semantic segmentation method has a blurred segmentation boundary, resulting in low segmentation accuracy of the remote sensing image.
[0005] To achieve the above-mentioned purpose, the present application provides a remote sensing image semantic segmentation method based on deep learning, which comprises the following steps:
[0006] performing feature extraction on the target remote sensing image to obtain abstract semantic information features and edge information features;
[0007] determining the first key features corresponding to the target remote sensing image in the horizontal direction and the second key features corresponding to the target remote sensing image in the vertical direction based on the abstract semantic information features and the edge information features;
[0008] performing feature operation on the first key features and the second key features to obtain the output feature map corresponding to the target remote sensing image;
[0009] performing semantic segmentation on the output feature map to obtain the semantic segmentation result corresponding to the target remote sensing image.
[0010] In an embodiment, the step of performing feature extraction on the target remote sensing image to obtain abstract semantic information features and edge information features comprises:
[0011] The target remote sensing image is feature-extracted through a deep convolutional layer to obtain abstract semantic information features, which include class features and attribute features of objects in the target remote sensing image.
[0012] The target remote sensing image is feature-extracted through a shallow convolutional layer and a gated channel attention transformation unit to obtain edge information features, which include contour features and texture features of objects in the target remote sensing image.
[0013] In an embodiment, the step of feature-extracting the target remote sensing image through a shallow convolutional layer and a gated channel attention transformation unit to obtain edge information features includes:
[0014] The target remote sensing image is feature-extracted through a shallow convolutional layer to obtain global features.
[0015] The global features are channel-normalized and adaptively gated through a gated channel attention transformation unit to enhance key feature channels and suppress noise boundaries in the global features, thereby obtaining edge information features.
[0016] In an embodiment, the step of determining the first key features corresponding to the target remote sensing image in a horizontal direction and the second key features corresponding to the target remote sensing image in a vertical direction based on the abstract semantic information features and the edge information features includes:
[0017] The abstract semantic information features and the edge information features are subjected to element-level operations and then depth separable convolution to obtain initial fusion features.
[0018] The initial fusion features are subjected to global average pooling in horizontal and vertical directions, respectively, to obtain the first key features corresponding to the target remote sensing image in a horizontal direction and the second key features corresponding to the target remote sensing image in a vertical direction.
[0019] In an embodiment, the step of performing feature operations on the first key features and the second key features to obtain output feature maps corresponding to the target remote sensing image includes:
[0020] The first key features and the second key features are subjected to two-dimensional convolution, batch normalization and nonlinear activation to obtain third key features and fourth key features.
[0021] The third key features and the fourth key features are subjected to one-dimensional convolution transformation and sigmoid normalization to obtain attention maps corresponding to the target remote sensing image in horizontal and vertical directions.
[0022] The attention map is element-level multiplied with the initial fusion feature to obtain an output feature map corresponding to the target remote sensing image.
[0023] In an embodiment, the step of performing semantic segmentation on the output feature map to obtain a semantic segmentation result corresponding to the target remote sensing image comprises:
[0024] The output feature map is segmented semantically by a multi-scale cascade hollow convolution module to obtain a semantic segmentation result corresponding to the target remote sensing image, and a combination of hollow rates of the multi-scale cascade hollow convolution module is 4, 8, 12 and 16.
[0025] In addition, to achieve the above-mentioned purpose, the present application also proposes a remote sensing image semantic segmentation device based on deep learning, which comprises:
[0026] A feature extraction module is configured to extract features of a target remote sensing image to obtain abstract semantic information features and edge information features.
[0027] A feature fusion module is configured to determine a first key feature corresponding to the target remote sensing image in a horizontal direction and a second key feature corresponding to the target remote sensing image in a vertical direction based on the abstract semantic information features and the edge information features.
[0028] A feature operation module is configured to perform feature operation on the first key feature and the second key feature to obtain an output feature map corresponding to the target remote sensing image.
[0029] A semantic segmentation module is configured to perform semantic segmentation on the output feature map to obtain a semantic segmentation result corresponding to the target remote sensing image.
[0030] In addition, to achieve the above-mentioned purpose, the present application also proposes a remote sensing image semantic segmentation device based on deep learning, which comprises a memory, a processor and a remote sensing image semantic segmentation program based on deep learning stored on the memory and executable on the processor, and the remote sensing image semantic segmentation program based on deep learning is configured to implement the steps of the remote sensing image semantic segmentation method based on deep learning as described above.
[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer readable storage medium, and a remote sensing image semantic segmentation program based on deep learning is stored on the storage medium, and the remote sensing image semantic segmentation program based on deep learning implements the steps of the remote sensing image semantic segmentation method based on deep learning when executed by a processor.
[0032] In addition, to achieve the above object, the application further provides a computer program product, which comprises a deep learning-based remote sensing image semantic segmentation program, and the deep learning-based remote sensing image semantic segmentation program, when executed by a processor, implements the steps of the deep learning-based remote sensing image semantic segmentation method as described above.
[0033] The application extracts features of a target remote sensing image to obtain abstract semantic information features and edge information features, determines first key features corresponding to the target remote sensing image in a horizontal direction and second key features corresponding to the target remote sensing image in a vertical direction based on the abstract semantic information features and the edge information features, performs feature operation on the first key features and the second key features to obtain an output feature map corresponding to the target remote sensing image, and performs semantic segmentation on the output feature map to obtain a semantic segmentation result corresponding to the target remote sensing image. Compared with a traditional remote sensing image semantic segmentation method, since the above method of the application determines the first key features corresponding to the target remote sensing image in the horizontal direction and the second key features corresponding to the target remote sensing image in the vertical direction based on the abstract semantic information features and the edge information features extracted from the target remote sensing image, and then performs semantic segmentation on the output feature map corresponding to the target remote sensing image obtained by performing feature operation based on the first key features and the second key features, the edge details of the target remote sensing image can be more accurately captured from the obtained semantic segmentation result. BRIEF DESCRIPTION OF DRAWINGS
[0034] Figure 1 A structural schematic diagram of a deep learning-based remote sensing image semantic segmentation device related to a hardware running environment of an embodiment scheme of the application;
[0035] Figure 2 A flowchart of a first embodiment of a deep learning-based remote sensing image semantic segmentation method of the application;
[0036] Figure 3 A flowchart of a second embodiment of a deep learning-based remote sensing image semantic segmentation method of the application;
[0037] Figure 4 A flowchart of a third embodiment of a deep learning-based remote sensing image semantic segmentation method of the application;
[0038] Figure 5 A schematic diagram of a multiscale joint-level hole convolution module of a deep learning-based remote sensing image semantic segmentation method of the application;
[0039] Figure 6 A structural block diagram of a first embodiment of a deep learning-based remote sensing image semantic segmentation device of the application.
[0040] The implementation, functional features and advantages of the present application will be further illustrated in conjunction with the embodiments and the accompanying drawings. DETAILED DESCRIPTION
[0041] It should be understood that the specific embodiments described herein are merely illustrative of the present application and do not limit the present application.
[0042] Reference Figure 1 , Figure 1 The hardware environment of the embodiment of the present application involves a deep learning-based remote sensing image semantic segmentation device structure diagram.
[0043] As Figure 1 shown, the deep learning-based remote sensing image semantic segmentation device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display screen, an input unit such as a keyboard, and an optional user interface 1003 can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (Wi-Fi) interface). The memory 1005 can be a high-speed random access memory (RAM), and can also be a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 can also be an independent storage device from the aforementioned processor 1001.
[0044] Those skilled in the art can understand Figure 1 that the structure shown in the figure does not constitute a limitation on the deep learning-based remote sensing image semantic segmentation device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.
[0045] As Figure 1 shown, the memory 1005 as a storage medium can include an operating system, a network communication module, a user interface module, and a deep learning-based remote sensing image semantic segmentation program.
[0046] In Figure 1In the illustrated deep learning-based remote sensing image semantic segmentation device, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the deep learning-based remote sensing image semantic segmentation device of the present application can be arranged in the deep learning-based remote sensing image semantic segmentation device, and the deep learning-based remote sensing image semantic segmentation device calls the deep learning-based remote sensing image semantic segmentation program stored in the memory 1005 through the processor 1001, and executes the deep learning-based remote sensing image semantic segmentation method provided in the embodiments of the present application.
[0047] The embodiments of the present application provide a deep learning-based remote sensing image semantic segmentation method, which refers to Figure 2 , Figure 2 The flowchart of the first embodiment of the deep learning-based remote sensing image semantic segmentation method of the present application is shown in the figure.
[0048] In the present embodiment, the deep learning-based remote sensing image semantic segmentation method comprises the following steps:
[0049] Step S10: performing feature extraction on the target remote sensing image to obtain abstract semantic information features and edge information features.
[0050] It should be noted that the execution subject of the method of the present embodiment can be a terminal device with feature extraction, data processing and program running functions, such as a smart phone, a computer, etc., or an electronic device with the same or similar functions, such as the above-mentioned deep learning-based remote sensing image semantic segmentation device. The deep learning-based remote sensing image semantic segmentation device will be taken as an example to describe the present embodiment and the following embodiments.
[0051] It should also be noted that the above deep learning-based remote sensing image semantic segmentation device contains an improved UnetFormer model. The UNetFormer model is a model for efficient semantic segmentation that combines the characteristics of UNet and Transformer. The model consists of two main parts: an encoder based on a convolutional neural network and a decoder based on a Transformer. The encoder uses a lightweight ResNet18 as a feature extractor. ResNet18 is a classic convolutional neural network structure with fewer parameter quantities and computational complexity, but it can still effectively extract image features. The decoder uses three global-local attention blocks to capture global and local information. Global-local attention blocks allow the model to focus on both the entire image and local details to obtain more comprehensive semantic information. In addition, the decoder also includes a feature refinement head to enhance channel and spatial feature representation. Although the UNetFormer model can effectively capture semantic information in images, its segmentation results cannot completely fit the shape of objects, resulting in blurred segmentation boundaries. Therefore, this embodiment will improve the UNetFormer model to more accurately capture edge details of the image when performing semantic segmentation on the target remote sensing image.
[0052] It can be understood that the above target remote sensing image refers to image information of the earth's surface and the atmosphere obtained by satellites, aircraft or other sensors. The target remote sensing image can be collected, detected and recorded based on sensors, which are usually mounted on platforms such as satellites and aircraft, and can remotely and non-contact detect the electromagnetic wave radiation and reflection characteristics of objects on the earth's surface. The target remote sensing image can record various information of the target object, such as shape, size, color, texture, etc., which can show the overall characteristics, distribution rules and other attribute information of the target object, and is therefore widely used in environmental monitoring, urban planning, agricultural management and other fields.
[0053] It should be understood that the above abstract semantic information feature refers to information in the target remote sensing image that can express the high-level meaning of the object or scene. These high-level meaning information is usually closely related to human understanding of images, such as object class features (including forests, mountains, farmland, etc.), attribute features (including building density, green space ratio, canopy density, etc.) in the target remote sensing image, etc. The above edge information feature refers to the features possessed by the edge part of the object or scene in the target remote sensing image. These features are usually related to the shape, contour and boundary of the object and are an important basis for image segmentation, target recognition and scene understanding tasks, such as contour features, texture features, etc. in the target remote sensing image.
[0054] Step S20: determining a first key feature corresponding to the target remote sensing image in a horizontal direction and a second key feature corresponding to the target remote sensing image in a vertical direction based on the abstract semantic information feature and the edge information feature.
[0055] It should be noted that the first key feature refers to a feature that the target remote sensing image contains edge details in the horizontal direction, and the second key feature refers to a feature that the target remote sensing image contains edge details in the vertical direction.
[0056] It should be understood that the first key feature corresponding to the target remote sensing image in the horizontal direction and the second key feature corresponding to the target remote sensing image in the vertical direction can be obtained by performing element operations (such as convolution, normalization, etc.) on the abstract semantic information feature and the edge information feature.
[0057] Step S30: performing feature operation on the first key feature and the second key feature to obtain an output feature map corresponding to the target remote sensing image.
[0058] In a specific implementation, the output feature map corresponding to the target remote sensing image can be obtained by performing feature operation including but not limited to convolution, normalization, nonlinear activation, etc. on the first key feature and the second key feature. In particular, the output feature map can contain object categories (such as forest, mountain, farmland, etc.), object attributes (such as building density, green rate, canopy density, etc.), object contour, object texture, etc. in the real scene corresponding to the target remote sensing image, which is not limited in this embodiment.
[0059] Step S40: performing semantic segmentation on the output feature map to obtain a semantic segmentation result corresponding to the target remote sensing image.
[0060] In a specific implementation, the semantic segmentation result corresponding to the target remote sensing image can be obtained by performing semantic segmentation on the output feature map through an encoder-decoder structure. More specifically, the encoder-decoder structure is a basic framework in image semantic segmentation tasks. The encoder performs dimension reduction and feature extraction on the input image through convolution operation, while the decoder is responsible for mapping these features back to the original resolution to generate pixel-level prediction results. This structure reduces the computational complexity and the number of parameters while achieving efficient and accurate image semantic segmentation. It should be noted that the method of performing semantic segmentation on the output feature map through the encoder-decoder structure in this embodiment is only used for illustration and not for limitation, and other methods that can perform semantic segmentation on the feature map are also applicable to this embodiment, which will not be described here.
[0061] The embodiment extracts features of a target remote sensing image to obtain abstract semantic information features and edge information features; determines first key features corresponding to the target remote sensing image in a horizontal direction and second key features corresponding to the target remote sensing image in a vertical direction based on the abstract semantic information features and the edge information features; performs feature operation on the first key features and the second key features to obtain an output feature map corresponding to the target remote sensing image; and performs semantic segmentation on the output feature map to obtain a semantic segmentation result corresponding to the target remote sensing image. Compared with a traditional remote sensing image semantic segmentation method, the above method of the embodiment determines first key features corresponding to a target remote sensing image in a horizontal direction and second key features corresponding to the target remote sensing image in a vertical direction based on abstract semantic information features and edge information features extracted from the target remote sensing image, and then performs semantic segmentation on an output feature map corresponding to the target remote sensing image obtained by performing feature operation based on the first key features and the second key features, so that edge details of the target remote sensing image can be more accurately captured from the obtained semantic segmentation result.
[0062] Reference Figure 3 , Figure 3 FIG. 2 is a flowchart of a second embodiment of a remote sensing image semantic segmentation method based on deep learning according to the present application.
[0063] In a feasible implementation, the step S10 can include:
[0064] Step S101: performing feature extraction on a target remote sensing image by a deep convolutional layer to obtain abstract semantic information features, the abstract semantic information features including class features and attribute features of objects in the target remote sensing image.
[0065] Step S102: performing feature extraction on the target remote sensing image by a shallow convolutional layer and a gated channel attention transformation unit to obtain edge information features, the edge information features including contour features and texture features of objects in the target remote sensing image.
[0066] In a specific implementation, the embodiment introduces a boundary awareness module (BAM) to extract the edge information of the target. However, due to the noise in the low-level features, a gating channel attention transformation unit (GCT) can be introduced to enhance important feature channels and suppress noise boundaries. GCT mainly includes three parts: global context embedding, channel normalization, and gating adaptation. GCT captures the global features of each channel by embedding global context information, thereby avoiding local semantic ambiguity and enabling shallow features to better capture general attributes. Through normalization operations, GCT can establish a competitive relationship between channels, making channels with larger responses stronger while suppressing channels with smaller feedback. This normalization method establishes a competitive relationship between neurons or channels, thereby improving model performance. GCT can also achieve channel selection through gating weights and biases: when the gating weight of a channel is actively activated, GCT promotes the channel to compete with other channels; when the gating weight is negatively activated, GCT encourages the channel to cooperate with other channels. This mechanism effectively enhances important feature (i.e., edge information feature) channels and suppresses noise boundaries.
[0067] In a feasible implementation, the step S102 can include:
[0068] Step S1021: performing feature extraction on the target remote sensing image through a shallow convolutional layer to obtain global features.
[0069] Step S1022: performing channel normalization and gating adaptation on the global features through a gating channel attention transformation unit to enhance key feature channels in the global features and suppress noise boundaries, thereby obtaining edge information features.
[0070] In a specific implementation, the global features can be input into the gating channel attention transformation unit, which adaptively adjusts the weights of the channels to enhance important feature channels, suppress noise boundaries, and enhance context relevance. Then, the features with adjusted weights are fused through two 3x3 convolution operations. Finally, the fused features are normalized through a 1x1 convolution and a sigmoid function, thereby obtaining the edge information features. In this way, the boundary awareness module can extract features with edge information from different levels of the encoder and enhance important features through GCT to obtain more accurate boundary predictions, which helps to improve the accuracy of remote sensing image segmentation tasks.
[0071] The embodiment extracts features of a target remote sensing image through a deep convolutional layer to obtain abstract semantic information features, the abstract semantic information features including class features and attribute features of objects in the target remote sensing image; extracts features of the target remote sensing image through a shallow convolutional layer to obtain global features; performs channel normalization and gated adaptation on the global features through a gated channel attention transformation unit to enhance key feature channels in the global features and suppress noise boundaries to obtain edge information features, the edge information features including contour features and texture features of objects in the target remote sensing image. Compared with a traditional remote sensing image semantic segmentation method, the above method of the embodiment extracts key features in global features from a target remote sensing image through a boundary perception module, and enhances the key features through a gated channel attention transformation unit to obtain more accurate boundary prediction, which helps to improve the semantic segmentation precision of the target remote sensing image.
[0072] Reference Figure 4 , Figure 4 FIG. 1 is a flowchart of a remote sensing image semantic segmentation method based on deep learning according to a third embodiment of the present application.
[0073] In a feasible implementation, the step S20 can include:
[0074] Step S201: performing element-level operation on the abstract semantic information features and the edge information features, and then performing depth separable convolution to obtain initial fusion features.
[0075] In a specific implementation, the abstract semantic information features can be down-sampled and multiplied with the edge information features at an element level to obtain a product, and the product is connected through a jump connection and 3x3 depth separable convolution to obtain the initial fusion features.
[0076] Step S202: performing global average pooling in horizontal and vertical directions on the initial fusion features to obtain first key features corresponding to the target remote sensing image in the horizontal direction and second key features corresponding to the target remote sensing image in the vertical direction.
[0077] In a specific implementation, the embodiment proposes an improved Boundary-guided Fusion Module (BFM), which realizes more effective fusion of the abstract semantic information features and the edge information features by adopting a depth separable convolution and a spatial information encoding improvement strategy. The embodiment adopts a 3x3 depth separable convolution to replace the traditional 3x3 convolution. The depth separable convolution decomposes the convolution into a depth convolution (Depthwise Convolution) and a pointwise convolution (Pointwise Convolution), which significantly reduces the computational complexity and the number of parameters while maintaining the feature extraction capability. In order to more effectively fuse the abstract semantic information features and the edge information features, the BFM introduces a Coordinate Attention (a kind of attention mechanism for computer vision tasks) to decompose the channel attention into two one-dimensional feature encoding processes. The Coordinate Attention can aggregate features along the horizontal and vertical directions respectively. This method not only preserves the inter-channel dependency, but also embeds the position information, so that the model can more accurately capture the position information of the target region. Specifically, the Coordinate Attention can realize its function through two steps: first, it performs global information aggregation along one direction (such as horizontal or vertical), and then performs the same operation in the other direction. This two-way aggregation process enables the model to capture more rich spatial information and long-distance dependencies. In order to further improve the ability of the BFM in edge feature extraction, the embodiment also proposes a spatial information encoding improvement strategy. This strategy introduces an additional spatial encoding layer to further encode and represent the spatial dimensions of the input features, so that the model can better understand the spatial relationships between different positions in the image. This explicit encoding of spatial information helps the model to more accurately locate the edge position and retain more edge detail information when fusing abstract semantic information features and edge information features.
[0078] In a feasible implementation, the step S30 can include:
[0079] Step S301: After two-dimensional convolution, batch normalization and nonlinear activation are performed on the first key feature and the second key feature, third key features and fourth key features are obtained.
[0080] Step S302: After one-dimensional convolution transformation and sigmoid normalization are performed on the third key feature and the fourth key feature, the attention maps corresponding to the target remote sensing image in the horizontal direction and the vertical direction are obtained.
[0081] Step S303: Element-level multiplication of the attention map and the initial fusion feature to obtain an output feature map corresponding to the target remote sensing image.
[0082] In a specific implementation, the first key feature and the second key feature can be subjected to two-dimensional convolution, batch normalization, and nonlinear activation to obtain a third key feature and a fourth key feature. Then, the third key feature and the fourth key feature are subjected to two-dimensional convolution transformation and sigmoid normalization to generate an attention map corresponding to the target remote sensing image in the horizontal direction and the vertical direction. Finally, element-level multiplication of the attention map and the initial fusion feature is performed to obtain an output feature map corresponding to the target remote sensing image.
[0083] In a feasible implementation, the step S40 can include:
[0084] Step S401: semantic segmentation of the output feature map by a multi-scale cascaded atrous spatial pyramid pooling module to obtain a semantic segmentation result corresponding to the target remote sensing image, wherein a combination of atrous rates of the multi-scale cascaded atrous spatial pyramid pooling module is 4, 8, 12, and 16.
[0085] In a specific implementation, in a conventional DeepLabv3+ (a kind of deep learning model), an atrous spatial pyramid pooling (ASPP) module uses an atrous convolution with a combination of atrous rates of 6, 12, and 18. However, as the backbone network extracts features, the resolution of the feature map gradually decreases, which causes the combination of 6, 12, and 18 to be unable to effectively extract features of multi-resolution feature maps. In addition, because a smaller atrous rate is not set, the network has poor ability to segment small targets, which causes the network to perform poorly when processing segmentation targets of different sizes. In order to more effectively extract features of multi-resolution feature maps and improve the segmentation ability for segmentation targets of different sizes, the embodiment changes the combination of atrous rates of the atrous convolution to 4, 8, 12, and 16. In order to reduce the calculation cost, the embodiment replaces the standard convolution with a depthwise separable atrous convolution. In order to further improve the utilization efficiency of the features, multi-level fusion feature information is obtained by fusion, that is, the input feature is spliced with the multi-level fusion feature information after being subjected to pooling. The improved ASPP is called a multi-scale cascaded atrous spatial pyramid pooling (MSC-ASPP) module, as shown in FIG. 4. Figure 5 Figure 5 In the formula, DCconv represents a depth separable dilated convolution, GConv represents a group convolution, Image Pooling represents image pooling, "1x1" represents a convolution kernel size of 1x1, "3x3" represents a convolution kernel size of 3x3, r represents a dilation rate, and add represents an element-wise addition operation.
[0086] The embodiment performs a depth separable convolution on the abstract semantic information features and the edge information features after element-level operation, to obtain initial fusion features; performs global average pooling in horizontal and vertical directions on the initial fusion features, to obtain first key features corresponding to the target remote sensing image in the horizontal direction and second key features corresponding to the target remote sensing image in the vertical direction; performs two-dimensional convolution, batch normalization and nonlinear activation on the first key features and the second key features, to obtain third key features and fourth key features; performs one-dimensional convolution transformation and sigmoid normalization on the third key features and the fourth key features, to obtain attention maps corresponding to the target remote sensing image in the horizontal and vertical directions; and performs element-level multiplication on the attention maps and the initial fusion features, to obtain an output feature map corresponding to the target remote sensing image. The output feature map is subjected to semantic segmentation through a multi-scale cascade dilated convolution module, to obtain a semantic segmentation result corresponding to the target remote sensing image, and the dilation rate combination of the multi-scale cascade dilated convolution module is 4, 8, 12 and 16. Compared with a traditional remote sensing image semantic segmentation method, the above method of the embodiment realizes more effective fusion of abstract semantic information features and edge information features through the edge-guided aggregation module, so that the edge position in the target remote sensing image can be more accurately located to retain more edge detail information; meanwhile, the output feature map is subjected to semantic segmentation through the multi-scale cascade dilated convolution module with a dilation rate combination of 4, 8, 12 and 16, so that the features of multi-resolution feature maps in the target remote sensing image can be more effectively extracted, and the segmentation capability for different sizes of segmentation targets can be improved.
[0087] In addition, the embodiment of the present application further provides a storage medium, wherein the storage medium stores a remote sensing image semantic segmentation program based on deep learning, and the remote sensing image semantic segmentation program based on deep learning realizes the steps of the remote sensing image semantic segmentation method based on deep learning when executed by a processor.
[0088] Reference Figure 6 , Figure 6 FIG. 1 is a structural block diagram of a remote sensing image semantic segmentation device based on deep learning according to a first embodiment of the present application.
[0089] As shown in FIG. 2, the remote sensing image semantic segmentation device based on deep learning according to the first embodiment of the present application comprises: Figure 6
[0090] The feature extraction module 601 is configured to perform feature extraction on the target remote sensing image to obtain abstract semantic information features and edge information features.
[0091] The feature fusion module 602 is configured to determine, based on the abstract semantic information features and the edge information features, a first key feature corresponding to the target remote sensing image in a horizontal direction and a second key feature corresponding to the target remote sensing image in a vertical direction.
[0092] The feature operation module 603 is configured to perform feature operation on the first key feature and the second key feature to obtain an output feature map corresponding to the target remote sensing image.
[0093] The semantic segmentation module 604 is configured to perform semantic segmentation on the output feature map to obtain a semantic segmentation result corresponding to the target remote sensing image.
[0094] The embodiment performs feature extraction on a target remote sensing image to obtain abstract semantic information features and edge information features, determines, based on the abstract semantic information features and the edge information features, a first key feature corresponding to the target remote sensing image in a horizontal direction and a second key feature corresponding to the target remote sensing image in a vertical direction, performs feature operation on the first key feature and the second key feature to obtain an output feature map corresponding to the target remote sensing image, and performs semantic segmentation on the output feature map to obtain a semantic segmentation result corresponding to the target remote sensing image. Compared with a traditional remote sensing image semantic segmentation method, since the above method of the embodiment determines, based on abstract semantic information features and edge information features extracted from a target remote sensing image, a first key feature corresponding to the target remote sensing image in a horizontal direction and a second key feature corresponding to the target remote sensing image in a vertical direction, and then performs semantic segmentation on an output feature map corresponding to the target remote sensing image obtained by performing feature operation based on the first key feature and the second key feature, the edge details of the target remote sensing image can be more accurately captured from the obtained semantic segmentation result.
[0095] Based on the first embodiment of the remote sensing image semantic segmentation device based on deep learning provided in the present application, the second embodiment of the remote sensing image semantic segmentation device based on deep learning provided in the present application is proposed.
[0096] In the embodiment, the feature extraction module 601 is further configured to perform feature extraction on the target remote sensing image through a deep convolutional layer to obtain abstract semantic information features, and the abstract semantic information features include class features and attribute features of objects in the target remote sensing image; and perform feature extraction on the target remote sensing image through a shallow convolutional layer and a gated channel attention transformation unit to obtain edge information features, and the edge information features include contour features and texture features of objects in the target remote sensing image.
[0097] Further, the feature extraction module 601 is further configured to perform feature extraction on the target remote sensing image through a shallow convolutional layer to obtain global features, and perform channel normalization and gated adaptation on the global features through a gated channel attention transformation unit to enhance key feature channels in the global features and suppress noise boundaries to obtain edge information features.
[0098] Further, the feature fusion module 602 is further configured to perform element-level operation on the abstract semantic information features and the edge information features, and then perform deep separable convolution to obtain initial fusion features, and perform global average pooling in horizontal and vertical directions on the initial fusion features to obtain first key features corresponding to the target remote sensing image in the horizontal direction and second key features corresponding to the target remote sensing image in the vertical direction.
[0099] Further, the feature operation module 603 is further configured to perform two-dimensional convolution, batch normalization and nonlinear activation on the first key features and the second key features to obtain third key features and fourth key features, perform one-dimensional convolution transformation and sigmoid normalization on the third key features and the fourth key features to obtain attention maps corresponding to the target remote sensing image in the horizontal and vertical directions, and perform element-level multiplication on the attention maps and the initial fusion features to obtain an output feature map corresponding to the target remote sensing image.
[0100] Further, the semantic segmentation module 604 is further configured to perform semantic segmentation on the output feature map through a multi-scale joint-level hollow convolution module to obtain a semantic segmentation result corresponding to the target remote sensing image, and a hollow rate combination of the multi-scale joint-level hollow convolution module is 4, 8, 12 and 16.
[0101] Other embodiments or specific implementations of the remote sensing image semantic segmentation device based on deep learning provided in the present application can refer to the above-mentioned method embodiments, and will not be described here.
[0102] It should be noted that in this paper, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the existence of another identical element in the process, method, article or system including the element.
[0103] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.
[0104] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, can also be through hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art contribution can be embodied in the form of software products, the computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disc), including a number of instructions to make a terminal device (may be a mobile phone, computer, server, or network equipment, etc.) executes the method described in various embodiments of the present application.
[0105] The above is only the preferred embodiment of the present application, not therefore limit the patent scope of the present application, all the equivalent structure or equivalent process transformation, or directly or indirectly applied in other related technical fields using the contents of the present application specification and drawings, are also included in the patent protection scope of the present application.
Claims
1. A semantic segmentation method for remote sensing images based on deep learning, characterized in that, The method includes the following steps: Feature extraction is performed on the remote sensing image of the target to obtain abstract semantic information features and edge information features; Based on the abstract semantic information features and the edge information features, determine the first key feature corresponding to the target remote sensing image in the horizontal direction and the second key feature corresponding to the target image in the vertical direction; Perform feature operations on the first key feature and the second key feature to obtain the output feature map corresponding to the remote sensing image of the target; Semantic segmentation is performed on the output feature map to obtain the semantic segmentation result corresponding to the target remote sensing image; The step of extracting features from the target remote sensing image to obtain abstract semantic information features and edge information features includes: Feature extraction is performed on the target remote sensing image through deep convolutional layers to obtain abstract semantic information features, which include the category features and attribute features of objects in the target remote sensing image; Global features are obtained by extracting features from the target remote sensing image through shallow convolutional layers. The global features are normalized and gating adaptively through a gated channel attention conversion unit to enhance key feature channels and suppress noise boundaries, thereby obtaining edge information features, which include the contour features and texture features of objects in the target remote sensing image.
2. The deep learning-based remote sensing image semantic segmentation method as described in claim 1, characterized in that, The step of determining the first key feature in the horizontal direction and the second key feature in the vertical direction of the target remote sensing image based on the abstract semantic information features and the edge information features includes: After performing element-level operations on the abstract semantic information features and the edge information features, a depthwise separable convolution is performed to obtain the initial fused features; The initial fused features are subjected to global average pooling in both the horizontal and vertical directions to obtain the first key feature of the target remote sensing image in the horizontal direction and the second key feature in the vertical direction.
3. The deep learning-based remote sensing image semantic segmentation method as described in claim 2, characterized in that, The step of performing feature operations on the first key feature and the second key feature to obtain the output feature map corresponding to the target remote sensing image includes: After performing two-dimensional convolution, batch normalization, and nonlinear activation on the first and second key features, the third and fourth key features are obtained. One-dimensional convolutional transformation and sigmoid normalization are performed on the third key feature and the fourth key feature to obtain the attention maps corresponding to the target remote sensing image in the horizontal and vertical directions. The attention map is element-wise multiplied with the initial fusion feature to obtain the output feature map corresponding to the target remote sensing image.
4. The deep learning-based semantic segmentation method for remote sensing images as described in claim 1, characterized in that, The step of performing semantic segmentation on the output feature map to obtain the semantic segmentation result corresponding to the target remote sensing image includes: The output feature map is semantically segmented by a multi-scale cascaded dilated convolution module to obtain the semantic segmentation result corresponding to the target remote sensing image. The multi-scale cascaded dilated convolution module is a module in which the dilation rate combination of the dilated spatial convolution pyramid pooling module is changed to 4, 8, 12, and 16.
5. A remote sensing image semantic segmentation device based on deep learning, characterized in that, The deep learning-based remote sensing image semantic segmentation device includes: The feature extraction module is used to extract features from the target remote sensing image to obtain abstract semantic information features and edge information features; The feature fusion module is used to determine, based on the abstract semantic information features and the edge information features, the first key feature corresponding to the target remote sensing image in the horizontal direction and the second key feature corresponding to the vertical direction; The feature processing module is used to perform feature processing on the first key feature and the second key feature to obtain the output feature map corresponding to the target remote sensing image; The semantic segmentation module is used to perform semantic segmentation on the output feature map to obtain the semantic segmentation result corresponding to the target remote sensing image; The feature extraction module is also used for: Feature extraction is performed on the target remote sensing image through deep convolutional layers to obtain abstract semantic information features, which include the category features and attribute features of objects in the target remote sensing image; Global features are obtained by extracting features from the target remote sensing image through shallow convolutional layers. The global features are normalized and gating adaptively through a gated channel attention conversion unit to enhance key feature channels and suppress noise boundaries, thereby obtaining edge information features, which include the contour features and texture features of objects in the target remote sensing image.
6. A remote sensing image semantic segmentation device based on deep learning, characterized in that, The device includes: a memory, a processor, and a deep learning-based remote sensing image semantic segmentation program stored in the memory and executable on the processor, the deep learning-based remote sensing image semantic segmentation program being configured to implement the steps of the deep learning-based remote sensing image semantic segmentation method as described in any one of claims 1 to 4.
7. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and the storage medium stores a deep learning-based remote sensing image semantic segmentation program. When the deep learning-based remote sensing image semantic segmentation program is executed by a processor, it implements the steps of the deep learning-based remote sensing image semantic segmentation method as described in any one of claims 1 to 4.
8. A computer program product, characterized in that, The computer program product includes a deep learning-based remote sensing image semantic segmentation program, which, when executed by a processor, implements the steps of the deep learning-based remote sensing image semantic segmentation method as described in any one of claims 1 to 4.