Clustering-driven panoramic segmentation method for optical remote sensing images

Abstract

The present invention relates to a clustering-driven optical remote sensing image panoramic segmentation method, belonging to the technical field of computer vision and remote sensing image processing. The method of the present invention preprocesses optical remote sensing images and remote sensing image panoramic segmentation labels, constructs a dual-branch transformer encoder-decoder network, and uses the preprocessed optical remote sensing images and remote sensing image labels to train and learn the dual-branch transformer encoder-decoder network to achieve panoramic segmentation of the optical remote sensing images. The method of the present invention can automatically generate optical remote sensing image segmentation masks of high-quality typical targets, solving the technical problems of target dispersion and insufficient object focus in current remote sensing image intelligent interpretation tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and remote sensing image processing, and in particular to a panoramic segmentation algorithm for optical remote sensing images. Background Art

[0002] In recent years, with the advancement of remote sensing technology, the spatial resolution of remote sensing images has gradually increased, allowing images to capture and express richer and more diverse ground features and hierarchical structures. Accurately and effectively extracting all the ground features within a specific scene and forming a comprehensive understanding of the scene are key steps in the successful application of remote sensing technology. Currently, the efficiency of remote sensing image recognition lags far behind the growth rate of image data and is increasingly worsening, resulting in a massive waste of data resources and a "big data, little knowledge" dilemma. The gap between the surge in satellite remote sensing data, the inadequacy of remote sensing data analysis capabilities, and the bottlenecks of traditional remote sensing technology urgently requires a new, efficient, accurate, and convenient intelligent interpretation method.

[0003] Although deep learning methods have made significant progress in specific target detection and recognition, semantic segmentation of objects, instance segmentation, and scene classification, they currently lack the comprehensive understanding and global cognition capabilities for complex scenes due to their discrete and unfocused nature. This makes it difficult to acquire the global knowledge needed to unlock the hidden value of remote sensing big data. For example, object detection and semantic instance segmentation methods can only interpret foreground objects in remote sensing images, failing to understand background information. Semantic segmentation algorithms, on the other hand, can only assign a category to each pixel in an image but fail to distinguish between individuals within the same category. Therefore, the ability to simultaneously identify irregular background objects and key objects in remote sensing images, achieving comprehensive scene interpretation and cognition, is essential for transforming remote sensing data into knowledge. Panoptic segmentation of remote sensing images is an advanced image processing algorithm and a key task in remote sensing scene understanding and cognition. It combines the strengths of semantic segmentation and instance segmentation to simultaneously classify foreground and background objects and distinguish between similar individuals. It has important applications in transportation hub monitoring, battlefield environment assessment, and national defense security.

[0004] Clustering is the process of dividing a set of data objects into several groups or clusters, so that objects within the same cluster have high similarity, while objects in different clusters have low similarity. Common clustering methods include K-means clustering and hierarchical clustering. The similarity measure between objects obtained by clustering methods is helpful for image classification and segmentation. Summary of the Invention

[0005] In order to solve the technical problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a clustering-driven optical remote sensing image panoramic segmentation method, which can realize automatic panoramic segmentation of optical remote sensing images.

[0006] To achieve the above-mentioned object, in a first aspect, the present invention provides a cluster-driven optical remote sensing image panoptic segmentation method, comprising:

[0007] Step S1, preprocessing the optical remote sensing images and remote sensing image panoptic segmentation labels to construct an optical remote sensing image panoptic segmentation training dataset;

[0008] Step S2, constructing a model based on a dual-branch transformer encoder-decoder network, the model including a feature encoder branch and a cluster decoder branch; the feature encoder branch is used to implement feature extraction and enhancement of optical remote sensing images; the cluster decoder branch is used to implement optical remote sensing image feature decoding and mask category prediction;

[0009] Step S3, using the preprocessed optical remote sensing image and remote sensing image label to train the model based on the dual-branch transformer encoder-decoder network to obtain a trained model;

[0010] Step S4: Automatically perform panoramic segmentation on the optical remote sensing image based on the trained model.

[0011] Furthermore, in step S1, the optical remote sensing image and the remote sensing image panoptic segmentation label are preprocessed, including:

[0012] The optical remote sensing images and their labels were cropped with a cropping step of 600 pixels to crop the original images to a standard size of 800 × 800 pixels;

[0013] For the cropped remote sensing images and labels, we strictly follow the COCO data format standard to construct a panoramic segmentation training dataset.

[0014] Furthermore, the feature encoder branch includes a feature extraction backbone network and a feature enhancement pixel encoder module;

[0015] The feature extraction backbone network adopts a backbone network residual neural network ResNets, which is composed of convolutional layers, maximum pooling layers and multi-layer residual units. Each residual unit contains multiple convolutional layers and directly connects the input to the output through skip connections. The feature extraction backbone network is used to learn and extract the feature information of various types of objects in the optical remote sensing image;

[0016] The feature enhancement pixel encoder module consists of an axial attention module, multiple residual units and a feature aggregation module; the feature enhancement pixel encoder module is used to restore the feature map resolution and integrate multi-scale and multi-level features.

[0017] Furthermore, the feature aggregation module is composed of a bilinear interpolation operation and a morphological perception module; the feature aggregation module is used to fuse features of different resolutions to balance high and low resolution information;

[0018] The morphological perception module consists of a spatial attention layer, a channel attention layer, and a 7×7 convolutional layer to effectively capture multi-scale features;

[0019] The spatial attention layer includes an average pooling layer, a maximum pooling layer and a 7×7 convolutional layer;

[0020] The channel attention layer is composed of a maximum pooling layer and two 1×1 convolutional layers. The output results of the attention features are summed up and concatenated with the initial feature C in the channel dimension. Finally, the feature weights are obtained after being processed by the 7×7 convolutional layer and the sigmoid activation function.

[0021] The axial attention module consists of self-attention calculation modules in the horizontal and vertical directions.

[0022] Furthermore, the cluster decoder branch consists of 6 layers of cluster decoder modules and a layer of loss function;

[0023] The cluster decoding module consists of a cluster cross attention module, a multi-head self-attention module and a feedforward module, which is used to extract contextual semantic information and perform mask category prediction;

[0024] The cluster cross attention module consists of a cluster cross attention layer, a multi-head self-attention layer and a feedforward network layer; the cluster cross attention module is used to classify features, thereby determining the feature category and ultimately generating mask features;

[0025] The clustering cross attention layer consists of three linear layers and one argmax operation, which is used to accurately determine the feature category.

[0026] Furthermore, step S3 further includes:

[0027] Step S31, inputting the pre-processed optical remote sensing image set and remote sensing image labels into the feature encoder branch to obtain an enhanced pixel feature map;

[0028] Step S32: input the enhanced pixel feature map into a cluster decoder branch to obtain a mask feature map, and further aggregate the pixel feature map and the mask feature map to generate a panoptic segmentation mask;

[0029] In step S33, the entire two-branch Transformer encoder-decoder network is optimized by backpropagation based on the panoptic segmentation mask and the input pre-processed panoptic segmentation labels to minimize the loss function, thereby obtaining a trained model. The minimum loss function is a cross entropy loss function and a Dice loss function.

[0030] In a second aspect of the present invention, a system for implementing any one of the methods described in the first aspect is provided, characterized in that it comprises:

[0031] An input module, used for inputting optical remote sensing images to be segmented;

[0032] A panoptic segmentation module is configured to segment the input optical remote sensing image; the panoptic segmentation module comprises a feature encoder branch and a cluster decoder branch, wherein the feature encoder branch is configured to obtain an enhanced pixel feature map based on the optical remote sensing image to be segmented; the cluster decoder branch is configured to obtain a mask feature map based on the enhanced pixel feature map, and further aggregate the pixel feature map with the mask feature map to generate a panoptic segmentation mask to obtain a segmented image;

[0033] The output module is used to output the segmented image.

[0034] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and program instructions stored in the memory for execution by the processor, wherein the processor executes the program instructions to implement the steps of any one of the methods described in the first aspect.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, characterized in that the computer program is executed by a processor according to any one of the methods in the first aspect.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The solution of an embodiment of the present invention achieves panoramic segmentation of optical remote sensing images by designing a cluster-driven dual-branch transformer encoder-decoder network. The network consists of a feature encoder (including a feature extraction backbone network and a feature enhancement pixel encoder module) and a cluster decoder. The training data includes two sets: optical remote sensing images and optical remote sensing image annotations. The feature extraction backbone network integrates and extracts salient information by learning the shape, color, texture, and other features of various objects in the optical remote sensing image, generating a preliminary feature map of the optical remote sensing image. The feature enhancement pixel encoder module then enhances the preliminary feature map. The feature enhancement pixel encoder module includes an axial attention module and a feature aggregation module, which can more finely perceive the scale and morphological differences of objects in the remote sensing image, resulting in an enhanced feature map of the optical remote sensing image. By designing cluster cross-attention instead of the cross-attention in the common transformer decoder, semantic information of the context is extracted, which can reduce the number of model parameters and improve feature classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart schematically illustrates a cluster-driven optical remote sensing image panoptic segmentation method disclosed in an embodiment of the present invention;

[0039] Figure 2 Schematically shows a flow chart of S130 disclosed in an embodiment of the present invention;

[0040] Figure 3 Schematically showing the network structure of a dual-branch transformer encoder-decoder network disclosed in an embodiment of the present invention;

[0041] Figure 4 A diagram schematically showing a network structure of a feature aggregation module disclosed in an embodiment of the present invention;

[0042] Figure 5 A diagram schematically showing the network structure of the morphology perception module disclosed in an embodiment of the present invention;

[0043] Figure 6 A diagram schematically showing a network structure of a cluster decoder module disclosed in an embodiment of the present invention;

[0044] Figure 7 The figure schematically shows the results of a clustering-driven optical remote sensing image panoramic segmentation method disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The description of the embodiments in this specification should be combined with the corresponding drawings, which should be considered a complete part of this specification. In the drawings, the shapes and thicknesses of the embodiments may be exaggerated and indicated for simplicity or convenience. Furthermore, the various structural components in the drawings will be described separately. It is worth noting that components not shown in the drawings or not described in words are known to those of ordinary skill in the art.

[0046] The description of the embodiments herein and any references to directions and orientations are for ease of description only and are not to be construed as limiting the scope of the present invention. The following description of the preferred embodiments may involve combinations of features, which may exist independently or in combination. The present invention is not specifically limited to the preferred embodiments. The scope of the present invention is defined by the claims.

[0047] To address the problems of target dispersion and insufficient object focus in current intelligent interpretation tasks of remote sensing images, as well as the lack of comprehensive understanding and global cognitive capabilities of existing methods for complex scenes, this paper proposes a clustering-driven panoramic segmentation method for optical remote sensing images. This scheme can automatically realize panoramic segmentation of optical remote sensing images, thereby achieving a comprehensive and in-depth understanding of the scene.

[0048] like Figure 1 As shown, in response to the demand for intelligent interpretation of remote sensing images, the first embodiment of the present invention discloses a cluster-driven optical remote sensing image panoramic segmentation method, which specifically includes the following steps:

[0049] Step S1, preprocessing the optical remote sensing images and remote sensing image panoptic segmentation labels to construct an optical remote sensing image panoptic segmentation training dataset;

[0050] Step S2, constructing a model based on a dual-branch transformer encoder-decoder network, the model including a feature encoder branch and a cluster decoder branch; the feature encoder branch is used to implement feature extraction and enhancement of optical remote sensing images; the cluster decoder branch is used to implement optical remote sensing image feature decoding and mask category prediction;

[0051] Step S3, using the preprocessed optical remote sensing image and remote sensing image label to train the model based on the dual-branch transformer encoder-decoder network to obtain a trained model;

[0052] Step S4: Automatically perform panoramic segmentation on the optical remote sensing image based on the trained model.

[0053] The specific implementation process of preprocessing the optical remote sensing image and the remote sensing image panoramic segmentation label in step S1 includes the following steps:

[0054] First, a specific cropping strategy was employed for the optical remote sensing images and their labels, cropping the original images to a standard size of 800×800 pixels to ensure data consistency and processing efficiency. Furthermore, a cropping step size of 600 pixels was set to ensure spatial continuity between cropped images. Based on this, a panoptic segmentation training dataset was constructed and partitioned based on the cropped remote sensing images and labels, strictly adhering to the COCO data format standard.

[0055] like Figure 2 As shown in the figure, the dual-branch Transformer encoder-decoder network flow chart constructed in step S2 mainly consists of a feature encoder branch and a clustering decoder branch. The feature encoder branch includes a feature extraction backbone network and a feature enhancement pixel encoder, which is designed to extract the pixel feature map mapping of the input optical remote sensing image. Among them, the feature extraction backbone network is responsible for performing preliminary multi-scale feature extraction on the optical remote sensing image, and the feature enhancement pixel encoder realizes the perception of object morphology and the aggregation enhancement of features. The clustering decoder branch is composed of multiple layers of clustering decoders, which is responsible for determining the ground object category in the pixel feature map and generating the corresponding mask features. Finally, the pixel features are combined with the mask features to obtain the final panoramic segmentation mask.

[0056] like Figure 3 Specifically, the feature extraction backbone network consists of a 7×7 convolutional layer, a max pooling layer, and multiple residual units. The feature enhancement pixel encoder is composed of a 7×7 convolutional layer, a max pooling layer, multiple residual units, and multiple axial attention blocks. The two networks are connected by a feature aggregation module. The cluster decoder branch consists of three cluster decoders and a feedforward layer.

[0057] like Figure 4 As shown, the feature aggregation module generates feature weights based on the morphological perception module. Specifically, feature L and feature K are summed and then fed into the morphological perception module to obtain feature aggregation weights W and 1-W. These weights are multiplied by feature L and feature K, respectively, and then summed to obtain the final fusion weight M. This achieves the aggregation of multi-scale features extracted by the feature extraction backbone network and pixel features obtained by the feature enhancement pixel encoder.

[0058] like Figure 5 As shown in the figure, the morphological perception module consists of spatial attention, channel attention, and a 7×7 convolutional layer to effectively capture multi-scale features. The spatial attention module includes one average pooling layer, one max pooling layer, and one 7×7 convolutional layer, while the channel attention module consists of one max pooling layer and two 1×1 convolutional layers. The feature weight W is obtained by summing the output of the attention features and concatenating them with the initial features C along the channel dimension. Finally, the feature weight W is processed through a 7×7 convolutional layer and a sigmoid activation function.

[0059] like Figure 6 As shown in the figure, the cluster decoder consists of a cluster cross-attention layer, a multi-head self-attention layer, and a feedforward network. In the cluster cross-attention layer, cluster centers and pixel features are processed by a linear layer and then multiplied. The final features are filtered through an argmax operation to accurately determine the feature category. Through these steps, this method can effectively improve the panoramic segmentation capabilities of optical remote sensing images and promote intelligent interpretation in complex scenes.

[0060] like Figure 3 As shown, in step S3, a dual-branch Transformer encoder-decoder network is trained using the pre-processed optical remote sensing images and their corresponding panoramic segmentation labels to achieve automatic panoramic segmentation of optical remote sensing images. The process can be summarized into the following key steps:

[0061] First, the preprocessed optical remote sensing imagery is fed into a feature extraction backbone network. Through a combination of convolutional layers and multiple residual units, this deep neural network effectively learns the characteristic information of different object categories in the optical remote sensing imagery, thereby extracting multi-scale features.

[0062] The obtained multi-scale features of the optical remote sensing image are then fed into a feature-enhanced pixel encoder. During this process, a pixel feature map is generated through multiple axial attention blocks, residual units, and 7×7 convolutional layers. Subsequently, a feature aggregation module effectively aggregates the multi-scale features with the pixel features to produce an enhanced pixel feature map.

[0063] The enhanced pixel feature map is then fed into a multi-layer clustering decoder module, where it is first classified through a clustering cross-attention layer, followed by a multi-head self-attention layer and a feedforward network, thereby determining the category of each feature and ultimately generating a mask feature.

[0064] Finally, a panoptic segmentation mask is generated by further aggregating pixel features and mask features. This panoptic segmentation mask is continuously optimized with the input pre-processed panoptic segmentation labels to minimize the loss function. Through backpropagation, the entire two-branch Transformer encoder-decoder network is optimized, ultimately achieving more accurate panoptic segmentation results. The minimum loss function used is the cross entropy loss function and the Dice loss function.

[0065] Finally, in step S4, the optical remote sensing image to be segmented is input into the trained model, and the output result is the image after automatic panoramic segmentation.

[0066] In a second embodiment of the present invention, the present invention provides a system for implementing any of the methods described in the first embodiment, characterized by comprising:

[0067] An input module, used for inputting optical remote sensing images to be segmented;

[0068] A panoptic segmentation module is configured to segment the input optical remote sensing image; the panoptic segmentation module comprises a feature encoder branch and a cluster decoder branch, wherein the feature encoder branch is configured to obtain an enhanced pixel feature map based on the optical remote sensing image to be segmented; the cluster decoder branch is configured to obtain a mask feature map based on the enhanced pixel feature map, and further aggregate the pixel feature map with the mask feature map to generate a panoptic segmentation mask to obtain a segmented image;

[0069] The output module is used to output the segmented image.

[0070] In a third embodiment of the present invention, the present invention provides a computer device comprising a memory, a processor, and program instructions stored in the memory for execution by the processor, wherein the processor executes the program instructions to implement the steps of any one of the methods described in the first embodiment.

[0071] In a fourth embodiment of the present invention, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program implements any one of the methods in the first embodiment when executed by a processor.

[0072] like Figure 7 As shown, exemplary results of the panoramic segmentation results of the optical remote sensing image and the original optical remote sensing image generated by the above method are shown.

[0073] The clustering-driven panoramic segmentation method for optical remote sensing images proposed in this embodiment uses computer vision and remote sensing image intelligent processing technology to perform intelligent interpretation and panoramic segmentation on images of typical remote sensing scenes such as airports and ports. The current intelligent interpretation task of remote sensing images faces multiple challenges, including target dispersion and insufficient object focus. In addition, existing methods lack the ability to comprehensively understand complex scenes and have a global cognitive ability. To this end, the present invention proposes an innovative clustering-driven panoramic segmentation technology solution for optical remote sensing images. Starting from the original image, this solution deeply analyzes the characteristics of the ground objects and their morphological characteristics in the remote sensing image, and can realize the automated processing of panoramic segmentation of optical remote sensing images. Through this technical path, it is possible to fully and deeply understand the scene characteristics presented by the image, and effectively improve the real-time intelligent interpretation capability of typical optical remote sensing images. Compared with previous inventions, it not only optimizes the efficiency of remote sensing image analysis, but also provides reliable support for intelligent decision-making in complex scenes, and has important application value and development potential.

[0074] The serial numbers of the above-mentioned steps involved in the method of the present invention do not mean the order of execution of the method. The execution order of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cluster-driven optical remote sensing image panoptic segmentation method, characterized in that: The method comprises: Step S1, preprocessing the optical remote sensing images and remote sensing image panoptic segmentation labels to construct an optical remote sensing image panoptic segmentation training dataset; Step S2, constructing a model based on a dual-branch transformer encoder-decoder network, the model including a feature encoder branch and a cluster decoder branch; the feature encoder branch is used to implement feature extraction and enhancement of optical remote sensing images; the cluster decoder branch is used to implement optical remote sensing image feature decoding and mask category prediction; Step S3, using the preprocessed optical remote sensing image and remote sensing image label to train the model based on the dual-branch transformer encoder-decoder network to obtain a trained model; step S3 further includes: Step S31, inputting the pre-processed optical remote sensing image set and remote sensing image labels into the feature encoder branch to obtain an enhanced pixel feature map; Step S32: input the enhanced pixel feature map into a cluster decoder branch to obtain a mask feature map, and further aggregate the pixel feature map and the mask feature map to generate a panoptic segmentation mask; Step S33, optimizing the entire dual-branch Transformer encoder-decoder network by minimizing the loss function based on the panoptic segmentation mask and the input pre-processed panoptic segmentation label, and obtaining a trained model through a back-propagation mechanism; Step S4: Automatically perform panoramic segmentation on the optical remote sensing image based on the trained model.

2. The method according to claim 1, characterized in that The step S1 pre-processes the optical remote sensing image and the remote sensing image panoptic segmentation label, including: The optical remote sensing images and their labels were cropped with a cropping step of 600 pixels to crop the original images to a standard size of 800 × 800 pixels; For the cropped remote sensing images and labels, we strictly follow the COCO data format standard to construct a panoramic segmentation training dataset.

3. The method according to claim 1, characterized in that The feature encoder branch includes a feature extraction backbone network and a feature enhancement pixel encoder module; The feature extraction backbone network adopts a backbone network residual neural network ResNets, which is composed of convolutional layers, maximum pooling layers and multi-layer residual units. Each residual unit contains multiple convolutional layers and directly connects the input to the output through skip connections. The feature extraction backbone network is used to learn and extract the feature information of various types of objects in the optical remote sensing image; The feature enhancement pixel encoder module consists of an axial attention module, multiple residual units and a feature aggregation module; the feature enhancement pixel encoder module is used to restore the feature map resolution and integrate multi-scale and multi-level features.

4. The method according to claim 3, characterized in that The feature aggregation module consists of a bilinear interpolation operation and a morphological perception module; the feature aggregation module is used to fuse features of different resolutions to balance high and low resolution information; The morphological perception module consists of a spatial attention layer, a channel attention layer, and a 7×7 convolutional layer to effectively capture multi-scale features; The spatial attention layer includes an average pooling layer, a maximum pooling layer and a 7×7 convolutional layer; The channel attention layer is composed of a maximum pooling layer and two 1×1 convolutional layers. The output results of the attention features are summed up and concatenated with the initial feature C in the channel dimension. Finally, the feature weights are obtained after being processed by the 7×7 convolutional layer and the sigmoid activation function.

5. The method according to claim 3, characterized in that The axial attention module is composed of self-attention calculation modules in the horizontal and vertical directions.

6. The method according to claim 1, characterized in that The cluster decoder branch consists of 6 layers of cluster decoder modules and a layer of loss function; The cluster decoding module consists of a cluster cross attention module, a multi-head self-attention module and a feedforward module, which is used to extract contextual semantic information and perform mask category prediction; The cluster cross attention module consists of a cluster cross attention layer, a multi-head self-attention layer and a feedforward network layer; the cluster cross attention module is used to classify features, thereby determining the feature category and ultimately generating mask features; The clustering cross attention layer consists of three linear layers and one argmax operation, which is used to determine the feature category.

7. A system for implementing the method according to any one of claims 1 to 6, characterized in that: include: An input module, used for inputting optical remote sensing images to be segmented; Panoramic segmentation module, used to segment the input optical remote sensing image; The panoramic segmentation module includes a feature encoder branch and a cluster decoder branch, wherein the feature encoder branch is used to obtain an enhanced pixel feature map according to the optical remote sensing image to be segmented; The cluster decoder branch is used to obtain a mask feature map according to the enhanced pixel feature map, and further aggregate the pixel feature map and the mask feature map to generate a panoramic segmentation mask to obtain a segmented image; The output module is used to output the segmented image.

8. A computer device comprising a memory, a processor, and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images