Clustering-driven optical remote sensing image panoramic segmentation method

Abstract

The invention relates to a cluster-driven optical remote sensing image panoramic segmentation method, which belongs to the technical field of computer vision and remote sensing image processing, and comprises the following steps of: preprocessing an optical remote sensing image and a remote sensing image panoramic segmentation label to construct a double-branch transform coder-decoder network; and performing training learning on the double-branch transform coder-decoder network by using the preprocessed optical remote sensing image and the remote sensing image label to realize panoramic segmentation of the optical remote sensing image. The method provided by the invention can realize automatic generation of the high-quality optical remote sensing image segmentation mask of a typical target; the technical problems of target dispersion and insufficient object focusing in the current remote sensing image intelligent interpretation task are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and remote sensing image processing, and in particular to a panoramic segmentation algorithm for optical remote sensing images. Background Art

[0002] In recent years, with the development of remote sensing technology, the spatial resolution of remote sensing images has gradually improved, and the details and hierarchical structures of objects that can be carried and expressed by images have become more diverse. Accurately and effectively extracting all-element object information in a specific scene and forming a comprehensive understanding of the scene are key steps in the successful application of remote sensing technology. At present, the efficiency of remote sensing image recognition lags far behind the growth rate of image data and is becoming increasingly severe, resulting in a large amount of data resources being wasted, forming a "big data, small knowledge" dilemma. The surge in the amount of satellite remote sensing data, the lack of remote sensing data analysis capabilities, and the gap between the bottlenecks of traditional remote sensing technology urgently require a new, efficient, accurate, and convenient intelligent interpretation method.

[0003] Although deep learning methods have made significant progress in the fields of specific target detection and recognition, semantic segmentation of objects, instance segmentation, scene classification, etc., due to the discreteness of task objectives and the lack of focus on objects, they still lack the ability to comprehensively understand and globally recognize complex scenes, and it is difficult to obtain global knowledge to mine the overall hidden value of remote sensing big data. For example, target detection and semantic instance segmentation methods can only interpret foreground objects in remote sensing images, but cannot understand the background information in the image; and semantic segmentation algorithms can only assign categories to each pixel in the image, but do not distinguish different individuals in the same category. Therefore, whether it is possible to simultaneously identify irregular background objects and key targets in remote sensing images and realize comprehensive interpretation and cognition of scenes is a necessary link in the transformation of remote sensing data into knowledge. Panoramic segmentation of remote sensing images is an advanced image processing algorithm and one of the important tasks of remote sensing scene understanding and cognition. It combines the characteristics of semantic segmentation and instance segmentation, and simultaneously classifies foreground and background and distinguishes similar individuals. It has important application value in transportation hub monitoring, battlefield environment assessment, national defense security, etc.

[0004] Clustering refers to the process of dividing a set of data objects into several groups or clusters, so that the data objects in the same cluster have a high degree of similarity, while the data objects in different clusters have a low degree of similarity. Common clustering methods include K-means clustering and hierarchical clustering. The similarity measure between objects obtained by the clustering method is helpful for image classification and segmentation. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned prior art, the object of the present invention is to provide a clustering-driven optical remote sensing image panoramic segmentation method, which can realize automatic panoramic segmentation of optical remote sensing images.

[0006] To achieve the above-mentioned object of the invention, in a first aspect, the present invention provides a cluster-driven optical remote sensing image panoramic segmentation method, comprising:

[0007] Step S1, preprocessing the optical remote sensing image and the remote sensing image panoramic segmentation label to construct an optical remote sensing image panoramic segmentation training data set;

[0008] Step S2, constructing a model based on a dual-branch transformer encoder-decoder network, the model comprising a feature encoder branch and a clustering decoder branch; the feature encoder branch is used to realize feature extraction and enhancement of optical remote sensing images; the clustering decoder branch is used to realize feature decoding of optical remote sensing images and mask category prediction;

[0009] Step S3, training the model based on the dual-branch transformer encoder-decoder network using the preprocessed optical remote sensing image and remote sensing image label to obtain a trained model;

[0010] Step S4, performing automatic panoramic segmentation on the optical remote sensing image based on the trained model.

[0011] Furthermore, in step S1, the optical remote sensing image and the remote sensing image panoramic segmentation label are preprocessed, including:

[0012] The optical remote sensing images and their labels were cropped with a cropping step of 600 pixels to crop the original images to a standard size of 800 × 800 pixels;

[0013] For the cropped remote sensing images and labels, we strictly follow the COCO data format standard to construct a panoramic segmentation training dataset.

[0014] Further, the feature encoder branch includes a feature extraction backbone network and a feature enhancement pixel encoder module;

[0015] The feature extraction backbone network adopts a backbone network residual neural network ResNets, which is composed of a convolutional layer, a maximum pooling layer and a multi-layer residual unit. Each residual unit contains multiple convolutional layers, and the input is directly connected to the output through a jump connection; the feature extraction backbone network is used to learn and extract the feature information of various types of objects in the optical remote sensing image;

[0016] The feature enhancement pixel encoder module consists of an axial attention module, multiple residual units and a feature aggregation module; the feature enhancement pixel encoder module is used to restore the feature map resolution and integrate multi-scale and multi-level features.

[0017] Furthermore, the feature aggregation module is composed of a bilinear interpolation operation and a morphological perception module; the feature aggregation module is used to fuse features of different resolutions to balance high and low resolution information;

[0018] The morphology perception module consists of a spatial attention layer, a channel attention layer, and a 7×7 convolutional layer to effectively capture multi-scale features;

[0019] The spatial attention layer includes an average pooling layer, a maximum pooling layer and a 7×7 convolution layer;

[0020] The channel attention layer is composed of a maximum pooling layer and two 1×1 convolutional layers. The output results of the attention features are added and concatenated with the initial feature C in the channel dimension. Finally, the feature weights are obtained after being processed by the 7×7 convolutional layer and the sigmoid activation function.

[0021] The axial attention module consists of self-attention calculation modules in the horizontal and vertical directions.

[0022] Furthermore, the cluster decoder branch consists of 6 layers of cluster decoder modules and a layer of loss function;

[0023] The cluster decoding module consists of a cluster cross attention module, a multi-head self-attention module and a feedforward module, which is used to extract contextual semantic information and perform mask category prediction;

[0024] The cluster cross attention module consists of a cluster cross attention layer, a multi-head self-attention layer and a feedforward network layer; the cluster cross attention module is used to classify features, thereby determining the feature category and finally generating mask features;

[0025] The clustering cross attention layer is composed of three linear layers and one argmax operation, which is used to accurately determine the feature category.

[0026] Furthermore, step S3 further includes:

[0027] Step S31, inputting the preprocessed optical remote sensing image set and remote sensing image labels into a feature encoder branch to obtain an enhanced pixel feature map;

[0028] Step S32, inputting the enhanced pixel feature map into a cluster encoder branch to obtain a mask feature map, further aggregating the pixel feature map with the mask feature map to generate a panoramic segmentation mask;

[0029] Step S33, optimizing the entire dual-branch Transformer encoder-decoder network by minimizing the loss function according to the panoptic segmentation mask and the input preprocessed panoptic segmentation label, and obtaining a trained model through a back-propagation mechanism. The minimum loss function adopts a cross entropy loss function and a Dice loss function.

[0030] In a second aspect of the present invention, the present invention provides a system for implementing any one of the methods described in the first aspect, characterized in that it comprises:

[0031] An input module, used for inputting optical remote sensing images to be segmented;

[0032] A panoramic segmentation module is used to segment the input optical remote sensing image; the panoramic segmentation module includes a feature encoder branch and a clustering decoder branch, the feature encoder branch is used to obtain an enhanced pixel feature map according to the optical remote sensing image to be segmented; the clustering encoder branch is used to obtain a mask feature map according to the enhanced pixel feature map, and further aggregate the pixel feature map with the mask feature map to generate a panoramic segmentation mask to obtain a segmented image;

[0033] The output module is used to output the segmented image.

[0034] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps of any one of the methods described in the first aspect.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, characterized in that the computer program is executed by a processor according to any one of the methods described in the first aspect.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The solution of the embodiment of the present invention realizes the panoramic segmentation of optical remote sensing images by designing a dual-branch transformer encoder-decoder network based on clustering drive. The network consists of a feature encoder (including a feature extraction backbone network and a feature enhancement pixel encoder module) and a cluster decoder. The training data includes two sets: optical remote sensing images and optical remote sensing image annotations. Among them, the feature extraction backbone network integrates and extracts salient information by learning the shape, color, texture and other features of various objects in the optical remote sensing image, generates a preliminary feature map of the optical remote sensing image, and enhances the features of the preliminary feature map through the feature enhancement pixel encoder module. The feature enhancement pixel encoder module includes an axial attention module and a feature aggregation module, which can more finely perceive the scale and morphological feature differences of the objects in the remote sensing image to obtain an enhanced feature map of the optical remote sensing image. By designing cluster cross attention instead of the cross attention in the common transformer decoder, the semantic information of the context is extracted, the number of parameters of the model can be reduced and the feature classification effect can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart schematically showing a cluster-driven optical remote sensing image panoramic segmentation method disclosed in an embodiment of the present invention;

[0039] Figure 2 The flowchart of S130 disclosed in the embodiment of the present invention is schematically shown;

[0040] Figure 3 A schematic diagram of a dual-branch transformer encoder-decoder network structure disclosed in an embodiment of the present invention is shown;

[0041] Figure 4 A network structure diagram schematically showing a feature aggregation module disclosed in an embodiment of the present invention;

[0042] Figure 5 A schematic diagram showing a network structure of a morphology perception module disclosed in an embodiment of the present invention;

[0043] Figure 6 A schematic diagram showing a network structure of a cluster decoder module disclosed in an embodiment of the present invention;

[0044] Figure 7 The result diagram of a clustering-driven optical remote sensing image panoramic segmentation method disclosed in an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0045] The description of the embodiments of this specification should be combined with the corresponding drawings, which should be considered as part of the complete specification. In the drawings, the shape or thickness of the embodiments may be enlarged and indicated for simplification or convenience. Furthermore, the parts of each structure in the drawings will be described separately. It is worth noting that the elements not shown in the drawings or not described in words are in a form known to ordinary technicians in the relevant technical field.

[0046] The description of the embodiments herein and any reference to directions and orientations are only for the convenience of description and are not to be construed as any limitation on the scope of protection of the present invention. The following description of the preferred embodiments may involve combinations of features, which may exist independently or in combination, and the present invention is not particularly limited to the preferred embodiments. The scope of the present invention is defined by the claims.

[0047] In response to the problems of target dispersion and insufficient object focus in the current intelligent interpretation tasks of remote sensing images, as well as the lack of comprehensive understanding and global cognitive ability of complex scenes in existing methods, the present invention proposes a clustering-driven panoramic segmentation method for optical remote sensing images. This scheme can automatically realize the panoramic segmentation of optical remote sensing images, thereby achieving a comprehensive and in-depth understanding of the scene.

[0048] like Figure 1 As shown, in response to the demand for intelligent interpretation of remote sensing images, the first embodiment of the present invention discloses a cluster-driven optical remote sensing image panoramic segmentation method, which specifically includes the following steps:

[0049] Step S1, preprocessing the optical remote sensing image and the remote sensing image panoramic segmentation label to construct an optical remote sensing image panoramic segmentation training data set;

[0050] Step S2, constructing a model based on a dual-branch transformer encoder-decoder network, the model comprising a feature encoder branch and a clustering decoder branch; the feature encoder branch is used to realize feature extraction and enhancement of optical remote sensing images; the clustering decoder branch is used to realize feature decoding of optical remote sensing images and mask category prediction;

[0051] Step S3, training the model based on the dual-branch transformer encoder-decoder network using the preprocessed optical remote sensing image and remote sensing image label to obtain a trained model;

[0052] Step S4, performing automatic panoramic segmentation on the optical remote sensing image based on the trained model.

[0053] The specific implementation process of preprocessing the optical remote sensing image and the remote sensing image panoramic segmentation label in step S1 includes the following steps:

[0054] First, for optical remote sensing images and their labels, a specific cropping strategy is used to crop the original images to a standard size of 800×800 pixels to ensure data consistency and processing efficiency. In addition, in order to ensure the spatial continuity between the cropped images, a cropping step of 600 pixels is set. On this basis, based on the cropped remote sensing images and labels, the COCO data format standard is strictly followed to construct a panoramic segmentation training dataset and divide it reasonably.

[0055] like Figure 2 As shown, the dual-branch Transformer encoder-decoder network flow chart constructed in step S2 is mainly composed of a feature encoder branch and a clustering decoder branch. The feature encoder branch includes a feature extraction backbone network and a feature enhancement pixel encoder, which is intended to extract the pixel feature map mapping of the input optical remote sensing image. Among them, the feature extraction backbone network is responsible for the preliminary multi-scale feature extraction of the optical remote sensing image, and the feature enhancement pixel encoder realizes the perception of the object morphology and the aggregation enhancement of the features. The clustering decoder branch is composed of a multi-layer clustering decoder, which is responsible for determining the category of the ground objects in the pixel feature map and generating the corresponding mask features. Finally, the pixel features are combined with the mask features to obtain the final panoramic segmentation mask.

[0056] like Figure 3 As shown, specifically, the feature extraction backbone network includes a 7×7 convolutional layer, a maximum pooling layer, and multiple residual units. The feature enhancement pixel encoder is composed of a 7×7 convolutional layer, a maximum pooling layer, multiple residual units, and multiple axial attention blocks. The two networks are connected through a feature aggregation module. The clustering decoder branch consists of three clustering decoders and a feedforward layer.

[0057] like Figure 4 As shown, the feature aggregation module generates feature weights based on the morphological perception module. Specifically, feature L and feature K are added and then enter the morphological perception module to obtain feature aggregation weights W and 1-W, which are multiplied with feature L and feature K respectively and then added to obtain the final fusion weight M. The multi-scale features extracted by the feature extraction backbone network and the pixel feature aggregation obtained by the feature enhancement pixel encoder are realized.

[0058] like Figure 5 As shown in the figure, the morphological perception module consists of spatial attention, channel attention and 7×7 convolutional layers to effectively capture multi-scale features. In the spatial attention part, the module includes an average pooling layer, a maximum pooling layer and a 7×7 convolutional layer, while the channel attention consists of a maximum pooling layer and two 1×1 convolutional layers. The output results of the attention features are summed and concatenated with the initial features C in the channel dimension, and finally processed by a 7×7 convolutional layer and a sigmoid activation function to obtain the feature weight W.

[0059] like Figure 6 As shown in the figure, the cluster decoder consists of a cluster cross attention layer, a multi-head self-attention layer and a feedforward network. In the cluster cross attention layer, the cluster center and the pixel feature are multiplied after being processed by the linear layer, and the final feature is screened out by the argmax operation to achieve accurate judgment of the feature category. Through the above steps, this method can effectively improve the panoramic segmentation ability of optical remote sensing images and promote intelligent interpretation in complex scenes.

[0060] like Figure 3 As shown, in step S3, a dual-branch Transformer encoder-decoder network is trained using the preprocessed optical remote sensing image and its corresponding panoramic segmentation label to achieve automatic panoramic segmentation of the optical remote sensing image. The process can be summarized into the following key steps:

[0061] First, the preprocessed optical remote sensing image set is input into the feature extraction backbone network. Through the combination of convolutional layers and multiple residual units, the deep neural network can effectively learn the feature information of different types of objects in optical remote sensing images, thereby extracting multi-scale features.

[0062] Next, the obtained multi-scale features of the optical remote sensing image will be input into the feature enhanced pixel encoder. In this process, a pixel feature map is generated through multiple axial attention blocks, residual units, and 7×7 convolution layers. Subsequently, the feature aggregation module is used to effectively aggregate the multi-scale features with the pixel features to obtain an enhanced pixel feature map.

[0063] Then, the enhanced pixel feature map is input into the multi-layer clustering decoder module, where the features are first classified through a clustering cross-attention layer, followed by a multi-head self-attention layer and a feed-forward network, so as to determine the category of each feature and finally generate a mask feature.

[0064] Finally, the panoramic segmentation mask is generated by further aggregating pixel features and mask features. The panoramic segmentation mask will be continuously optimized with the input preprocessed panoramic segmentation label to minimize the loss function. Through the back-propagation mechanism, the entire dual-branch Transformer encoder-decoder network is optimized to achieve a more accurate panoramic segmentation result. Among them, the minimum loss function uses the cross entropy loss function and the Dice loss function.

[0065] Finally, in step S4, the optical remote sensing image to be segmented is input into the trained model, and the output result is the image after automatic panoramic segmentation.

[0066] In a second embodiment of the present invention, the present invention provides a system for implementing any method described in any one of the first embodiments, characterized by comprising:

[0067] An input module, used for inputting optical remote sensing images to be segmented;

[0068] A panoramic segmentation module is used to segment the input optical remote sensing image; the panoramic segmentation module includes a feature encoder branch and a clustering decoder branch, the feature encoder branch is used to obtain an enhanced pixel feature map according to the optical remote sensing image to be segmented; the clustering encoder branch is used to obtain a mask feature map according to the enhanced pixel feature map, and further aggregate the pixel feature map with the mask feature map to generate a panoramic segmentation mask to obtain a segmented image;

[0069] The output module is used to output the segmented image.

[0070] In a third embodiment of the present invention, the present invention provides a computer device, comprising a memory, a processor, and program instructions stored in the memory for execution by the processor, wherein the processor executes the program instructions to implement the steps of any one of the methods described in the first embodiment.

[0071] In a fourth embodiment of the present invention, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program implements any one of the methods in the first embodiment when executed by a processor.

[0072] like Figure 7 As shown, exemplary results of the panoramic segmentation results of the optical remote sensing image and the original optical remote sensing image generated by the above method are shown.

[0073] The clustering-driven optical remote sensing image panoramic segmentation method proposed in this embodiment, with the help of computer vision and remote sensing image intelligent processing technology, is mainly used for intelligent interpretation and panoramic segmentation of images of typical remote sensing scenes such as airports and ports. The current remote sensing image intelligent interpretation task faces multiple challenges, including target dispersion and insufficient object focus. In addition, the existing methods lack the problem of comprehensive understanding and global cognitive ability of complex scenes. To this end, the present invention proposes an innovative clustering-driven optical remote sensing image panoramic segmentation technology solution. Starting from the original image, the scheme deeply analyzes the characteristics of the ground object elements and their morphological characteristics in the remote sensing image, and can realize the automatic processing of the panoramic segmentation of the optical remote sensing image. Through this technical path, the scene characteristics presented by the image can be fully and deeply understood, and the real-time intelligent interpretation capability of typical optical remote sensing images can be effectively improved. Compared with previous inventions, it not only optimizes the efficiency of remote sensing image analysis, but also provides reliable support for intelligent decision-making in complex scenes, and has important application value and development potential.

[0074] The serial numbers of the above-mentioned steps involved in the method of the present invention do not mean the order of execution of the method. The execution order of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation mode of the present invention.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cluster-driven optical remote sensing image panoramic segmentation method, characterized in that: The method comprises: Step S1, preprocessing the optical remote sensing image and the remote sensing image panoramic segmentation label to construct an optical remote sensing image panoramic segmentation training data set; Step S2, constructing a model based on a dual-branch transformer encoder-decoder network, the model comprising a feature encoder branch and a clustering decoder branch; the feature encoder branch is used to realize feature extraction and enhancement of optical remote sensing images; the clustering decoder branch is used to realize feature decoding of optical remote sensing images and mask category prediction; Step S3, training the model based on the dual-branch transformer encoder-decoder network using the preprocessed optical remote sensing image and remote sensing image label to obtain a trained model; Step S4, performing automatic panoramic segmentation on the optical remote sensing image based on the trained model.

2. The method according to claim 1, characterized in that The step S1 pre-processes the optical remote sensing image and the remote sensing image panoramic segmentation label, including: The optical remote sensing images and their labels were cropped with a cropping step of 600 pixels to crop the original images to a standard size of 800 × 800 pixels; For the cropped remote sensing images and labels, we strictly follow the COCO data format standard to construct a panoramic segmentation training dataset.

3. The method according to claim 1, characterized in that The feature encoder branch includes a feature extraction backbone network and a feature enhancement pixel encoder module; The feature extraction backbone network adopts a backbone network residual neural network ResNets, which is composed of a convolutional layer, a maximum pooling layer and a multi-layer residual unit. Each residual unit contains multiple convolutional layers, and the input is directly connected to the output through a jump connection; the feature extraction backbone network is used to learn and extract the feature information of various types of objects in the optical remote sensing image; The feature enhancement pixel encoder module consists of an axial attention module, multiple residual units and a feature aggregation module; the feature enhancement pixel encoder module is used to restore the feature map resolution and integrate multi-scale and multi-level features.

4. The method according to claim 3, characterized in that The feature aggregation module is composed of a bilinear interpolation operation and a morphological perception module; the feature aggregation module is used to fuse features of different resolutions to balance high and low resolution information; The morphology perception module consists of a spatial attention layer, a channel attention layer, and a 7×7 convolutional layer to effectively capture multi-scale features; The spatial attention layer includes an average pooling layer, a maximum pooling layer and a 7×7 convolution layer; The channel attention layer is composed of a maximum pooling layer and two 1×1 convolutional layers. The output results of the attention features are added and concatenated with the initial feature C in the channel dimension. Finally, the feature weights are obtained after being processed by the 7×7 convolutional layer and the sigmoid activation function.

5. The method according to claim 3, characterized in that: The axial attention module is composed of self-attention calculation modules in the horizontal and vertical directions.

6. The method according to claim 1, characterized in that The cluster decoder branch includes 6 layers of cluster decoder modules and a layer of loss function; The cluster decoding module consists of a cluster cross attention module, a multi-head self-attention module and a feedforward module, which is used to extract contextual semantic information and perform mask category prediction; The cluster cross attention module consists of a cluster cross attention layer, a multi-head self-attention layer and a feedforward network layer; the cluster cross attention module is used to classify features, thereby determining the feature category and finally generating mask features; The clustering cross attention layer is composed of three linear layers and one argmax operation, which is used to accurately determine the feature category.

7. The method according to claim 1, characterized in that The step S3 further comprises: Step S31, inputting the preprocessed optical remote sensing image set and remote sensing image labels into a feature encoder branch to obtain an enhanced pixel feature map; Step S32, inputting the enhanced pixel feature map into a cluster encoder branch to obtain a mask feature map, further aggregating the pixel feature map with the mask feature map to generate a panoramic segmentation mask; Step S33, optimizing the entire dual-branch Transformer encoder-decoder network by minimizing the loss function according to the panoramic segmentation mask and the input preprocessed panoramic segmentation label, and obtaining a trained model through a back-propagation mechanism.

8. A system for implementing the method according to any one of claims 1 to 7, characterized in that: include: An input module, used for inputting optical remote sensing images to be segmented; Panoramic segmentation module, used to segment the input optical remote sensing image; The panoramic segmentation module includes a feature encoder branch and a cluster decoder branch, wherein the feature encoder branch is used to obtain an enhanced pixel feature map according to the optical remote sensing image to be segmented; The cluster encoder branch is used to obtain a mask feature map according to the enhanced pixel feature map, and further aggregate the pixel feature map with the mask feature map to generate a panoramic segmentation mask to obtain a segmented image; The output module is used to output the segmented image.

9. A computer device comprising a memory, a processor and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images