Remote sensing image fine-grained ground feature extraction method based on feature conversion and knowledge enhancement

Abstract

The invention discloses a remote sensing image fine-grained ground feature extraction method based on feature conversion and knowledge enhancement, and the method comprises the steps: converting the pixel information of a remote sensing image into feature level information through a feature conversion module, and enhancing the extraction and utilization capability of a network for information in different fields; therefore, the extraction and utilization capabilities of the network on feature information of different scales and different aspects are enhanced; the knowledge enhancement module is utilized, information of different stages and information of different scales are fused, feature knowledge extracted by the network is further enhanced, the distinguishing and extracting capability of the network on fine-grained ground features is improved while the feature knowledge is utilized, automatic and intelligent recognition of the fine-grained ground features is achieved, and the recognition efficiency is improved. And the identification precision of the ground features with high inter-class similarity is relatively high.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing, and more specifically, relates to a remote sensing image fine-grained object extraction method based on feature conversion and knowledge enhancement. Background Art

[0002] The task of fine-grained object extraction is one of the key links in remote sensing image analysis. Its goal is to accurately identify and extract objects with high similarity and fine distinction, such as irrigated land and water-irrigated land, lakes and rivers, residential houses and factories, from high-resolution satellite or drone images. This technology has important application value in urban planning, environmental protection, agricultural monitoring and other fields. For example, in urban planning, by extracting the distribution information of residential areas and factory buildings, it can assist in the assessment of urban heat island effects; in environmental protection, the fine identification of tree species and health status is helpful for the protection of biodiversity; and in the field of agricultural monitoring, accurate judgment of crop growth status is crucial to increasing crop yields.

[0003] In order to extract these fine-grained features, traditional methods rely on manual visual interpretation or simple threshold segmentation methods. However, these methods are often time-consuming and labor-intensive and it is difficult to ensure consistency. Therefore, in recent years, researchers have been committed to developing more advanced automatic processing technologies to meet this challenge. The development of machine learning, especially deep learning technology, has provided new ideas for solving this problem. By constructing complex neural network models, not only can efficient processing of remote sensing images be achieved, but also the knowledge and experience of human experts can be simulated to a certain extent, thereby achieving or even exceeding the accuracy of traditional methods. The application of these technologies has greatly promoted the automation process of remote sensing image analysis, and is expected to further improve its intelligence level in the future.

[0004] In recent years, with the progress in the field of deep learning, computer vision technology has continued to make remarkable achievements. In daily image classification tasks, deep learning models have been able to achieve performance comparable to that of humans. As for the extraction of multi-category objects in remote sensing images, that is, semantic segmentation tasks, a major leap in accuracy has been achieved with the support of deep learning technology. However, there is still a certain gap with the level of human experts. Therefore, one of the current research hotspots is how to use deep learning technology to achieve higher accuracy and higher efficiency in fine-grained object information extraction.

[0005] In the prior art, the patent name is: A method for extracting typical objects from remote sensing images based on spectral enhancement and dual-path coding. It uses spectral enhancement modules and dual-path coding modules to start from spatial information and spectral information, integrate multiple features, and increase the model's ability to extract and integrate different types of information, thereby solving the problem of more manual intervention steps and low accuracy in previous methods; four attention mechanisms are used to construct different types of decoders to improve the network's ability to restore information of different types of features; spectral attention is used to improve the network's ability to extract and enhance features of spectral information, and improve the robustness of the network; however, the method for extracting typical objects from remote sensing images based on spectral enhancement and dual-path coding has the following problems:

[0006] 1. Complex network structure. The spectrally enhanced dual-path coding network designs a decoder for each type of ground object, which makes the network structure bloated and cannot handle tasks that require extracting multiple types of ground objects.

[0007] 2. Poor ability to extract and identify objects with high similarity between classes. Although the spectral enhancement dual-path coding network can distinguish objects with high similarity from a spectral perspective to a certain extent, its feature extraction and feature enhancement capabilities are poor, and it is difficult to distinguish objects with high similarity such as irrigated land and irrigated land. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for extracting fine-grained objects from remote sensing images based on feature conversion and knowledge enhancement. The method uses the feature conversion method to fully explore and utilize the pixel space information of the remote sensing image and encode it into the feature space. The feature space information of different scales and stages is repeatedly mined and fused through knowledge enhancement, thereby realizing the intelligent extraction of fine-grained objects and solving the problem that the spectral enhancement dual-path coding has poor resolution for objects with high inter-class similarity and low extraction accuracy.

[0009] The present invention provides a remote sensing image fine-grained object extraction method based on feature conversion and knowledge enhancement, characterized in that it comprises the following steps:

[0010] (1) Construct a training dataset for training the feature knowledge enhancement network;

[0011] (2) Build and train a feature knowledge enhancement network;

[0012] (3) Use the trained feature knowledge to enhance the network for fine-grained object visualization extraction in remote sensing images.

[0013] The object of the invention of the present invention is achieved in this way:

[0014] The present invention discloses a remote sensing image fine-grained object extraction method based on feature conversion and knowledge enhancement. The feature conversion module is introduced to convert the pixel information of the remote sensing image into feature-level information, so as to enhance the network's ability to extract and utilize information in different fields, thereby enhancing the network's ability to extract and utilize feature information at different scales and aspects. The knowledge enhancement module is utilized to integrate information at different stages and different scales, further enhance the network's ability to extract feature knowledge, utilize feature knowledge, and improve the network's ability to distinguish and extract fine-grained objects, thereby realizing automatic and intelligent recognition of fine-grained objects, and achieving high recognition accuracy for objects with high similarity between classes.

[0015] At the same time, the remote sensing image fine-grained object extraction method based on feature conversion and knowledge enhancement of the present invention also has the following beneficial effects:

[0016] (1) Based on the traditional deep learning network structure, the present invention introduces a feature conversion module to improve the ability of the network model to extract feature spatial information. The feature conversion module can also improve the accuracy of the network model in extracting fine-grained objects;

[0017] (2) To address the problem of high similarity between some ground objects in remote sensing images, a knowledge enhancement module is designed. The knowledge enhancement module can improve the network's ability to extract fine-grained information and solve the problem of low extraction accuracy of similar ground objects.

[0018] (3) In order to solve the problem that similar objects in remote sensing images have small differences and different feature information is difficult to integrate and utilize, the present invention introduces a time step-controlled loop method to iteratively optimize the segmentation results. This method includes multiple repetitions of network processing, each of which is controlled by a time step. The results of each iteration are integrated into the next step, providing additional refinement information to generate more accurate and complete results. This iterative process ensures that the segmentation accuracy can be gradually improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is the overall structure diagram of the feature knowledge enhancement network of the present invention;

[0020] Figure 2 Feature conversion module structure diagram;

[0021] Figure 3 Knowledge enhancement module diagram;

[0022] Figure 4 ST module structure diagram;

[0023] Figure 5 RBT structure diagram;

[0024] Figure 6The experimental results are shown in Figure 1. (a) shows the original image, (b) shows the label image, (c) shows the UNet experimental result, (d) shows the spectral enhanced dual-path coding network experimental result, and (e) shows the feature knowledge enhanced network experimental result. DETAILED DESCRIPTION

[0025] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0026] Example

[0027] For the convenience of description, the relevant professional terms appearing in the specific implementation methods are first explained:

[0028] ST (Spatial Transformer): Spatial Transformation Module

[0029] RBT (Resnet Block with Timestep): Time Residual Module

[0030] In this embodiment, the present invention provides a method for extracting typical objects from remote sensing images based on spectral enhancement and dual-path coding, comprising the following steps:

[0031] (1) Construct a training dataset;

[0032] (1.1) Download multiple remote sensing images and crop each remote sensing image into m*n blocks;

[0033] (1.2) Use semantic segmentation and annotation tools to mark fine-grained features in remote sensing images with different shapes, where fine-grained features include industrial areas, urban areas, rural areas, sports fields, squares, roads, overpasses, railway stations, airports, irrigated land, irrigated land, dry land, gardens, arbor forests, shrub forests, parks, natural meadows, artificial meadows, rivers, lakes, ponds, fish ponds, snowfields, and wastelands;

[0034] (1.3) Set the pixel values ​​corresponding to each typical feature to 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, so as to generate a label image, where the pixel value of the industrial area is set to 0, the pixel value of the urban area is set to 1, and so on;

[0035] (1.4) Each remote sensing image and the corresponding label image are used as a set of training data to form a training data set;

[0036] (2) Construction Figure 1 The feature knowledge shown is used to enhance the network and train it;

[0037] A set of training data is used as the input of the feature knowledge enhancement network;

[0038] In this embodiment, the feature conversion module structure is as follows: Figure 2 As shown in Figure 1, the feature knowledge enhancement network starts with the encoder E of the feature conversion module. The encoder E contains three residual blocks and three 2x2 average pooling layers. Each residual block includes two convolution modules. The convolution module contains a 3x3 convolution layer, a batch normalization layer, and a Relu activation function. The two convolution modules form a residual link. After the original image passes through the first residual block and the 2x2 average pooling layer, the intermediate feature map is obtained. Intermediate feature map After the second residual block and the 2x2 average pooling layer, the intermediate feature map is obtained. Intermediate feature map After the third residual block and the 2x2 average pooling layer, the initial knowledge feature map Z is obtained. C×H×W ; Where C, H, W represent the number of channels, height and width of the initial knowledge feature map respectively; Then the initial knowledge feature map Z C×H×W Input into the knowledge enhancement module for knowledge enhancement operation;

[0039] like Figure 3 As shown, the knowledge enhancement module includes a loop counter t, a knowledge fusion module and a loop predictor; the loop counter t controls the knowledge fusion module and the loop predictor to loop three times to obtain the output of the knowledge enhancement module;

[0040] Among them, the initial knowledge feature graph Z C×H×W After entering the knowledge enhancement module, the knowledge fusion module is first entered. At this time, the loop counter t first determines the number of loops. If t>1, Z C×H×W and Superimpose in the channel direction to obtain the feature map If t = 1, then Z C×H×W It is superimposed with a tensor of size C×W×H that is all 0 in the channel direction to obtain the feature map Then the feature map After two convolution modules and a 1×1 convolution layer, the fused feature map P is obtained. t C×H×W ;

[0041] Fusion feature map P t C×H×W Then enter the loop predictor; fusion feature map P tC×H×W First enter an RBT module to get the feature map Then, the feature map is obtained by passing through a 2x2 average pooling downsampling layer, an ST module, and an RBT module. Then, the feature map is obtained by sequentially passing through a 2x2 average pooling downsampling layer, an ST module, and an RBT module. After a 2x2 average pooling downsampling layer and an ST module, the feature map is obtained. Then enter an RBT module and an upsampling layer to get the feature map and Enter the ST module together to get the feature map After an RBT module and an upsampling layer, the feature map is obtained and Enter an ST module together to get the feature map After an RBT module and upsampling layer, the feature map Y is obtained. t C×H×W ;

[0042] In this embodiment, the ST module structure is as follows Figure 4 As shown, assuming that the input feature map of the ST module is F1 C×H×W and F1 C×H×W Through three parallel fully connected layers, we get Will and After point multiplication, softmax operation is performed to obtain and Perform matrix multiplication and then add it to F1 C ×H×W Add together to get Through two parallel fully connected layers, we get and Through a fully connected layer Will and After point multiplication, softmax operation is performed to obtain and Perform matrix multiplication and then add it to F1 C×H×W Add together to get the output of the ST module If the ST module has only one input, then

[0043] In this embodiment, the RBT structure is as follows Figure 5 As shown, the RBT module includes two input branches, one input branch is the time step T 1×C , T 1×C It is a vector of size 1×C with all values ​​t; the other input branch is the feature map;

[0044] Assume that the feature map of an input branch is F1 C×H×W ; Time step T 1×C First, after two fully connected layers, we get T1 C ×H×W ; F1 C×H×W First, after a convolution module, we get F1 C×H×W 、T1 C×H×W Add bit by bit and get F3 C×H×W , F3 C×H×W After passing through a convolution module, the output of the RBT module is obtained

[0045]

[0046] Y t C×H×W Input to the decoder D of the feature conversion module; Y t C×H×W First and Z C×H×W Superimpose in the channel direction to obtain the feature map After passing through an upsampling layer and a residual block, it is combined with the feature map Superimpose in the channel direction to obtain the feature map Feature Map After passing through an upsampling layer and a residual block, it is combined with the feature map Superimpose in the channel direction to obtain the feature map Final feature map After an upsampling layer and a residual block, the stage classification probability weight map pre is obtained. t ;

[0047] At this time, the time counter t is judged; if the time counter t = 3, the loop ends and pre3 is used as the final classification probability weight map; otherwise, let t = t + 1, and then the feature map Y t C×H×W and the initial knowledge feature graph Z C×H×W Enter the knowledge enhancement module together to enter the next cycle;

[0048] For example: If the time counter t = 1, then add 1 to t, Y1 C×H×W and the initial knowledge feature graph Z C×H×WEnter the knowledge enhancement module together and enter the next cycle; if the time counter t = 2, then add 1 to t. and the initial knowledge feature graph Z C ×H×W Enter the knowledge enhancement module together and enter the next loop; if the time counter t is equal to 3, end the loop;

[0049] Calculate the loss function value Loss of the spectral enhancement two-way coding model after this round of training:

[0050]

[0051] Among them, pre t represents the classification probability weight map obtained in the tth cycle, label represents the input label map, p ti Represents the classification probability weight map pre t The value of the i-th pixel in ,l i Represents the value of the i-th pixel in the label image, N represents the total number of pixels, N = m*n.

[0052] Finally, each set of training data is used to train the feature knowledge enhancement network. When the loss function converges, the training is stopped to obtain a trained feature knowledge enhancement network.

[0053] (3) Visual extraction of fine-grained objects in remote sensing images;

[0054] The remote sensing image to be extracted is cropped into m*n-sized blocks, and then input into the trained feature knowledge enhancement network to output the label values ​​0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23 corresponding to each fine-grained object in the remote sensing image, and then map the label values ​​to a color range to form a visual image.

[0055] Figure 6 The following are the experimental results, where (a) is the original image, (b) is the label image, (c) is the UNet experimental result, (d) is the spectral enhancement dual-path coding network experimental result, and (e) is the feature knowledge enhancement network experimental result. It can be seen from the comparison that (e) has the following advantages over (d): 1. It has better ability to distinguish and extract objects with higher similarity; 2. It rarely misclassifies the object category into other categories. In general, the feature knowledge enhancement network has a higher accuracy of fine-grained object extraction for remote sensing images than the spectral enhancement dual-path coding network.

[0056] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A method for extracting fine-grained objects from remote sensing images based on feature conversion and knowledge enhancement, characterized in that: The following steps are involved: (1) Construct a training dataset for training the feature knowledge enhancement network; (2) Build and train a feature knowledge enhancement network; (3) Use the trained feature knowledge to enhance the network for fine-grained object visualization extraction in remote sensing images.

2. The method for extracting fine-grained objects from remote sensing images based on feature conversion and knowledge enhancement according to claim 1 is characterized in that: The training data set is constructed as follows: (2.1) Download multiple remote sensing images and crop each remote sensing image into a block of size m*n; (2.2) Use semantic segmentation and annotation tools to mark fine-grained features in remote sensing images with different shapes, where fine-grained features include industrial areas, urban areas, rural areas, sports fields, squares, roads, overpasses, railway stations, airports, irrigated land, irrigated land, dry land, gardens, arbor forests, shrub forests, parks, natural meadows, artificial meadows, rivers, lakes, ponds, fish ponds, snowfields, and wastelands; (2.3) Set the pixel value corresponding to each fine-grained feature to 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, so as to generate a label image corresponding to each fine-grained feature, where the pixel value of the industrial area is set to 0, the pixel value of the urban area is set to 1, and so on; (2.4) Each remote sensing image and the corresponding label image are used as a set of training data to form a training data set.

3. The method for extracting fine-grained objects from remote sensing images based on feature conversion and knowledge enhancement according to claim 1 is characterized in that: The construction and training process of the feature knowledge enhancement network is as follows: A set of training data is used as the input of the feature knowledge enhancement network; The feature knowledge enhancement network starts with the encoder E of the feature conversion module. The encoder E contains three residual blocks and three 2x2 average pooling layers. Each residual block includes two convolution modules. The convolution module contains a 3x3 convolution layer, a batch normalization layer, and a Relu activation function. The two convolution modules form a residual link. After the original image passes through the first residual block and the 2x2 average pooling layer, the intermediate feature map is obtained. Intermediate feature map After the second residual block and the 2x2 average pooling layer, the intermediate feature map is obtained. Intermediate feature map After the third residual block and the 2x2 average pooling layer, the initial knowledge feature map Z is obtained. C×H×W ; Where C, H, W represent the number of channels, height and width of the initial knowledge feature map respectively; Then the initial knowledge feature map Z C×H×W Input into the knowledge enhancement module for knowledge enhancement operation; The knowledge enhancement module includes a loop counter t, a knowledge fusion module and a loop predictor; the loop counter t controls the knowledge fusion module and the loop predictor to loop three times to obtain the output of the knowledge enhancement module; Among them, the initial knowledge feature graph Z C×H×W After entering the knowledge enhancement module, the knowledge fusion module is first entered. At this time, the loop counter t first determines the number of loops. If t>1, Z C×H×W and Superimpose in the channel direction to obtain the feature map If t = 1, then Z C×H×W It is superimposed with a tensor of size C×W×H that is all 0 in the channel direction to obtain the feature map Then the feature map After two convolution modules and a 1×1 convolution layer, the fused feature map P is obtained. t C×H×W ; Fusion feature map P t C×H×W Then enter the loop predictor; fusion feature map P t C×H×W First enter an RBT module to get the feature map Then, the feature map is obtained by passing through a 2x2 average pooling downsampling layer, an ST module, and an RBT module. Then, the feature map is obtained by sequentially passing through a 2x2 average pooling downsampling layer, an ST module, and an RBT module. After a 2x2 average pooling downsampling layer and an ST module, the feature map is obtained. Then enter an RBT module and an upsampling layer to get the feature map and Enter the ST module together to get the feature map After an RBT module and an upsampling layer, the feature map is obtained and Enter an ST module together to get the feature map After an RBT module and upsampling layer, the feature map Y is obtained. t C×H×W ; Y t C×H×W Input to the decoder D of the feature conversion module; Y t C×H×W First and Z C×H×W Superimpose in the channel direction to obtain the feature map After passing through an upsampling layer and a residual block, it is combined with the feature map Superimpose in the channel direction to obtain the feature map Feature Map After passing through an upsampling layer and a residual block, it is combined with the feature map Superimpose in the channel direction to obtain the feature map Final feature map After an upsampling layer and a residual block, the stage classification probability weight map pre is obtained. t ; At this time, the time counter t is judged; if the time counter t = 3, the loop ends and pre3 is used as the final classification probability weight map; otherwise, let t = t + 1, and then the feature map Y t C×H×W and the initial knowledge feature graph Z C×H×W Enter the knowledge enhancement module together to enter the next cycle; Calculate the loss function value Loss of the spectral enhancement two-way coding model after this round of training: Among them, pre t represents the classification probability weight map obtained in the tth cycle, label represents the input label map, p ti Represents the classification probability weight map pre t The value of the i-th pixel in ,l i Represents the value of the i-th pixel in the label image, N represents the total number of pixels, N = m*n. Finally, each set of training data is used to train the feature knowledge enhancement network. When the loss function converges, the training is stopped to obtain a trained feature knowledge enhancement network.

4. The method for extracting fine-grained objects from remote sensing images based on feature conversion and knowledge enhancement according to claim 2 is characterized in that: The structure of the RBT module is: The RBT module consists of two input branches, one of which is the time step T. 1×C , T 1×C It is a vector of size 1×C with all values ​​t; the other input branch is the feature map; Assume that the feature map of an input branch is F1 C×H×W ; Time step T 1×C First, after two fully connected layers, we get T1 C×H×W ; F1 C×H×W First, after a convolution module, we get F1 C×H×W 、T1 C×H×W Add bit by bit, and we get After passing through a convolution module, the output of the RBT module is obtained 5. The method for extracting fine-grained objects from remote sensing images based on feature conversion and knowledge enhancement according to claim 2 is characterized in that: The structure of the ST module is: Assume that the input feature map of the ST module is F1 C×H×W and F1 C×H×W Through three parallel fully connected layers, we get Will and After point multiplication, softmax operation is performed to obtain and Perform matrix multiplication and then add it to F1 C×H×W Add together to get Through two parallel fully connected layers, we get and Through a fully connected layer Will and After point multiplication, softmax operation is performed to obtain and Perform matrix multiplication and then add it to F1 C ×H×W Add together to get the output of the ST module If the ST module has only one input, then 6. The method for extracting fine-grained objects from remote sensing images based on feature conversion and knowledge enhancement according to claim 1 is characterized in that: The remote sensing image fine-grained object visualization extraction process is as follows: The remote sensing image to be extracted is cropped into m*n-sized blocks, and then input into the trained feature knowledge enhancement network to output the label values ​​0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23 corresponding to each fine-grained object in the remote sensing image, and then map the label values ​​to a color range to form a visual image.

Images