A land use classification method based on deep spatiotemporal pattern interaction network

Abstract

The present invention discloses a land use classification method based on a deep spatiotemporal pattern interaction network, comprising the following steps: inputting a sample library of land use classification into a feature extraction module to obtain feature maps of different resolutions; inputting feature maps of different resolutions into a feature fusion module to obtain an enhanced feature map; inputting the enhanced feature map into a semantic segmentation module to calculate the confidence of semantic segmentation, and when the confidence is greater than or equal to a threshold, outputting a classification result; when the confidence of semantic segmentation is less than a threshold, outputting a pre-classification result and inputting the enhanced feature map into a spatiotemporal pattern reasoning module to perform reasoning verification of spatiotemporal relations to obtain a reasoning verification result; inputting the pre-classification result and the reasoning verification result into a correction module to adjust the pre-classification result and feed it back to the semantic segmentation module. The present invention improves the accuracy of land use classification and reduces the workload of later manual editing.

Description

Technical Field

[0001] The present invention relates to the technical field of land use classification, and in particular to a land use classification method based on a deep spatiotemporal pattern interaction network. Background Art

[0002] Land use classification technology based on deep learning and high-resolution remote sensing images has been widely used in large-scale land use surveys.

[0003] However, existing deep learning-based methods usually use an end-to-end approach to directly map images to feature space. Due to the characteristics of natural objects in remote sensing images, such as "same object, different spectrum, same spectrum, different object", and the differences caused by changes in remote sensing image imaging conditions, deep learning-based methods have poor generalization capabilities. At the same time, due to the lack of logical reasoning and the ability to express causal relationships in deep neural networks, they are insufficient in handling tasks with complex spatiotemporal correlations, and it is difficult to describe and explain the evolution of objects and phenomena in remote sensing images and their interactions.

[0004] It can be seen that although the land use classification technology based on deep learning and high-resolution remote sensing images has improved the efficiency and accuracy of land use classification and made great improvements in land use classification prediction methods, it still has shortcomings such as poor generalization ability, lack of interpretation and reasoning ability, and it is difficult to effectively utilize the spatiotemporal correlation of land use classification itself, making it difficult for deep learning technology to adapt to complex surface cover, especially for the superposition of multiple sources and multiple phases. The current practical interpretation method is still difficult to be separated from the participation of manual interpretation. Therefore, it is an urgent problem for technical personnel at this stage to realize spatial cognition and interactive reasoning of land use classification and obtain better interpretation results. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a land use classification method based on a deep spatiotemporal pattern interactive network, which can realize spatial cognition and interactive reasoning of land use classification and obtain better interpretation effect.

[0006] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows.

[0007] A land use classification method based on a deep spatiotemporal pattern interaction network includes constructing a deep spatiotemporal pattern interaction network model for land use classification, the model includes a feature extraction module, a feature fusion module, a semantic segmentation module, a spatiotemporal pattern reasoning module and a correction module, and the land use classification method includes the following steps:

[0008] S1. Obtain remote sensing image data and corresponding land use vector data, and perform pixel normalization processing based on the differences in color characteristics of the same type of land object;

[0009] S2. Create a land use classification sample library and confidence rule library;

[0010] S3. Conduct basic training on the feature extraction module, semantic segmentation module and spatiotemporal pattern reasoning module in the deep spatiotemporal pattern interaction network model;

[0011] S4. Input the sample library of land use classification into the feature extraction module to obtain feature maps with different resolutions;

[0012] S5. Inputting feature maps of different resolutions into a feature fusion module to obtain enhanced feature maps;

[0013] S6. Input the enhanced feature map into the semantic segmentation module, calculate the confidence of the semantic segmentation, and output the classification result when the confidence is greater than or equal to the threshold;

[0014] S7. When the confidence of the semantic segmentation is less than the threshold, the pre-classification result is output and the enhanced feature map is input into the spatiotemporal pattern reasoning module to perform reasoning verification of the spatiotemporal relationship and obtain the reasoning verification result;

[0015] S8. Input the pre-classification results and the reasoning verification results into the correction module, adjust the pre-classification results, feed them back to the semantic segmentation module, and re-execute step S6.

[0016] Preferably, in step S2, the classification sample library is randomly divided into a training data set and a validation data set.

[0017] Preferably, the basic stage training in step S3 includes a first stage of using the training data set to train a feature extraction module and a semantic segmentation module that perform well on the training data set; and a second stage of using the confidence rule base to train the spatiotemporal pattern reasoning module.

[0018] Preferably, the design of the feature fusion module in step S5 is used to structurally share, fuse and enhance feature maps of different resolutions. The backbone network preferably uses an HRnet network to improve the resolution of the feature map, and its output contains more detailed spatial information and global information to achieve global spatial correlation analysis. The specific method of enhancing the feature map includes:

[0019] The four branch features P output by the feature extraction module n , n∈{1,2,3,4} performs channel attention mechanism screening based on frequency domain analysis to obtain valuable feature information F of each branch n , n∈{1,2,3,4};

[0020] After bilinear interpolation and 1×1 convolution operations, F nThe feature maps of the last three branches in the above example are normalized in scale and dimension and reset to the same size as the first branch F. 1 ∈R 32×56×56 Output the same scale and number of channels to reduce the aliasing effect caused by later feature fusion and eliminate the differences in feature distribution between branch features;

[0021] Calculate the spatial weight parameter matrix between different branch features and compare the corresponding spatial weight parameter matrix with the adjusted feature F n→1 , n∈{2,3,4} performs channel matrix multiplication operation to obtain weighted features, and then sums the pixel values ​​of the features after the spatial weight size based on the channel dimension to obtain the fused feature H.

[0022] Preferably, after the semantic segmentation module in step S6 obtains the enhanced feature map output by the feature fusion module, the semantic segmentation module segments the enhanced feature map into a semantic segmentation map, and maps the semantic segmentation map to a two-dimensional space through convolution and upsampling operations to generate a segmentation result map, and then obtains the confidence of the prediction result through variational inference; the segmentation result map fuses multi-resolution feature information of multiple branches, and obtains rich feature map representation capabilities.

[0023] Preferably, the construction of the spatiotemporal pattern reasoning module in step S7 includes:

[0024] Offline establishment of confidence rule base R = {R1, R2, ... R L}, where R k ∈{θ,A,δ}, θ is the rule weight, A is the indicator set, δ is the attribute weight, and the confidence rule base is used to describe the feature categories, attributes and their relationships. The learning mechanism is used to optimize the parameters in the confidence rule base.

[0025] The semantic segmentation map obtained from the semantic segmentation module includes the input of N ground objects. The relational features of the entire target set of the nth object can be expressed as:

[0026]

[0027] Among them, f G is the geometric feature of the object, f A is the category of the object, ω mn represents the relationship weight between object n and other objects, and W represents the transformation matrix;

[0028] Aggregate the obtained relationship features with the rules in the confidence rule base, and the activation weight of the kth rule can be expressed as:

[0029]

[0030] Among them, a k and a i are the credibility of the kth rule and the ith rule on the attribute set, The i-th attribute in the k-th rule belongs to the indicator value Similarly, θ k and θ i are the rule weights of the kth rule and the ith rule, L is the number of rules in the confidence rule base, and N k is the number of attributes of the kth rule, δ i is the normalized attribute weight;

[0031] The obtained aggregation result is output as a confidence distribution, which indicates the degree to which the feature belongs to each class, and the class to which the feature belongs is the class with the highest confidence.

[0032] Preferably, the correction module in step S8 searches for and corrects erroneous classifications in the segmentation result map based on the results output by the spatiotemporal pattern reasoning module, obtains a classification correction result map, and feeds the classification correction result map back to the semantic segmentation module.

[0033] Due to the adoption of the above technical scheme, the technical progress achieved by the present invention is as follows.

[0034] The present invention establishes a spatiotemporal pattern reasoning network through spatiotemporal relationship modeling and knowledge integration. By integrating the bottom-up spatiotemporal pattern reasoning network into the existing deep learning framework for remote sensing image interpretation, the data-driven and knowledge-driven methods are combined and interacted, and the geometric and semantic features are comprehensively considered to achieve spatial cognition and interactive reasoning of land use classification, which can obtain better interpretation effects, thereby improving the accuracy of land use classification and reducing the workload of later manual editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] A land use classification method based on deep spatiotemporal pattern interaction network, combined with Figure 1 As shown, it includes constructing a deep spatiotemporal pattern interactive network model for land use classification. The model includes a feature extraction module, a feature fusion module, a semantic segmentation module, a spatiotemporal pattern reasoning module and a correction module. The land use classification method includes the following steps:

[0038] S1. Obtain remote sensing image data and corresponding land use vector data, and perform pixel normalization processing based on the differences in color characteristics of the same land object type.

[0039] S2. Create a land use classification sample library and confidence rule library.

[0040] Among them, the classification sample library is randomly divided into a training data set and a verification data set, and the classification sample library can be randomly divided into a training data set and a verification data set in a ratio of 4:1.

[0041] S3. Perform basic stage training on the feature extraction module, semantic segmentation module and spatiotemporal pattern reasoning module in the deep spatiotemporal pattern interaction network model.

[0042] The training in the basic stage includes the first stage of using the training data set to train a feature extraction module and a semantic segmentation module that perform well on the training data set. The trained feature extraction module and semantic segmentation module are verified through the validation data set to ensure the effectiveness of the model. The second stage uses the confidence rule base to train the spatiotemporal pattern reasoning module.

[0043] S4. Input the sample library of land use classification into the feature extraction module to obtain feature maps of different resolutions.

[0044] S5. Input feature maps of different resolutions into the feature fusion module to obtain enhanced feature maps.

[0045] The design of the feature fusion module is used to share, fuse and enhance the structure of feature maps with different resolutions. The backbone network preferably uses the HRnet network to improve the resolution of the feature map. Its output contains more detailed spatial and global information to achieve global spatial correlation analysis. The specific methods of feature map enhancement include:

[0046] The four branch features P output by the feature extraction module n , n∈{1,2,3,4} performs channel attention mechanism screening based on frequency domain analysis to obtain valuable feature information F of each branch n , n∈{1,2,3,4};

[0047] After bilinear interpolation and 1×1 convolution operations, F n The feature maps of the last three branches in the above example are normalized in scale and dimension and reset to the same size as the first branch F. 1 ∈R 32×56×56 Output the same scale and number of channels to reduce the aliasing effect caused by later feature fusion and eliminate the differences in feature distribution between branch features;

[0048] Calculate the spatial weight parameter matrix between different branch features and compare the corresponding spatial weight parameter matrix with the adjusted feature F n→1 , n∈{2,3,4} performs channel matrix multiplication operation to obtain weighted features, and then sums the pixel values ​​of the features after the spatial weight size based on the channel dimension to obtain the fused feature H.

[0049] S6. Input the enhanced feature map into the semantic segmentation module, calculate the confidence of the semantic segmentation, and output the classification result when the confidence is greater than or equal to the threshold.

[0050] After obtaining the enhanced feature map output by the feature fusion module, the semantic segmentation module divides the enhanced feature map into a semantic segmentation map, and maps the semantic segmentation map to the two-dimensional space through convolution and upsampling operations to generate a segmentation result map, and then obtains the confidence of the prediction result through variational inference; at the same time, the segmentation result map fuses the multi-resolution feature information of multiple branches, thus obtaining rich feature map representation capabilities.

[0051] S7. When the confidence of semantic segmentation is less than the threshold, the pre-classification result is output and the enhanced feature map is input into the spatiotemporal pattern reasoning module to perform reasoning verification of the spatiotemporal relationship and obtain the reasoning verification result.

[0052] The construction of the spatiotemporal pattern reasoning module includes:

[0053] Offline establishment of confidence rule base R = {R1, R2, ... R L}, where R k ∈{θ,A,δ}, θ is the rule weight, A is the indicator set, δ is the attribute weight, and the confidence rule base is used to describe the feature categories, attributes and their relationships. The learning mechanism is used to optimize the parameters in the confidence rule base.

[0054] The semantic segmentation map obtained from the semantic segmentation module includes the input of N ground objects. The relational features of the entire target set of the nth object can be expressed as:

[0055]

[0056] Among them, f G is the geometric feature of the object, f A is the category of the object, ω mn represents the relationship weight between object n and other objects, and W represents the transformation matrix;

[0057] Aggregate the obtained relationship features with the rules in the confidence rule base, and the activation weight of the kth rule can be expressed as:

[0058]

[0059] Among them, a k and a i are the credibility of the kth rule and the ith rule on the attribute set, The i-th attribute in the k-th rule belongs to the indicator value Similarly, θ k and θ i are the rule weights of the kth rule and the ith rule, L is the number of rules in the confidence rule base, and N k is the number of attributes of the kth rule, δ i is the normalized attribute weight;

[0060] The obtained aggregation result is output as a confidence distribution, which indicates the degree to which the feature belongs to each class, and the class to which the feature belongs is the class with the highest confidence.

[0061] S8. Input the pre-classification results and the reasoning verification results into the correction module, adjust the pre-classification results, feed them back to the semantic segmentation module, and re-execute step S6.

[0062] The correction module is used to integrate the trained deep learning semantic segmentation module with the spatiotemporal pattern reasoning module, construct a hierarchical semantic space for high-resolution remote sensing images, and perform training through feedback interaction.

[0063] The correction module searches for and corrects the incorrect classifications in the segmentation result map based on the output of the spatiotemporal pattern reasoning module, obtains the classification correction result map and feeds the classification correction result map back to the semantic segmentation module.

[0064] When the present invention is used, by integrating the bottom-up spatiotemporal pattern reasoning network into the existing remote sensing image interpretation deep learning framework, through the combination and interaction of data-driven and knowledge-driven, the accuracy of land use classification can be improved and the workload of later manual editing can be reduced.

Claims

1. A land use classification method based on deep spatiotemporal pattern interaction network, characterized by: The method includes constructing a deep spatiotemporal pattern interactive network model for land use classification. The model includes a feature extraction module, a feature fusion module, a semantic segmentation module, a spatiotemporal pattern reasoning module and a correction module. The land use classification method includes the following steps: S1. Obtain remote sensing image data and corresponding land use vector data, and perform pixel normalization processing based on the differences in color characteristics of the same type of land object; S2. Create a land use classification sample library and confidence rule library; S3. Conduct basic training on the feature extraction module, semantic segmentation module and spatiotemporal pattern reasoning module in the deep spatiotemporal pattern interaction network model; S4. Input the sample library of land use classification into the feature extraction module to obtain feature maps with different resolutions; S5. Inputting feature maps of different resolutions into a feature fusion module to obtain enhanced feature maps; S6. Input the enhanced feature map into the semantic segmentation module, calculate the confidence of the semantic segmentation, and output the classification result when the confidence is greater than or equal to the threshold; S7. When the confidence of the semantic segmentation is less than the threshold, the pre-classification result is output and the enhanced feature map is input into the spatiotemporal pattern reasoning module to perform reasoning verification of the spatiotemporal relationship and obtain the reasoning verification result; The construction of the spatiotemporal pattern reasoning module in step S7 includes: Offline building of confidence rule base ,in, , is the rule weight, is a set of indicators, is the attribute weight. The confidence rule base is used to describe the types of objects, attributes and their relationships. The learning mechanism is used to optimize the parameters in the confidence rule base. The semantic segmentation map obtained from the semantic segmentation module includes the input of N ground objects. , the relational features of the entire target set of the nth object can be expressed as: in, is the geometric feature of the object. is the category of the feature object, represents the relationship weight between object n and other objects, Represents the transformation matrix; aggregate the obtained relationship features with the rules in the confidence rule base, The activation weight of the rule can be expressed as: in, and They are Rules and The credibility of the rules on the attribute set, , It is Rule No. Attributes belong to the index value Similarly, ; and They are The rules and The rule weight of a rule, is the number of rules in the confidence rule base, It is The number of attributes of the rule, is the normalized attribute weight; The obtained aggregation result is output as a confidence distribution, which indicates the degree to which the feature belongs to each class, and the class to which the feature belongs is the class with the highest confidence. S8. Input the pre-classification results and the reasoning verification results into the correction module, adjust the pre-classification results, feed them back to the semantic segmentation module, and re-execute step S6.

2. A land use classification method based on deep spatiotemporal pattern interaction network according to claim 1, characterized in that: In step S2, the classification sample library is randomly divided into a training data set and a validation data set.

3. A land use classification method based on deep spatiotemporal pattern interaction network according to claim 2, characterized in that: The basic stage training in step S3 includes the first stage of using the training data set to train a feature extraction module and a semantic segmentation module that perform well on the training data set; and the second stage of using the confidence rule base to train the spatiotemporal pattern reasoning module.

4. The land use classification method based on deep spatiotemporal pattern interaction network according to claim 1, characterized in that: The design of the feature fusion module in step S5 is used to structurally share, fuse and enhance feature maps of different resolutions. The backbone network uses the HRnet network to improve the resolution of the feature map, and its output contains more detailed spatial information and global information to achieve global spatial correlation analysis. The specific method of enhancing the feature map includes: The four branch features output by the feature extraction module , Perform channel attention mechanism screening based on frequency domain analysis to obtain valuable feature information of each branch , ; After bilinear interpolation and 1×1 convolution operations, The feature maps of the last three branches in the above example are normalized in scale and dimension and reset to the same size as the first branch. Output the same scale and number of channels to reduce the aliasing effect caused by later feature fusion and eliminate the differences in feature distribution between branch features; Calculate the spatial weight parameter matrix between different branch features and compare the corresponding spatial weight parameter matrix with the adjusted features , Perform channel matrix multiplication to obtain weighted features, then sum the pixel values ​​of the features with spatial weights based on the channel dimension to obtain the fused features. .

5. The land use classification method based on deep spatiotemporal pattern interaction network according to claim 4 is characterized by: In step S6, after the semantic segmentation module obtains the enhanced feature map output by the feature fusion module, the semantic segmentation module segments the enhanced feature map into a semantic segmentation map, and maps the semantic segmentation map to a two-dimensional space through convolution and upsampling operations to generate a segmentation result map, and then obtains the confidence of the prediction result through variational inference; the segmentation result map fuses the multi-resolution feature information of multiple branches, and obtains a rich feature map representation capability.

6. The land use classification method based on deep spatiotemporal pattern interaction network according to claim 5 is characterized by: The correction module in step S8 searches for and corrects the incorrect classification in the segmentation result map according to the result output by the spatiotemporal pattern reasoning module, obtains the classification correction result map and feeds the classification correction result map back to the semantic segmentation module.

Images