Remote sensing scene classification method and system based on semantic perception and dynamic graph convolution
Through the method based on semantic perception and dynamic graph convolution, using deep residual network and graph convolution technology, the problem of insufficient utilization of multi-semantic information in high-resolution remote sensing images is solved, and higher classification accuracy is achieved and stability.
Patent Information
- Application Number
- CN202111267094.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-28
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-10-28
AI Technical Summary
In the high-resolution remote sensing image scene classification, traditional DCNNs are difficult to reasonably use multiple semantic information due to the single training labels, resulting in low classification accuracy and difficulty in fully mining image information by the attention mechanism.
Using a method based on semantic perception and dynamic graph convolution, multi-scale features are extracted by constructing a deep residual network feature pyramid, and the adaptive semantic recognition module generates regional indexes. The graph convolution is used to update features to capture the relationship between different semantics and realize the global Sexual feature extraction.
The accuracy and stability of remote sensing image scene classification is improved, the relationship between multiple semantic information in the image is fully considered, and the recognition and classification accuracy of features are enhanced.
Smart Images

Figure CN113989671B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to a remote sensing scene classification method and system based on semantic perception and dynamic graph convolution. Background Art
[0002] With the improvement of remote sensing image resolution, high-resolution remote sensing images can show more detailed land cover information. According to different land cover semantics, HRRS images can be divided into different scenes. In recent years, scene classification of HRRS images has become increasingly important because it can be applied to many remote sensing image applications, such as urban and rural planning, surface detection, etc. However, HRRS images have the characteristics of complex content, diverse semantics, multiple target scales, and large volume. These characteristics make HRRS scene classification a difficult and challenging task. Therefore, how to improve the classification accuracy of high-resolution remote sensing scenes has become a research hotspot in the field of remote sensing. According to the above characteristics of HRRS images, people have proposed a variety of HRRS scene classification methods. Among them, the two main parts of feature extractor and classifier play a vital role. The purpose of the feature extractor is to map the HRRS image into appropriate visual features. The classifier focuses on dividing the HRRS scene into different semantic categories.
[0003] Since traditional machine learning methods have good stability and high efficiency, manual features (such as texture features, spectral features, color features, shape features) and traditional classifiers (such as support vector machines, decision trees) are widely used. However, since manual features are difficult to fully describe the information of high-resolution remote sensing images, and traditional classifiers cannot match the information distribution of manual features well, the performance of high-resolution remote sensing image scene classification using traditional machine learning methods cannot meet expectations.
[0004] With the development of deep convolutional neural networks (DCNNs), remote sensing scene classification methods based on DCNNs are becoming more and more popular. Compared with hand-crafted features, the deep features learned by hierarchical DCNNs can fully describe HRRS images. At the same time, training the classifier based on DCNNs method with feature extractors enables the classifier to properly track the deep feature distribution. Therefore, more and more DCNN-based methods have been proposed for HRRS image scene classification tasks, and they have achieved impressive performance in various applications. Although DCNN-based methods have achieved great success in HRRS image scene classification, there is still room for improvement. Generally speaking, HRRS image scene classification is a single-label task. However, a pre-defined single label cannot fully describe the complex content of HRRS scenes. Taking the "bridge" scene as an example, in addition to the "bridge", some areas may be covered by "river", "car", "grass", or even "dense buildings". Due to the overly single label, DCNN-based methods will equally consider all semantics during feature learning, but fail to consider the relationship between different semantics. This will cause the information between different semantics to interfere with each other, thereby reducing the discriminability of the obtained features and further having a negative impact on the classification accuracy. In order to alleviate the above problems, researchers are accustomed to introducing attention mechanisms in DCNNs-based methods to encourage the network to pay more attention to important semantic areas. Therefore, the learned deep features can only focus on important semantics. However, although the attention mechanism can partially solve the problem of information interference, it cannot fully mine all the information in the HRRS image. At the same time, it is difficult to ensure that all attention areas are correct, which will directly affect the classification results. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a remote sensing scene classification method and system based on semantic perception and dynamic graph convolution in view of the deficiencies in the above-mentioned prior art, so as to solve the problem that due to the single training label, traditional DCNN is difficult to reasonably use the various semantic information in the remote sensing image to improve the classification accuracy.
[0006] The present invention adopts the following technical solutions:
[0007] The remote sensing scene classification method based on semantic perception and dynamic graph convolution includes the following steps:
[0008] S1. Construct a feature pyramid based on a deep residual network to complete the initial feature extraction and obtain feature F;
[0009] S2, construct an adaptive semantic recognition module, and obtain the regional index I of feature F through semantic recognition;
[0010] S3, according to the feature F obtained in step S1, the regional index I obtained in step S2 and the feature F are used to construct a graph node, and a feature information capture network based on dynamic graph convolution is obtained to obtain a global feature F*;
[0011] S4. Using the global feature F* obtained in step S3, the scene classification of the remote sensing image is realized.
[0012] Specifically, step S1 is as follows:
[0013] The remote sensing image is input into the feature pyramid based on the deep residual network to obtain four sets of multi-scale features C1, C2, C3, and C4. A feature fusion strategy is adopted to compress and purify the multi-scale features C1, C2, C3, and C4 into P1, P2, P3, and P4 using a convolution layer with a convolution kernel of 1×1. For the i-th feature fusion layer whose size changes from small to large, a bilinear interpolation method is used to purify the multi-scale features C1, C2, C3, and C4 with a size less than C1. i The features are upsampled; they are gradually cascaded to feature C1, and the convolutional layer is used to fuse the features after cascading C1 to obtain feature P1 containing multi-scale and multi-level information, and P1 is used as the final feature representation F generated by the feature pyramid.
[0014] Furthermore, P i Specifically:
[0015]
[0016] Among them, P i is the output of the i-th feature fusion layer, Conv k×k (·) represents a convolutional layer with a kernel size of k×k, Upsample(·) represents a bilinear interpolation upsampling operation, Represents a cascade operation of features.
[0017] Specifically, in step S1, the feature pyramid is a top-down network structure with horizontal links, and the deep residual network includes a first convolutional layer, a second maximum pooling layer, and four residual layers.
[0018] Specifically, step S2 is as follows:
[0019] Use manual or adaptive analysis methods to determine how many types of semantic information need to be identified, and the number of preset semantic categories is determined according to the complexity of the data set; after determining the number of semantic categories, pass feature F through a convolution layer and an optimization layer to generate feature F', the convolution layer converts the number of channels of feature F into the number of semantic categories, and the optimization layer performs numerical optimization on the convolved features; obtain the optimized feature F', the number of channels of the optimized feature F' is equal to the preset number of semantic categories; select a position with the largest value from the channel of each feature pixel, and use the maximum position as the semantic index to which the pixel belongs; when all feature pixels obtain a semantic index, generate a semantic region-level index I, and the region index I identifies the original feature F as different semantic regions.
[0020] Specifically, step S3 is as follows:
[0021] Divide feature F into N r The mean or maximum value of the features in the region is calculated in the channel to obtain the central feature. The central feature is used to represent each region as a graph node to obtain the graph node set F of the central features of all regions. g , construct the adjacency matrix A according to the graph convolution theory, and use the constructed graph node set F g , A, W construct a strategy to update the feature F, W is the linear transformation matrix of the feature, after all the feature points in the feature are updated, the feature F is updated to a feature that contains both local and global information, and the global feature F* is obtained.
[0022] Further, update each pixel f in feature F i * for:
[0023]
[0024] Among them, f i is the pixel of the original feature, σ(·) represents the activation function, A h represents the jth row of the adjacency matrix A, x i Represents the i-th graph node.
[0025] Furthermore, the adjacency matrix A is constructed as follows:
[0026]
[0027] Among them, Υ is a hyperparameter, x i ,x j represents two graph nodes, D(·) is a distance metric function, A ij is the value in the i-th row and j-th column of the adjacency matrix.
[0028] Specifically, step S4 is as follows:
[0029] A convolutional layer with a kernel function of 1×1 is used to process F*, and the output channel is equal to the number of semantic classes in the remote sensing scene dataset. The output of the output channel dimension is normalized using the Softmax function. The scene class corresponding to the maximum value on the output channel dimension is selected as the final label of the remote sensing scene to achieve scene classification of remote sensing images.
[0030] Another technical solution of the present invention is a remote sensing scene classification system based on semantic perception and dynamic graph convolution, comprising:
[0031] The extraction module constructs a feature pyramid based on a deep residual network to complete the initial feature extraction and obtain feature F;
[0032] Recognition module, builds an adaptive semantic recognition module, and obtains the regional index I of feature F through semantic recognition;
[0033] The construction module constructs graph nodes based on the feature F of the extraction module and the regional index I and feature F obtained by the recognition module, and obtains the global feature F* based on the feature information capture network of the dynamic graph convolution;
[0034] The classification module uses the global feature F* obtained by the construction module to realize the scene classification of remote sensing images.
[0035] Compared with the prior art, the present invention has at least the following beneficial effects:
[0036] The present invention is a remote sensing scene classification method based on semantic perception and dynamic graph convolution. Due to the complex and changeable semantic information in remote sensing images, the traditional convolutional neural network information capture method is difficult to consider the relationship between different semantics, and interference will occur between different semantic information, thus affecting the network accuracy. The method based on graph convolution can make up for this defect. By capturing the relationship between different semantic information, the remote sensing image can be fully interpreted to obtain accurate scene classification, and the relationship between multiple semantic information in the remote sensing image can be fully considered to obtain accurate scene classification results.
[0037] Furthermore, scenes are classified using features containing multi-scale and multi-level information. Each layer of features extracted by the deep residual network has information of different scales and semantic levels. The single information of a single layer of features is difficult to describe the remote sensing image as a whole. Therefore, the extracted multi-scale and multi-level features can better perform scene classification.
[0038] Furthermore, it is used to fuse features from different levels; upsampling features from different levels that are smaller than the features of the current level can enable features from different layers to be cascaded between channels, and convolution using a 3×3 convolution kernel can fuse features from different levels to obtain features with richer information.
[0039] Furthermore, rich features containing multi-scale and multi-semantic level information are extracted; each layer of features extracted by the deep residual network has information of different scales and semantic levels. By fusing features of different layers, image features rich in rich information can be obtained.
[0040] Furthermore, we can quickly capture different semantics in the image. Considering the principle that features with different semantics are different between channels and features with the same semantics are similar, we take the features with the same maximum value between channels as semantically similar features.
[0041] Furthermore, graph convolution is used to capture the relationship between different semantics to update the features, so that it has a more comprehensive description of the image. Graph convolution has a very powerful representation and information transmission ability for unstructured data. Using graph convolution to capture the relationship between different semantics can better describe the properties of the image to obtain a more accurate scene representation.
[0042] Furthermore, the features of the entire image are updated by updating each pixel; each pixel is considered as a graph node, and a graph is constructed with other semantic blocks except the semantic block where the pixel is located to update the features of the graph node.
[0043] Furthermore, the relationship between different semantics is constructed; the similarity between different semantic features is used to construct a relationship matrix, so that the network can generate a more consistent feature representation based on considering the relationship between different semantics.
[0044] Furthermore, the updated features are used to generate the final scene category; the features updated by graph convolution have a more comprehensive feature representation of the image, and using this feature for image classification will achieve better classification accuracy.
[0045] In summary, the present invention uses feature pyramid to extract multi-scale and multi-level features, uses adaptive semantic recognition module to construct graph nodes, uses graph convolution to update features to obtain more comprehensive and consistent features, and finally uses the features to obtain accurate classification results of remote sensing images. Compared with other current remote sensing scene classification methods, the present invention has more stable and accurate classification performance.
[0046] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 Schematic diagram of residual learning;
[0048] Figure 2 Schematic diagram of feature pyramid model based on deep residual network;
[0049] Figure 3 It is a schematic diagram of the adaptive semantic recognition module;
[0050] Figure 4 Schematic diagram of graph convolution operation;
[0051] Figure 5 Schematic diagram of feature update strategy based on dynamic graph convolution. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.
[0054] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.
[0055] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0056] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.
[0057] The present invention provides a remote sensing scene classification method based on semantic perception and dynamic graph convolution, using dense feature pyramid network (DFPN) to extract representative features; an adaptive semantic analysis module is proposed to obtain the semantic diversity of HRRS images. By analyzing the extracted features, the HRRS scene can be adaptively divided into different semantic regions; a feature update strategy is developed based on dynamic graph theory, and different semantic features are appropriately combined according to the relationship between them; finally, different semantic regions are determined.
[0058] The present invention provides a remote sensing scene classification method based on semantic perception and dynamic graph convolution, comprising the following steps:
[0059] S1. Construct a feature pyramid based on a deep residual network to complete the initial feature extraction and obtain feature F;
[0060] In the feature pyramid based on deep residual network, multiple simple feature fusions are used to enhance the features.
[0061] The deep residual network first proposed the concept of residual learning, and the residual block is expressed as:
[0062]
[0063] See also Figure 1 , the residual block is divided into two parts: the direct mapping part and the residual part; h(x l ) is a direct mapping, reflected in Figure 1 The middle is the curve on the right; is the residual part, which is generally composed of two or three weight layers, namely Figure 1 The left side of the figure contains the weight layer.
[0064] According to the hierarchical structure distribution, the deep residual network includes the first convolution layer, the second maximum pooling layer, and four residual layers. Therefore, after the remote sensing image is input into the deep residual network, features of different scales and multiple semantics can be mined in different layers of the deep residual network.
[0065] The four sets of features obtained from the four layers are denoted as C1, C2, C3, and C4 respectively;
[0066] Then the feature pyramid is applied to fuse these four features to generate multi-scale and multi-level features.
[0067] The feature pyramid adopts a top-down network structure with horizontal links, such as Figure 2As shown in Figure 2, the top-down dense link structure ensures that the network can generate multi-level and multi-scale information. The horizontal link can ensure that the location of features with multiple semantic information is accurate while strengthening the representation of information at different scales.
[0068] The feature fusion strategy is adopted. The features obtained from different levels of the feature pyramid usually have different dimensions, which makes the hierarchical information inaccurate when the features are fused. In order to ensure that the information at different levels can be obtained evenly, the convolution layer with a convolution kernel of 1×1 is used to compress and purify the multi-scale features C1, C2, C3, and C4 into P1, P2, P3, and P4; in order to fuse features of different spatial sizes and ensure that feature information is not lost, for the i-th feature fusion layer whose size changes from small to large, the size is less than C i The features of the top layer are gradually cascaded to the bottom layer features and convolutional layers are used to match these cascaded features to fuse multi-scale and multi-level features.
[0069] The specific formula is:
[0070] P4=Conv 1×1 (C4),
[0071]
[0072] Among them, Conv k×k (·) represents a convolutional layer with a kernel size of k×k, Upsample(·) represents a bilinear interpolation upsampling operation, Represents a cascade operation of features.
[0073] P1 is used as the final feature representation generated by the feature pyramid, which contains multi-scale and multi-level feature information. Therefore, feature F is obtained for the remote sensing image.
[0074] S2, construct an adaptive semantic recognition module, and obtain the semantic recognition index I of feature F through semantic recognition;
[0075] Based on the extracted rich features, different semantic information in the image is identified, and the corresponding positions of the rich features are marked according to the identification results; there are often complex semantics in remote sensing images, and the indiscriminate fusion of these complex semantics will make it difficult for the network to understand the image information. In general convolutional neural networks, as the degree of convolution continues to deepen, the mutual interference between different semantic information will become more obvious. In order to more appropriately fuse these complex semantic information, different semantics must first be distinguished.
[0076] The adaptive semantic recognition module can analyze the features of the image and adaptively generate regions with different semantics. According to these given semantic regions, more effective methods (such as graph convolution) can be used to fuse these semantic information. Therefore, the feature pyramid based on the deep residual network constructed in step S1 is first used to extract the image features F with strong representation. Based on the extracted features F, the following are used: Figure 3 The strategy generates the corresponding semantic region.
[0077] First, use manual or adaptive analysis methods to determine how many types of semantic information need to be identified. The number of preset semantic types (referred to as the semantic number) will be determined based on the complexity of the data set;
[0078] After determining the semantic number, feature F is passed through a convolution layer and an optimization layer to generate feature F'. Here, the convolution layer converts the number of channels of feature F into semantic numbers, and the optimization layer performs numerical optimization (such as normalization, regularization, etc.) on the convolutional features.
[0079] Then, the optimized feature F' is obtained, and the number of channels of feature F' is equal to the preset number of semantics; to assign a semantic label to each feature pixel, a position with the largest value is selected from the channel of each feature pixel, and this position will be used as the semantic index to which the pixel belongs. When all feature pixels obtain a semantic index, a semantic region-level index is generated. This semantic region index identifies the original feature F as different semantic regions.
[0080] During the network training process, the convolutional layers and optimization layers will be continuously trained to identify the semantic regions that are most effective for the target task.
[0081] S3, according to the feature F obtained in step S1, the regional index I obtained in step S2 and the feature F are used to construct a graph node, and a feature information capture network based on dynamic graph convolution is obtained to obtain a global feature F*;
[0082] Because the information obtained by general convolution operations is mainly local information, even after multiple maximum pooling operations, the feature information of farther distances has been captured as much as possible. However, the global information they contain is still insufficient, which affects the acquisition of image target information and the ability to capture large targets.
[0083] In order to better capture global information, graph convolution operation is introduced. At present, graph convolution is a very popular method, which has a very powerful ability in processing unstructured data. Graph convolution extracts global information by aggregating information from the neighborhood of graph nodes.
[0084] Assume the graph node set F g ={x1,x2,x3,…,x n}, use x i Represents the i-th graph node, and n is equal to the number of graph nodes. In order to describe the properties between graph nodes, graph convolution defines an adjacency matrix:
[0085]
[0086] Among them, Υ is a hyperparameter, x i ,x j Represents two graph nodes, N(x i ) represents the graph node x i The set of neighboring nodes.
[0087] In order to update the representation of the nodes, a learnable weight matrix is introduced and the l-th layer graph convolution operation is defined as follows:
[0088]
[0089] in, and denote the input and output of the lth layer, σ(·) denotes the activation function, and A (l) With W (l) Represents the adjacency matrix of layer l and the learnable weight matrix. This operation is shown in Figure 4 middle.
[0090] According to the basic graph convolution theory, a dynamic graph convolution strategy based on 2D data is proposed. The regional index I and feature F obtained in step S2 are used to construct graph nodes. Suppose feature F is divided into N r regions, each region will be represented as a graph node using the central feature. Here, the central feature can be calculated by calculating the mean or maximum value of the features in the region within the channel. The central feature of all regions can be represented as F g ={x1,x2,x3,…,x n}. In order to further capture the information between graph nodes, we will construct the adjacency matrix A based on graph convolution theory. The construction method of A is as follows:
[0091]
[0092] Among them, Υ is a hyperparameter, x i ,x j Represents two graph nodes, and D· is a distance metric function. In order to better describe the distance relationship between graph nodes, we plan to use the learnable Mahalanobis distance metric function to dynamically measure the distance between two graph nodes. Compared with directly using the Euclidean distance metric, the learnable Mahalanobis distance metric function has a more powerful distance metric scale that is more consistent with data distribution. It is defined as:
[0093]
[0094] Among them, M consists of a learnable matrix W d Multiply by its own transpose W d T Get, that is, MW d W d T Next, different from the traditional graph convolution strategy, we use the constructed graph node set F g , A, W construct a strategy that can update the feature F. This strategy can update each pixel in F according to the graph structure.
[0095] Specifically, for each pixel f in feature F i , if it belongs to the jth segmentation region, the following update strategy will be applied:
[0096]
[0097] Among them, f i With f i * are the original and updated features respectively, σ(·) represents the activation function, A j represents the jth row of the adjacency matrix A, x i represents the i-th graph node, and j∶· represents the j-th row of the matrix. After all feature points in the feature are updated, F will be updated to a feature that contains both local and global information. Figure 5 A graphical representation of the feature update process is shown in .
[0098] On the one hand, compared with using only deep residual networks, dynamic graph convolution can capture global and local information in multi-dimensional / multi-source images and fuse them into regional features with different semantics. This can increase the discriminative power of generated features. On the other hand, instead of directly updating the graph node set, we use the graph node set F g To update the obtained feature F. This plays a very important auxiliary role in the feature update. In short, dynamic graph convolution not only makes the updated features contain more global information, but also further optimizes the non-structural organization in the features, which is more conducive to the completion of our task. Finally, we will obtain the global feature F*.
[0099] S4. Using the global feature F* obtained in step S3, the scene classification of the remote sensing image is realized.
[0100] The updated visual feature F* contains rich discriminable information and can accurately identify remote sensing image scenes.
[0101] S401, in order to quickly and accurately obtain the semantic labels of the remote sensing scene, a convolutional layer with a kernel function of 1×1 is used to process F*, and the output channel is equal to the number of semantic classes in the remote sensing scene dataset;
[0102] S402, using the Softmax function to normalize the output of the channel dimension;
[0103] S403: Select the scene class corresponding to the maximum value in the channel dimension as the final label of the remote sensing scene.
[0104] During the training phase, only cross entropy is used as the loss function to train the network.
[0105] In another embodiment of the present invention, a remote sensing scene classification system based on semantic perception and dynamic graph convolution is provided. The system can be used to implement the above-mentioned remote sensing scene classification method based on semantic perception and dynamic graph convolution. Specifically, the remote sensing scene classification system based on semantic perception and dynamic graph convolution includes an extraction module, a recognition module, a construction module and a classification module.
[0106] Among them, the extraction module constructs a feature pyramid based on a deep residual network to complete the initial feature extraction and obtain the feature F;
[0107] Recognition module, builds an adaptive semantic recognition module, and obtains the regional index I of feature F through semantic recognition;
[0108] The construction module constructs graph nodes based on the feature F of the extraction module and the regional index I and feature F obtained by the recognition module, and obtains the global feature F* based on the feature information capture network of the dynamic graph convolution;
[0109] The classification module uses the global feature F* obtained by the construction module to realize the scene classification of remote sensing images.
[0110] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the remote sensing scene classification method based on semantic perception and dynamic graph convolution, including:
[0111] A feature pyramid based on a deep residual network is constructed to complete the preliminary extraction of features and obtain feature F. An adaptive semantic recognition module is constructed to obtain the regional index I of feature F through semantic recognition. According to feature F, graph nodes are constructed using the regional index I and feature F, and a feature information capture network based on dynamic graph convolution is used to obtain the global feature F*. The global feature F* is used to realize scene classification of remote sensing images.
[0112] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage.
[0113] The processor may load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the remote sensing scene classification method based on semantic perception and dynamic graph convolution in the above embodiment; the processor may load and execute the following steps:
[0114] A feature pyramid based on a deep residual network is constructed to complete the preliminary extraction of features and obtain feature F. An adaptive semantic recognition module is constructed to obtain the regional index I of feature F through semantic recognition. According to feature F, graph nodes are constructed using the regional index I and feature F, and a feature information capture network based on dynamic graph convolution is used to obtain the global feature F*. The global feature F* is used to realize scene classification of remote sensing images.
[0115] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0116] Validation analysis
[0117] (1) The deep residual network can effectively learn various useful information in complex content, and the feature pyramid can fuse multi-scale and multi-level features to construct features containing richer and more comprehensive information. Therefore, the feature pyramid constructed by the deep residual network can not only extract useful information from the image, but also further mine the potential multi-scale and multi-level semantic information in the image.
[0118] (2) In recent years, a large amount of work has proved that analyzing and selectively fusing different semantic information of an image plays a significant role in improving the representation of image features.
[0119] (3) Graph structures have a wide range of applications in the field of processing irregular data. The semantic information in high-resolution remote sensing images is usually complex and irregular. Therefore, using the different semantics in high-resolution remote sensing images to construct graph nodes and mine the corresponding boundary relationships to construct the corresponding graph structure can further improve the network's understanding of high-resolution remote sensing images.
[0120] (4) Based on traditional graph convolution, the proposed dynamic graph convolution uses semantic nodes in the image to update the features of the image, which not only reduces the amount of computation but also can establish a more accurate semantic graph structure to make the features more representative.
[0121]
[0122]
[0123]
[0124]
[0125]
[0126] In summary, the present invention provides a remote sensing scene classification method and system based on semantic perception and dynamic graph convolution. Based on different training and testing ratios, this method has achieved the highest accuracy on the three mainstream data sets of UCM, AID and NWPU, and obtained a stable accuracy representation (small standard deviation), which proves the effectiveness of the method.
[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0129] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0131] The above contents are only for explaining the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.
Claims
1. Remote sensing scene classification method based on semantic perception and dynamic graph convolution, It is characterized in that The following steps are involved: S1. Construct a feature pyramid based on a deep residual network to complete the initial feature extraction and obtain feature F; S2, construct an adaptive semantic recognition module, and obtain the regional index I of feature F through semantic recognition; S3, according to the feature F obtained in step S1, the regional index I obtained in step S2 and the feature F are used to construct the graph node, and the feature information capture network based on the dynamic graph convolution is obtained to obtain the global feature F*, specifically: Divide feature F into N r The mean or maximum value of the features in the region is calculated in the channel to obtain the central feature. The central feature is used to represent each region as a graph node to obtain the graph node set F of the central features of all regions. g , construct the adjacency matrix A according to the graph convolution theory, and use the constructed graph node set F g , A, W construct a strategy to update the feature F, W is the linear transformation matrix of the feature, after all the feature points in the feature are updated, the feature F is updated to contain both local and global information, and the global feature F* is obtained; Update each pixel in feature F for: Among them, f i is the pixel of the original feature, σ(·) represents the activation function, A j represents the jth row of the adjacency matrix A, x i represents the i-th graph node; The adjacency matrix A is constructed as follows: Among them, Υ is a hyperparameter, x i ,x j represents two graph nodes, D(·) is a distance metric function, A ij is the value of the i-th row and j-th column of the adjacency matrix; S4. Using the global feature F* obtained in step S3, the scene classification of the remote sensing image is realized.
2. The method according to claim 1, It is characterized in that Step S1 is specifically as follows: The remote sensing image is input into the feature pyramid based on the deep residual network to obtain four sets of multi-scale features C 1 ,C 2 ,C 3 ,C 4 ; Adopt the feature fusion strategy and use the convolution layer with a convolution kernel of 1×1 to transform the multi-scale feature C 1 ,C 2 ,C 3 ,C 4 Compression and purification for P 1 ,P 2 ,P 3 ,P 4 ; For the i-th feature fusion layer whose size changes from small to large, the bilinear interpolation method is used for the feature fusion layer with a size smaller than C i The features are upsampled; gradually cascaded to feature C 1 , and use convolutional layer fusion cascade C 1 The features after the calculation are obtained to obtain the features P containing multi-scale and multi-level information 1 , P 1 The final feature representation F generated as a feature pyramid.
3. The method according to claim 2, It is characterized in that P i Specifically: Among them, P o is the output of the i-th feature fusion layer, Conv k×k (·) represents a convolutional layer with a kernel size of k×k, Upsample(·) represents a bilinear interpolation upsampling operation, Represents a cascade operation of features.
4. The method according to claim 1, It is characterized in that In step S1, the feature pyramid is a top-down network structure with horizontal links, and the deep residual network includes the first convolutional layer, the second maximum pooling layer, and four residual layers.
5. The method according to claim 1, It is characterized in that Step S2 is specifically as follows: Use manual or adaptive analysis methods to determine how many types of semantic information need to be identified, and the number of preset semantic categories is determined according to the complexity of the data set; after determining the number of semantic categories, pass feature F through a convolution layer and an optimization layer to generate feature F', the convolution layer converts the number of channels of feature F into the number of semantic categories, and the optimization layer performs numerical optimization on the convolved features; obtain the optimized feature F', the number of channels of the optimized feature F' is equal to the preset number of semantic categories; select a position with the largest value from the channel of each feature pixel, and use the maximum position as the semantic index to which the pixel belongs; when all feature pixels obtain a semantic index, generate a semantic region-level index I, and the region index I identifies the original feature F as different semantic regions.
6. The method according to claim 1, It is characterized in that Step S4 is specifically as follows: A convolutional layer with a kernel function of 1×1 is used to process F*, and the output channels are equal to the number of semantic classes in the remote sensing scene dataset; the output of the output channel dimension is normalized using the Softmax function; The scene class corresponding to the maximum value on the output channel dimension is selected as the final label of the remote sensing scene to achieve scene classification of remote sensing images.
7. A remote sensing scene classification system based on semantic perception and dynamic graph convolution, It is characterized in that include: The extraction module constructs a feature pyramid based on a deep residual network to complete the initial feature extraction and obtain feature F; Recognition module, builds an adaptive semantic recognition module, and obtains the regional index I of feature F through semantic recognition; The construction module constructs the graph node based on the feature F of the extraction module and the regional index I and feature F obtained by the recognition module. The feature information capture network based on dynamic graph convolution obtains the global feature F*. Specifically: Divide feature F into N r The mean or maximum value of the features in the region is calculated in the channel to obtain the central feature. The central feature is used to represent each region as a graph node to obtain the graph node set F of the central features of all regions. g , construct the adjacency matrix A according to the graph convolution theory, and use the constructed graph node set F g , A, W construct a strategy to update the feature F, W is the linear transformation matrix of the feature, after all the feature points in the feature are updated, the feature F is updated to contain both local and global information, and the global feature F* is obtained; Update each pixel in feature F for: Among them, f i is the pixel of the original feature, σ(·) represents the activation function, A j represents the jth row of the adjacency matrix A, x i represents the i-th graph node; The adjacency matrix A is constructed as follows: Among them, Υ is a hyperparameter, x i ,x j represents two graph nodes, D(·) is a distance metric function, A ij is the value of the i-th row and j-th column of the adjacency matrix; The classification module uses the global feature F* obtained by the construction module to realize the scene classification of remote sensing images.
