Knowledge graph-based double-branch zero-sample remote sensing scene classification method

Abstract

The invention discloses a double-branch zero-sample remote sensing scene classification method based on a knowledge graph, which relates to the technical field of scene recognition, and automatically constructs a'scene-landscape-ground feature 'three-level knowledge graph by analyzing ground feature composition and spatial distribution information in a local area; on this basis, the designed double-branch supervision zero sample remote sensing scene classification network DBSS supervises mapping from the semantic vector to the visual space from a global branch and a local branch, so that the visual space can fully reflect a correlation structure contained in the semantic space; a large number of experiments carried out on UCM, AID and NWPU data sets show that the class average accuracy and the overall precision of classification of invisible class scenes by AKG-DBSS can reach 98% and 59.56% at most, the standard deviation is smaller than 6.91%, the method is remarkably superior to other methods in the prior art, and good expandability is achieved.

Description

Technical Field

[0001] The present invention relates to the field of scene recognition technology, and in particular to a dual-branch zero-sample remote sensing scene classification method based on knowledge graph. Background Art

[0002] One of the landmark tasks in the field of computer vision is to perform semantic analysis and understanding of high-resolution remote sensing images as a whole to achieve scene recognition. It has received widespread attention due to its great application potential in tasks such as urban mapping that explain land distribution and utilization at a semantic level. Nevertheless, since scenes are a type of abstract concept with rich semantic information, a scene usually has a many-to-many correspondence with different geographic objects, local landscapes, etc., which makes traditional pixel-level or object-level target recognition methods usually difficult to apply. Therefore, how to achieve higher-level, more abstract, and more discriminating scene interpretation is a very challenging problem faced by remote sensing scene classification tasks.

[0003] At present, deep learning models have demonstrated excellent performance in scene classification by relying on their powerful abstract representation capabilities. However, most of the existing deep learning-based methods belong to the category of supervised or semi-supervised learning. These methods usually require sufficient labeled samples for training, otherwise problems such as "overfitting" will occur, which will significantly reduce the classification accuracy. In practical applications, due to the richness and diversity of remote sensing scenes, accompanied by the continuous acceleration of global urbanization and the continuous emergence of new scene types, such methods often face the problem of lack of training samples or insufficient number of samples, which greatly reduces their usability.

[0004] In order to meet the above challenges, some researchers have proposed Few-Shot Learning (FSL) and Zero-Shot Learning (ZSL) methods for Remote Sensing Scene Classification (RSSC). Among them, FSL, as an important branch of machine learning, aims to build a machine learning model that can learn efficiently and predict accurately using only a small number of labeled samples. Compared with FSL, ZSL can get rid of the dependence on unseen class labeled samples, so it has stronger feasibility and applicability in RSSC tasks.

[0005] As a special transfer learning classification method, ZSL infers the prototype representation of the invisible class in the image feature space by transferring the knowledge in the annotated samples of the visible class, so that the model has the ability to recognize the invisible class samples. The existing ZSL methods in RSSC tasks generally use natural language models such as Word2vec, Bert, and PLM. Since these language models are trained based on generalized corpora, it is difficult to fully represent the correlation and differences between remote sensing scenes.

[0006] To this end, some scholars proposed Remote Sensing Knowledge Graph (RSKG) to refine the scene. RSKG describes the concept of the scene in a symbolic form and reveals the relationship between them, showing a strong ability of knowledge modeling and reasoning application in zero-shot scene classification tasks.

[0007] However, the existing knowledge graphs still have the following four limitations in practical applications: 1. The construction of knowledge graphs currently relies mainly on manual interpretation by experts. This method is not only easily affected by differences in prior knowledge, making it difficult to ensure its representativeness and reliability, but also inconvenient for updating and expanding the knowledge graph; 2. Existing knowledge graphs only use natural language to qualitatively describe the association relationship, but lack quantitative evaluation of the specific degree of association, and fail to fully explore the rich semantic information implied in the association relationship; 3. Existing knowledge graphs only focus on the association relationship between "scene-land features", but ignore the representative landscape composed of typical land feature combinations in the scene. In fact, the landscape-level features corresponding to the land feature composition and spatial distribution information in the local area have been proven to have strong discriminative power in scene classification applications; 4. Knowledge graphs are only used to generate semantic vectors, while ignoring the supervision of the visual space construction process. In ZSL, the visual space has a scene semantic correlation structure consistent with the semantic space, which has been proven to achieve better classification results. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a dual-branch zero-sample remote sensing scene classification method based on knowledge graph, comprising the following steps:

[0009] S1. Obtain a remote sensing urban scene sample data set, and select some categories from it as training sets, and the rest as test sets; the training set samples include category information, and the test set samples do not include category information;

[0010] S2. Construct a zero-shot recognition model AKG-DBSS. The zero-shot recognition model AKG-DBSS uses the ridge regression network as the basic network and the visual space as the embedding space.

[0011] S3. In the zero-shot recognition model, typical objects in the scene are extracted based on the pre-trained YOLO-V5 model. The spatial distribution of objects is analyzed to construct a set of landscape descriptions in the local area, and the three-level knowledge graph of scene, landscape and object is automatically constructed.

[0012] S4. In the zero-shot recognition model, global and local visual features are extracted based on VGGnet-16. At the same time, the TransH model is used to learn the representation of the knowledge graph to obtain the semantic vector corresponding to each scene.

[0013] S5. In the zero-shot recognition model, the scene-level semantic correlation matrix Simg and the landscape-level semantic correlation matrix Siml are extracted respectively;

[0014] S6. In the zero-shot recognition model, a dual-branch structure is used to map the semantic vector to the global and local visual feature spaces constructed by VGGnet-16, and the mapping process is supervised based on Simg and Siml. The zero-shot recognition model is trained using the training set by weighted fusion of the local and global branches.

[0015] S7, selecting multiple remote sensing scene images and inputting them into the trained zero-shot recognition model, obtaining scene classification results, comparing them with the actual categories, and performing reliability testing on the trained zero-shot recognition model;

[0016] S8. Quantitative evaluation of zero-shot recognition model;

[0017] S9. Input the unknown urban scene image into the zero-sample recognition model to obtain the category to which the unknown urban scene belongs.

[0018] The technical solution further defined in the present invention is:

[0019] Furthermore, in step S3, extracting typical objects in the scene based on the pre-trained YOLO-V5 model includes the following steps:

[0020] S3.1.1. Use the public dataset DOTA to pre-train the YOLO-V5 model, so that the YOLO-V5 model has the ability to recognize 15 typical objects in the city. The 15 typical objects are airplanes, ships, storage tanks, baseball fields, tennis courts, basketball courts, ground runways, ports, bridges, large vehicles, small vehicles, helicopters, roundabouts, football fields, and basketball courts.

[0021] S3.1.2. Obtain the typical object detection results corresponding to each image, and record {k, [(x max ,y max ), (x min ,y min), (X, Y)]}, which correspond to the object category output by the yolo-v5 model and the coordinate values ​​of the upper left corner, lower right corner, and center point of the object marking box.

[0022] In the aforementioned knowledge graph-based dual-branch zero-shot remote sensing scene classification method, in step S3, constructing a landscape description set in a local area includes the following steps:

[0023] S3.2.1. Select 9 spatial relationships R between typical objects r , which are on the surface, contained, surrounded, aligned, near, adjacent, overlapped, crossed, and far away, respectively; on this basis, for the objects k and k′ belonging to different categories, by analyzing the characteristics of each spatial relationship and ensuring that different spatial relationships are mutually exclusive, the corresponding discrimination rules for each spatial relationship are specified;

[0024] S3.2.2. Describing the index R by the degree of association CI Further quantitative evaluation of the strength of spatial relationships:

[0025]

[0026] Where H and W represent the height and width of the image respectively; X k , Y k With X k′ , Y k′ Then they represent the horizontal and vertical coordinate values ​​of the center points k and k′ respectively;

[0027] S3.2.3. In a certain scene, if there are multiple spatial relationships between two types of objects, only R CI Get the maximum value R CI_m A spatial relationship R corresponding to r_m ; If there are multiple spatial relationships at the same time, R CI_m , then randomly retain one of the spatial relationships; finally, we get a class of scenes corresponding to <k,R r_m , R CI_m , k′> is a set of local landscape descriptions.

[0028] In the aforementioned dual-branch zero-shot remote sensing scene classification method based on knowledge graph, in step S3, the public remote sensing scene dataset NWPU is used for testing. By extracting typical objects and constructing a set of local landscape descriptions, a three-level knowledge graph of scene-landscape-object is obtained, which is specifically represented as follows: a remote sensing scene includes multiple local landscapes; each local landscape includes multiple typical objects, spatial relationships between objects, and the degree of correlation; different remote sensing scenes are associated by having the same objects.

[0029] In the aforementioned dual-branch zero-shot remote sensing scene classification method based on knowledge graph, in step S4, visual features are extracted from both global and local aspects, the output of the last fully connected layer of VGGnet-16 is used as the global feature of the image, and a feature vector of size 1024×1 is output through the Linear layer, and finally the global visual feature set VG = [VG1, VG2, ..., VG N ]; the output of the last convolutional layer of VGGnet-16 is used as the local feature of the image, and the feature vector of size 1024×1 is output through the Linear layer, and finally the local visual feature set VL = [VL1, VL2, ..., VL N ].

[0030] In the aforementioned dual-branch zero-shot remote sensing scene classification method based on knowledge graph, in step S4, the TransH model is used to transform the automated scene-landscape-ground feature three-level knowledge graph ASLG-KG into a semantic vector set S = {s1, s2, ..., s T}, where s t (t=1, 2, ..., T) represents the semantic vector obtained for the t-th scene, and T represents the total number of scene categories, including visible and invisible categories.

[0031] The aforementioned dual-branch zero-shot remote sensing scene classification method based on knowledge graph, step S5 specifically includes the following sub-steps:

[0032] S5.1. Calculate the scene-level semantic relevance Simg by the following formula:

[0033]

[0034] Among them, s t and t′ They represent the semantic vectors of two different scenes respectively, and T represents the total number of scene categories, including visible and invisible categories;

[0035] S5.2. Construct a landscape-level coding matrix corresponding to a scene category:

[0036]

[0037] Where num represents the number of elements in the local landscape description set; for the i-th element, Q i , Q i ′ and QR i are k, k′ and R respectively r_m The corresponding one-hot encoded vector, Represents the concatenation of vectors; by replacing R CI_m With R r_mThe corresponding coding vectors are multiplied so that the landscape-level coding matrix is ​​used to reflect the local spatial relationships contained in the scene and the corresponding degree of association;

[0038] S5.3. Calculate the landscape-level semantic correlation Siml between scenes by the following formula:

[0039] Siml=1 T ·L t ·L t′ T 1, (t≠t′)

[0040] Among them, L t and L t′ They represent the landscape encoding matrices of two different scenes, with dimensions p and q respectively. The 1 on the left side of the formula represents a full-1 column vector with dimension p, and the 1 on the right side of the formula represents a full-1 column vector with dimension q.

[0041] In the aforementioned knowledge graph-based dual-branch zero-shot remote sensing scene classification method, step S6 specifically includes the following sub-steps:

[0042] S6.1, multiplying the semantic vector set S corresponding to the remote sensing scene by Simg and Siml respectively, and then mapping them to the global feature space and the local feature space respectively through the mapping function;

[0043] S6.2, the local branch and the global branch match the mapping results with the global and local visual features respectively;

[0044] S6.3. The local and global branches are fused by weighted summation.

[0045] In the above-mentioned knowledge graph-based dual-branch zero-shot remote sensing scene classification method, in step S6, the ridge regression function is used as the basis function to jointly train the local and global mapping branches under correlation supervision, and the following loss function is used to jointly train the mapping functions:

[0046]

[0047] Among them, W1 and W2 correspond to the mapping functions of the global and local branches respectively, Sg and Siml supervise the mapping process of the two branches, B is the training data batch, and α is the weighted fusion weight of the global and local branches.

[0048] In the aforementioned knowledge graph-based dual-branch zero-shot remote sensing scene classification method, in step S8, the zero-shot recognition model is quantitatively evaluated through overall accuracy, standard deviation, class average accuracy and confusion matrix indicators.

[0049] The beneficial effects of the present invention are:

[0050] (1) In this invention, starting from the sample itself, on the basis of extracting typical features, the landscape model is constructed by analyzing the features composition and spatial distribution information in the local area; then, combined with the spatial location information of each feature in the scene, a quantitative description model of the degree of association is proposed, thereby realizing the automatic construction of the "scene-landscape-feature" knowledge graph; at the same time, the construction and update of ASLG-KG do not require manual interpretation by experts, so it has good scalability;

[0051] (2) In the present invention, combined with ASLG-KG, this embodiment designs a dual-branch supervision module DBSS for semantic-visual mapping; this module constructs scene-level and landscape-level correlation matrices based on ASLG-KG respectively, and supervises the matching process of semantic vectors and visual features, thereby ensuring that the semantic correlation structure in the visual space is consistent with the semantic space; finally, by effectively weighted fusion of the scene-level and landscape-level mapping branches, the scene feature expression capability of the model is significantly enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0053] Figure 2 This is a schematic diagram of the network architecture of the AKG-DBSS model in an embodiment of the present invention;

[0054] Figure 3 This is a visualization diagram of ASLG-KG in an embodiment of the present invention;

[0055] Figure 4 Schematic diagram of dual-branch mapping matching under correlation supervision in an embodiment of the present invention;

[0056] Figure 5 Schematic diagrams of sample images of three data sets in embodiments of the present invention, wherein (a) is a schematic diagram of a sample image of UCM, (b) is a schematic diagram of a sample image of AID, and (c) is a schematic diagram of a sample image of NWPU;

[0057] Figure 6 Schematic diagram of the confusion matrix for classification of single-class scenes in an embodiment of the present invention, wherein (a) is a schematic diagram of the classification result of a single-class scene by UCM, (b) is a schematic diagram of the classification result of a single-class scene by AID, and (c) is a schematic diagram of the classification result of a single-class scene by NWPU;

[0058] Figure 7 0A schematic diagram of common dimensional vectors corresponding to different graphs in an embodiment of the present invention;

[0059] Figure 8 0A schematic diagram of different α values ​​in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] This embodiment provides a dual-branch zero-shot remote sensing scene classification method based on knowledge graph, such as Figure 1 As shown, the following steps are included:

[0061] S1. Obtain a remote sensing urban scene sample data set, and select some categories from it as a training set, and the rest as a test set; the training set samples include category information, and the test set samples do not include category information.

[0062] S2. Construct a zero-shot recognition model AKG-DBSS, namely the Automatic Knowledge Graph and Dual-Branch Semantic Correlation Supervision (AKG-DBSS), such as Figure 2 As shown in the figure, the zero-shot recognition model AKG-DBSS uses the ridge regression network as the basic network and the visual space as the embedding space; among them, the ridge regression network helps to reduce the overfitting phenomenon and has demonstrated good classification performance in the zero-shot learning (ZSL) task.

[0063] In the process of constructing the automated "Scene-Landscape-Ground Object" knowledge graph (ASLG-KG), firstly, typical objects in the scene are extracted, and the spatial distribution of objects is analyzed to describe the landscape in the local area, so as to realize the automated construction of the "Scene-Landscape-Ground Object" knowledge graph; in the dual-branch semantic-visual mapping supervision module (DBSS), the scene semantic vector S is obtained through TransH representation learning, and the scene-level semantic correlation Simg and the landscape-level semantic correlation Siml are extracted respectively; the semantic vectors are mapped to the global and local visual feature spaces constructed by VGGnet-16 respectively using a dual-branch structure, and the mapping process is supervised based on Simg and Siml to enable the visual space to fully retain the correlation structure of the semantic space; then, the dual-branch visual features are matched with the semantic vectors respectively; the scene-level and landscape-level mapping branches are effectively weighted fused to obtain the scene classification results.

[0064] S3. In the zero-shot recognition model, typical objects in the scene are extracted based on the pre-trained YOLO-V5 model. The spatial distribution of objects is analyzed to construct a set of landscape descriptions in the local area, thus realizing the automatic construction of a three-level knowledge graph of scene, landscape and object.

[0065] like Figure 2 As shown, firstly, typical objects are extracted from the original image to lay the foundation for the subsequent construction of a local landscape description set. The extraction of typical objects in the scene based on the pre-trained YOLO-V5 model includes the following steps:

[0066] S3.1.1. Use the public dataset DOTA to pre-train the YOLO-V5 model, so that the YOLO-V5 model has the ability to recognize 15 typical objects in the city. The 15 typical objects are airplanes, ships, storage tanks, baseball fields, tennis courts, basketball courts, ground runways, ports, bridges, large vehicles, small vehicles, helicopters, roundabouts, football fields and basketball courts.

[0067] S3.1.2. Obtain the typical object detection results corresponding to each image, and record {k, [(x max ,y max ), (x min ,y min ), (X, Y)]}, which correspond to the object category output by the yolo-v5 model and the coordinate values ​​of the upper left corner, lower right corner, and center point of the object marking box.

[0068] Based on the extracted typical objects, the local landscape is described using the object composition and spatial distribution information in the scene, which specifically includes the following steps:

[0069] S3.2.1. Spatial relationship judgment: Based on the reference to the existing knowledge graph, this embodiment selects 9 types of spatial relationships R between typical objects. r , specifically including on the surface, contained, surrounded, aligned, near, adjacent, overlapped, crossed and far away; on this basis, for the features k and k′ belonging to different categories, by analyzing the characteristics of each spatial relationship and ensuring the mutual exclusivity between different spatial relationships, the corresponding discrimination rules for each spatial relationship are specified, as shown in Table 1 below.

[0070] Table 1 Spatial relationship and discrimination rules

[0071]

[0072] In Table 1 above, k and k′ represent different types of features; (x max ,y max ) and (x min ,y min) are the coordinate values ​​of the upper left corner and lower right corner of the feature marking box output by the yolo-v5 model, and S represents the number of pixels contained in the feature.

[0073] S3.2.2. Correlation Degree Extraction: This embodiment designs a correlation degree description index R CI To further quantitatively evaluate the strength of spatial relationships:

[0074]

[0075] Where H and W represent the height and width of the image respectively; X k , Y k With X k′ , Y k′ They represent the horizontal and vertical coordinate values ​​of the center points k and k′ respectively.

[0076] S3.2.3. Construction of local landscape description set: In a certain scene, if there are multiple spatial relationships between two types of features, only R CI Get the maximum value (denoted as R CI_m , that is, the spatial relationship R corresponding to the strongest correlation r_m ; If there are multiple spatial relationships at the same time, R CI_m , then randomly retain one of the spatial relationships; finally, we get a class of scenes corresponding to <k,R r_m , R CI_m , k′> is a set of local landscape descriptions.

[0077] Taking the classic public remote sensing scene dataset NWPU as an example, we conducted a test and extracted typical objects and constructed a local landscape description set. The obtained “scene-landscape-object” three-level knowledge graph ASLG-KG visualization results are shown below: Figure 3 As shown in the figure, it can be specifically expressed as follows: a scene has multiple local landscapes; local landscapes are composed of multiple typical objects, spatial relationships between objects, and the degree of correlation; different scenes are related because they have the same objects; Figure 3 The straight lines represent the inclusion relationship between scene and landscape, and landscape and objects. The dashed circle is a schematic diagram of the local landscape of the scene, and the two sides of the arrow in the landscape show the association relationship R between any two objects. r_m Correlation R CI_m .

[0078] It can be seen that ASLG-KG can automatically construct the three-level knowledge graph of "scene-landscape-landscape" based on the samples themselves without relying on expert manual interpretation; it should be pointed out that the pre-trained DOTA dataset used in this embodiment and the NWPU dataset used for testing are only used to illustrate the construction process and effect of ASLG-KG; in fact, ASLG-KG is a general and well-scalable Remote Sensing Scene Knowledge Graphs (RSKG) construction method. Users can choose appropriate datasets according to the needs of actual applications to build or expand ASLG-KG.

[0079] The DBSS designed in this embodiment mainly includes three parts: visual feature and semantic vector extraction, semantic relevance matrix extraction, and dual-branch mapping under relevance supervision.

[0080] S4. Visual feature and semantic vector extraction: Visual features are extracted from global and local aspects to correspond to the scene-level and landscape-level semantic information in ASLG-KG respectively. In this embodiment, the output of the last fully connected layer of VGGnet-16 is used as the global feature of the image, and a feature vector of size 1024×1 is output through the Linear layer, and finally a global visual feature set VG = [VG1, VG2, ..., VG N ]; In addition, the output of the last convolutional layer of VGGnet-16 is used as the local feature of the image. Similarly, the feature vector of size 1024×1 is output through the Linear layer, and finally the local visual feature set VL = [VL1, VL2, ..., VL N ].

[0081] In semantic vector extraction, considering that the representation learning model TransH has the advantages of translation invariance, this embodiment uses the TransH model to transform ASLG-KG into a semantic vector set S = {s1, s2, ..., s T}, where s t (t=1, 2, ..., T) represents the semantic vector obtained for the t-th scene, and T represents the total number of scene categories, including visible and invisible categories.

[0082] S5, semantic relevance matrix extraction: extracting the scene-level semantic relevance matrix Simg and the landscape-level semantic relevance matrix Siml respectively, specifically including the following sub-steps:

[0083] S5.1. Calculate the scene-level semantic relevance Simg by the following formula:

[0084]

[0085] Among them, s t and t′ They represent the semantic vectors of two different scenes respectively, and T represents the total number of scene categories, including visible and invisible categories.

[0086] S5.2. Construct a landscape-level coding matrix corresponding to a scene category:

[0087]

[0088] Where num represents the number of elements in the local landscape description set; for the i-th element, Q i , Q i ′ and QR i are k, k′ and R respectively r_m The corresponding one-hot encoded vector, Represents the concatenation of vectors; by replacing R CI_m With R r_m The corresponding coding vectors are multiplied to make the landscape-level coding matrix reflect the local spatial relationships contained in the scene and the corresponding degree of association.

[0089] S5.3. Calculate the landscape-level semantic correlation Siml between scenes by the following formula:

[0090] Siml=1 T ·L t ·L t′ T 1, (t≠t′)

[0091] Among them, L t and L t′ They represent the landscape encoding matrices of two different scenes, with dimensions p and q respectively. The 1 on the left side of the formula represents a full-1 column vector with dimension p, and the 1 on the right side of the formula represents a full-1 column vector with dimension q. Finally, the correlation matrices Simg and Siml are output to represent the landscape-level semantic correlation and scene-level semantic correlation between any two scenes.

[0092] S6. Dual-branch mapping under correlation supervision: Using the extracted Simg and Siml for supervision, the semantic vectors are mapped to the local and global embedding spaces respectively through two branches and matched with the visual features.

[0093] like Figure 4 As shown, step S6 specifically includes the following sub-steps:

[0094] S6.1. Multiply the semantic vector set S corresponding to the remote sensing scene by Simg and Siml respectively, and then map them to the global feature space and the local feature space respectively through the mapping function to fully retain the semantic correlation structure contained in ASLG-KG.

[0095] S6.2, the local branch and the global branch match the mapping results with the global and local visual features respectively.

[0096] S6.3. The local and global branches are fused by weighted summation, and the weight parameter α is set to 1.

[0097] On this basis, this paper adopts the ridge regression function with improved model generalization performance as the basic function, and then jointly trains the local and global mapping branches under correlation supervision, and uses the following loss function to jointly train the mapping function:

[0098]

[0099] Among them, W1 and W2 correspond to the mapping functions of the global and local branches respectively, Simg and Siml supervise the mapping process of the two branches, B is the batch-size of training data, and α is the weighted fusion weight of the global and local branches.

[0100] S7. Select multiple remote sensing scene images and input them into the trained zero-shot recognition model to obtain scene classification results, compare them with the actual categories, and perform reliability testing on the trained zero-shot recognition model.

[0101] The experiment uses three popular public remote sensing scene datasets, namely UC Merced (UCM), Aerial Image Dataset (AID) and NWPU-RESISC45 (NWPU). The reliability of the trained zero-shot recognition model is tested on the three datasets of UCM, AID and NWPU respectively.

[0102] Among them, UCM is taken from the U.S. Geological Survey National Map Urban Area Image Series, and provides 21 categories of typical urban scene labeled samples including intersection and dense_residential; AID is released by Huazhong University of Science and Technology and Wuhan University, and is a large-scale aerial image dataset constructed by collecting samples from Google map images; NWPU is a public dataset released by Northwestern Polytechnical University, and its different scene samples have significant differences in translation, spatial resolution, etc.; The use of these three datasets is helpful to analyze the performance of the method of this embodiment from different angles, and the detailed parameter comparison of each dataset is shown in Table 2 below.

[0103] Table 2 Comparison of experimental data sets

[0104]

[0105] In order to fully verify the performance of the AKG-DBSS method, this embodiment selects four advanced zero-shot remote sensing scene recognition methods for comparison, namely, Variational Autoencoder with Generative Adversarial Network (VAE-GAN), Multilevel Cross-Modal Feature Alignment method (MCFA), Local-Global Feature Fusion and Weight Mapping Loss method (LGFFWM) and Adaptive Relation Aware Network (ARAN); these methods are designed for remote sensing scene recognition tasks from different perspectives, and comparative experiments are helpful for a comprehensive and objective evaluation of AKG-DBSS.

[0106] Among them, VAE-GAN introduces GAN to learn a reconstruction quality measurement method suitable for VAE, and then proposes a cross-modal feature matching loss CMFM-Loss to promote the deep alignment of visual features and semantic features of each scene; MCFA proposes a multi-level feature alignment method to fully explore the comparative relationship between cross-modal features contained in the scene; LGFFWM improves the model's ability to identify different scenes by fusing local and global features. On this basis, a weight mapping loss (WM Loss) function is proposed to force the model to prioritize learning visible scenes that have a strong correlation with unseen scenes; ARAN proposes an adaptive perception network to learn the inter-class and intra-class relationships of different categories in the dataset; these methods start from different angles and make targeted designs for remote sensing scene recognition tasks, and comparative experiments are helpful to make a comprehensive and objective evaluation of AKG-DBSS.

[0107] To ensure the fairness of the experiment, the experimental environment was built under the Ubuntu 16.04 system, and the Pytorch-1.3.1 framework was used. The hardware configuration was Nvidia GeForce RTX 2080ti GPU with 11GB RAM. The target recognition was based on the Yolo-v5 network and pre-trained with the DOTA dataset. The img-size was set to 1024*1024, and 16 was used as the batch-size for training 100 epochs. At the same time, the comparative experiments used VGGnet to obtain 1024-dimensional visual feature vectors, and obtained 300-dimensional semantic feature vectors based on TransH. During the training process, the learning rate of the Adam optimizer was set to 0.00001, and the mini-batch size was set to 16.

[0108] Finally, in order to objectively compare with the existing methods, according to the above experimental settings, UCM, AID and NWPU in the experiment respectively use four different visible class / invisible class division ratios; at the same time, it is ensured that there is no overlap between the visible class and the invisible class. The specific division is shown in Table 3 below; In addition, this embodiment shows how Figure 5 Some sample images on the three datasets shown.

[0109] Table 3. Dataset visible class / invisible class division table

[0110]

[0111] S8. The zero-shot recognition model is quantitatively evaluated by overall accuracy (OA), standard deviation (SD), class average accuracy (CA), and confusion matrix (CM) indicators; OA is a direct measure of the classification accuracy of the model on the entire dataset, as shown in the following formula:

[0112]

[0113] Among them, N t and N f They represent the number of correctly classified and incorrectly classified samples respectively; SD reflects the degree of discreteness of OA; CM illustrates the predicted class and actual class of the sample through rows and columns, so it can analyze the positive and false classification of different categories in detail.

[0114] In addition, CA provides an indicator to measure the average accuracy of the model in classifying various scene categories, as shown in the following formula:

[0115]

[0116] in, represents the number of samples correctly classified in the i-th category, Represents the total number of samples in the i-th category.

[0117] Based on the three sets of data sets of UCM, AID, and NWPU, the accuracy evaluation results of AKG-DBSS and the comparison methods are shown in Tables 4-6 below. The underlined bold in the table indicates the best result. The highest OA of AKG-DBSS in the three sets of data can reach 59.56%, 53.48%, and 47.84%, respectively, which are significantly better than other comparison methods.

[0118] Table 4 0A (%) and SD (%) of AKG-DBSS method and state-of-the-art method on UCM

[0119]

[0120] Table 5 0A (%) and SD (%) of AKG-DBSS method and state-of-the-art method on AID

[0121]

[0122] Table 6 0A (%) and SD (%) of AKG-DBSS method and state-of-the-art method on NWPU

[0123]

[0124] VAE_GAN, MCFA and ARAN all ignored the highly discriminative local features in the scene recognition task during training. In the experiments on the three datasets, their overall accuracy was reduced by more than 4.04% compared with AKG-DBSS. Although LGFFWM, which performed second best, considered the dual-branch structure, it ignored the potential correlation between different scenes. In comparison, AKG-DBSS not only considered the fusion of local and global visual features, but also introduced semantic correlation between scenes to supervise the mapping process, and its overall performance was relatively better in the experiment.

[0125] In the UCM dataset, the overall accuracy of AKG-DBSS is only 2.32% lower than that of LGFFWM at a visible / invisible class ratio of 10 / 11, while higher accuracy is achieved in the other three ratios. In the AID dataset experiment, although AKG-DBSS is better than the comparison method in OA only at visible / invisible class ratios of 25 / 5 and 15 / 15, its SD performance under the four visible / invisible class ratios is the best, that is, it has better stability. In the NWPU dataset experiment, compared with LGFFWM, although AKG-DBSS's 0A is reduced by 2.43% and 1.8% under visible / invisible class ratios of 30 / 15 and 20 / 25, respectively, its SD is reduced by more than 0.07% under the four ratios. In summary, in the three groups of experiments, AKG-DBSS shows better performance in overall accuracy and stability than other comparison methods.

[0126] Comparing the three data sets, the UCM data set only contains 21 types of scenes, and the number of samples for each type of scene is only 100, which is significantly less than the AID data set and the NWPU data set. Under this condition, AKG-DBSS and LGFFWM both adopt a dual-branch structure, which is more than 10.09% higher than the 0A of VAE-GAN, MCFA, and ARAN. This also shows that the combination of global and local features is discriminative in scene recognition tasks, especially suitable for applications with small samples.

[0127] The number of samples of each scene in the AID dataset is between 220 and 420, so there is an imbalance problem in the sample data during the training process. Based on this dataset, AKG-DBSS improves OA by more than 1.62% compared with the other comparison methods. Therefore, the semantic relevance supervision strategy introduced by AKG-DBSS can prompt the model to pay more attention to the learning of "strongly related" samples, effectively alleviating the impact of the sample imbalance problem; however, when the proportion of visible class samples is low (such as 10 / 20), the semantic relevance of AKG-DBSS is difficult to be fully expressed, resulting in AKG-DBSS's OA being 1.75% lower than LGFFWM.

[0128] The NWPU dataset has 45 types of scenes, and each scene has 700 samples, which helps to train the model more fully; the detection effects of AKG-DBSS and the comparison method are generally improved compared with the first two datasets, but AKG-DBSS still achieved better performance.

[0129] To further analyze the classification performance of AKG-DBSS for each invisible class scene, the confusion matrix was used in combination with the CA indicator to conduct experiments. The three datasets of UCM, AID, and NWPU were divided into three groups according to the mainstream method, with the ratio of visible class to invisible class set to 16 / 5, 25 / 5, and 35 / 10. The classification results of single-class scenes are shown in Figure 2. Figure 6 shown.

[0130] like Figure 6 As shown in the figure, for UCM, the classification accuracy for "golf course", "medium-sized residence" and "storage tank" is relatively high, all reaching more than 60%; for AID, the classification accuracy for "desert" and "industrial area" is more than 90%, while "pond" and "forest" are easily misclassified; and for NWPU, "airplane", "basketball court" and "sparse residential area" are more likely to be accurately classified.

[0131] In order to further verify the effectiveness of the proposed ASLG-KG, the currently very popular remote sensing scene knowledge graph semantic representation remote sensing knowledge graph (Semantic Representation of Remote Sensing Knowledge Graph, SR-RSKG) is selected for comparison; the knowledge graph (KG) is manually constructed based on expert knowledge, specifically including 70 scenes, 117 entity vectors, and 26 relationship vectors; this embodiment selects 21, 30 and 45 scene semantic vectors shared by ASLG-KG and SR-RSKG, and then experiments on UCM, AID and NWPU data sets respectively; on this basis, in order to further discuss the impact of different semantic vector dimensions on 0A, this embodiment conducts comparative analysis from the three vector dimensions of 100, 300 and 500, and adopts the same visible / invisible class division ratio described in the previous text. The specific evaluation results are as follows Figure 7 shown.

[0132] like Figure 7 As shown in the figure, compared with SR-RSKG, ASLG-KG improves OA by 6.3%, 4.6% and 3.9% in the three data sets respectively; therefore, compared with the knowledge graph manually constructed based on expert knowledge, ASLG-KG not only has a high degree of automation, but also shows a stronger scene representation ability; in addition, with the gradual increase of vector dimension from 100, 300 to 500, OA generally shows a trend of first rising and then falling; when the vector dimension is 300, the best classification performance is achieved; we believe that when the dimension is low, the important feature information of the semantic vector is not fully expressed, and it is difficult to distinguish the subtle differences between semantics; and when the vector dimension is too high, redundant and interfering information is generated in the semantic vector, resulting in a decrease in accuracy.

[0133] In order to verify the effectiveness of the DBSS dual-branch strategy, the corresponding OAs are trained separately for the global and local branches under the supervision of semantic relevance, and compared with the OAs obtained by joint training of the two branches. The results are shown in Table 7 below, where √ represents the embedding of the module and the underlined bold indicates the best result.

[0134] Table 7 DBSS dual-branch strategy effectiveness analysis table

[0135]

[0136] It can be seen that similar conclusions were obtained for the three datasets: compared with the mapping of a global or local branch, the dual-branch strategy in DBSS improves 0A by more than 14.68%; this fully demonstrates that in the zero-shot remote sensing scene classification (Zero-shot RSSC) task, it is feasible and necessary to further introduce a local correlation mapping branch based on the global supervision mapping branch, and the two have a complementary effect.

[0137] In order to verify the effectiveness of semantic relevance supervision in DBSS, this embodiment introduces scene-level semantic relevance (Simg) and landscape-level semantic relevance (Siml) separately or simultaneously for supervision under the dual-branch strategy, and compares the results with those without supervision. The effectiveness is evaluated by 0A. The specific results are shown in Table 8 below, where √ represents the embedding of the module, and the underlined bold indicates the best result.

[0138] Table 8 Correlation supervision effectiveness analysis table

[0139]

[0140] It can be seen that compared with no supervision, the introduction of scene-level and landscape-level semantic relevance supervision alone or simultaneously has improved the classification accuracy of the three datasets by more than 1%. This fully demonstrates that the introduction of relevance supervision can enable the visual space to fully reflect the semantic relevance structure contained in ASLG-KG, which in turn helps to significantly improve 0A; at the same time, it verifies that scene-level semantic relevance supervision has a greater impact on 0A, while landscape-level semantic relevance plays a complementary role.

[0141] In the construction of DBSS, the parameter α reflects the weight in the weighted fusion process of local and global branches. In order to analyze the impact of different changes in α on the classification of the dual-branch network, this embodiment sets α to increase in intervals of 0.5 and uses 0A as the evaluation index. The specific results are as follows: Figure 8 shown.

[0142] like Figure 8As shown in the figure, as the value of α increases, the 0A obtained by the three data sets on AKG-DBSS generally shows a trend of first increasing and then gradually decreasing; we believe that this is because in the initial stage, the local branches complement the expression of the global branches, so that 0A shows an upward trend; then, when the proportion of local branches is too high, it is easy to suppress the expression of the global branch, so that 0A shows a downward trend; in addition, when α=1, the optimal or suboptimal effect is achieved in all three groups of experiments. For this reason, this embodiment recommends setting α to 1 in practical applications.

[0143] S9. Input the unknown urban scene image into the zero-sample recognition model to obtain the category to which the unknown urban scene belongs.

[0144] This embodiment proposes a dual-branch zero-shot remote sensing scene classification method AKG-DBSS based on knowledge graph. Different from the traditional manual labeling method, ASLG-KG can automatically construct a three-level knowledge graph of "scene-landscape-feature" by analyzing the composition and spatial distribution information of objects in the local area of ​​the scene; on this basis, DBSS supervises the mapping of semantic vectors to visual space from two branches, global and local, respectively, so that the visual space can fully reflect the correlation structure contained in the semantic space; finally, the scene classification result is obtained through weighted fusion.

[0145] A large number of experiments on UCM, AID and NWPU datasets show that AKG-DBSS can achieve a 0A of up to 59.56% for the classification of invisible scenes, with a standard deviation of less than 6.91%, which is significantly better than four advanced comparative methods; in addition, compared with SR-RSKG based on expert knowledge, the proposed ASLG-KG can improve the accuracy by more than 4.8%; finally, the dual-branch fusion strategy and the introduction of semantic relevance supervision in DBSS can improve 0A by more than 14.68% and 8.98%, respectively.

[0146] The ASLG-KG proposed in this embodiment can start from the sample itself, extract typical objects, and construct a landscape model by analyzing the composition and spatial distribution information of objects in the local area. Then, combined with the spatial location information of each object in the scene, a relationship and degree description model (RDD) is proposed, thereby realizing the automatic construction of the "scene-landscape-object" knowledge graph. At the same time, the construction and update of ASLG-KG do not require manual interpretation by experts, so it has good scalability.

[0147] In combination with ASLG-KG, this embodiment designs a dual-branch supervision module (DBSS) for semantic-visual mapping; this module constructs scene-level and landscape-level correlation matrices based on ASLG-KG respectively, and supervises the matching process of semantic vectors and visual features, thereby ensuring that the semantic correlation structure in the visual space is consistent with the semantic space; finally, by effectively weighted fusion of the scene-level and landscape-level mapping branches, the scene feature expression capability of the model is significantly enhanced.

[0148] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope required by the present invention.

Claims

1. A dual-branch zero-shot remote sensing scene classification method based on knowledge graph, characterized by: The following steps are involved: S1. Obtain a remote sensing urban scene sample data set, and select some categories from it as training sets, and the rest as test sets; the training set samples include category information, and the test set samples do not include category information; S2. Construct a zero-shot recognition model AKG-DBSS. The zero-shot recognition model AKG-DBSS uses the ridge regression network as the basic network and the visual space as the embedding space. S3. In the zero-shot recognition model, typical objects in the scene are extracted based on the pre-trained YOLO-V5 model. The spatial distribution of objects is analyzed to construct a set of landscape descriptions in the local area, and the three-level knowledge graph of scene, landscape and object is automatically constructed. S4. In the zero-shot recognition model, global and local visual features are extracted based on VGGnet-16. At the same time, the TransH model is used to learn the representation of the knowledge graph to obtain the semantic vector corresponding to each scene. S5. In the zero-shot recognition model, the scene-level semantic correlation matrix Simg and the landscape-level semantic correlation matrix Siml are extracted respectively; S6. In the zero-shot recognition model, a dual-branch structure is used to map the semantic vector to the global and local visual feature spaces constructed by VGGnet-16, and the mapping process is supervised based on Simg and Siml. The zero-shot recognition model is trained using the training set by weighted fusion of the local and global branches. S7, selecting multiple remote sensing scene images and inputting them into the trained zero-shot recognition model, obtaining scene classification results, comparing them with the actual categories, and performing reliability testing on the trained zero-shot recognition model; S8. Quantitative evaluation of zero-shot recognition model; S9. Input the unknown urban scene image into the zero-sample recognition model to obtain the category to which the unknown urban scene belongs.

2. According to claim 1, a dual-branch zero-shot remote sensing scene classification method based on knowledge graph is characterized by: In step S3, extracting typical objects in the scene based on the pre-trained YOLO-V5 model includes the following steps: S3.1.

1. Use the public dataset DOTA to pre-train the YOLO-V5 model, so that the YOLO-V5 model has the ability to recognize 15 typical objects in the city. The 15 typical objects are airplanes, ships, storage tanks, baseball fields, tennis courts, basketball courts, ground runways, ports, bridges, large vehicles, small vehicles, helicopters, roundabouts, football fields, and basketball courts. S3.1.

2. Obtain the typical object detection results corresponding to each image, and record {k, [(x max ,y max ), (x min ,y min ), (X, Y)]}, which correspond to the object category output by the yolo-v5 model and the coordinate values ​​of the upper left corner, lower right corner, and center point of the object marking box.

3. According to claim 1, a dual-branch zero-shot remote sensing scene classification method based on knowledge graph is characterized by: In step S3, constructing a landscape description set within a local area includes the following steps: S3.2.

1. Select 9 spatial relationships R between typical objects r , which are on the surface, contained, surrounded, aligned, near, adjacent, overlapped, crossed, and far away, respectively; on this basis, for the objects k and k′ belonging to different categories, by analyzing the characteristics of each spatial relationship and ensuring that different spatial relationships are mutually exclusive, the corresponding discrimination rules for each spatial relationship are specified; S3.2.

2. Describing the index R by the degree of association CI Further quantitative evaluation of the strength of spatial relationships: Where H and W represent the height and width of the image respectively; X k , Y k With X k′ , Y k′ Then they represent the abscissa and ordinate values ​​of the center points k and k′ respectively; S3.2.

3. In a certain scene, if there are multiple spatial relationships between two types of objects, only R CI Get the maximum value R CI_m A spatial relationship R corresponding to r_m ; If there are multiple spatial relationships at the same time, R CI_m , then randomly retain one of the spatial relationships; finally, we get a class of scenes corresponding to <k,R r_m , R CI_m , k′>composed of a set of local landscape descriptions.

4. The dual-branch zero-shot remote sensing scene classification method based on knowledge graph according to claim 1, characterized in that: In step S3, the public remote sensing scene dataset NWPU is used for testing. By extracting typical objects and building a set of local landscape descriptions, a three-level knowledge graph of scene-landscape-object is obtained, which is specifically represented as follows: a remote sensing scene includes multiple local landscapes; each local landscape includes multiple typical objects, spatial relationships between objects, and the degree of association; different remote sensing scenes are associated with each other through having the same objects.

5. The dual-branch zero-shot remote sensing scene classification method based on knowledge graph according to claim 1, characterized in that: In step S4, visual features are extracted from both global and local aspects, the output of the last fully connected layer of VGGnet-16 is used as the global feature of the image, and a feature vector of size 1024×1 is output through the Linear layer, and finally a global visual feature set VG=[VG1, VG2, ..., VG N ]; the output of the last convolutional layer of VGGnet-16 is used as the local feature of the image, and the feature vector of size 1024×1 is output through the Linear layer, and finally the local visual feature set VL = [VL1, VL2, ..., VL N ].

6. The dual-branch zero-shot remote sensing scene classification method based on knowledge graph according to claim 5 is characterized by: In step S4, the TransH model is used to transform the automated scene-landscape-feature three-level knowledge graph ASLG-KG into a semantic vector set S = {s1, s2, ..., s T }, where s t (t=1, 2, ..., T) represents the semantic vector obtained for the t-th scene, and T represents the total number of scene categories, including visible and invisible categories.

7. The dual-branch zero-shot remote sensing scene classification method based on knowledge graph according to claim 1, characterized in that: The step S5 specifically includes the following sub-steps: S5.

1. Calculate the scene-level semantic relevance Simg by the following formula: Among them, s t and t′ They represent the semantic vectors of two different scenes respectively, and T represents the total number of scene categories, including visible and invisible categories; S5.

2. Construct a landscape-level coding matrix corresponding to a scene category: Where num represents the number of elements in the local landscape description set; for the i-th element, Q i , Q i ′ and QR i are k, k′ and R respectively r_m The corresponding one-hot encoded vector, Represents the concatenation of vectors; by replacing R CI_m With R r_m The corresponding coding vectors are multiplied so that the landscape-level coding matrix is ​​used to reflect the local spatial relationships contained in the scene and the corresponding degree of association; S5.

3. Calculate the landscape-level semantic correlation Siml between scenes by the following formula: Siml=1 T ·L t ·L t′ T ·1,(t≠t′) Among them, L t and L t′ They represent the landscape encoding matrices of two different scenes, with dimensions p and q respectively. The 1 on the left side of the formula represents a full-1 column vector with dimension p, and the 1 on the right side of the formula represents a full-1 column vector with dimension q.

8. The dual-branch zero-shot remote sensing scene classification method based on knowledge graph according to claim 1, characterized in that: The step S6 specifically includes the following sub-steps: S6.1, multiplying the semantic vector set S corresponding to the remote sensing scene by Simg and Siml respectively, and then mapping them to the global feature space and the local feature space respectively through the mapping function; S6.2, the local branch and the global branch match the mapping results with the global and local visual features respectively; S6.

3. The local and global branches are fused by weighted summation.

9. The dual-branch zero-shot remote sensing scene classification method based on knowledge graph according to claim 8, characterized in that: In step S6, the ridge regression function is used as the basis function to jointly train the local and global mapping branches under correlation supervision, and the following loss function is used to jointly train the mapping function: Among them, W1 and W2 correspond to the mapping functions of the global and local branches respectively, Simg and Siml supervise the mapping process of the two branches, B is the training data batch, and α is the weighted fusion weight of the global and local branches.

10. The dual-branch zero-shot remote sensing scene classification method based on knowledge graph according to claim 1, characterized in that: In step S8, the zero-shot recognition model is quantitatively evaluated through overall accuracy, standard deviation, class average accuracy, and confusion matrix indicators.

Images