A double-branch zero-shot remote sensing scene classification method based on a knowledge graph

Abstract

The application discloses a kind of double-branch zero sample remote sensing scene classification method based on knowledge graph, it is related to scene identification technical field, by analyzing the spatial distribution information of local area ground object composition, automatically constructs a kind of " scene-landscape-ground object " three-level knowledge graph;On this basis, the double-branch supervision zero sample remote sensing scene classification network DBSS designed from global and local two branches respectively to the mapping of semantic vector to visual space is supervised, to promote visual space can fully reflect the correlation structure contained in semantic space;Through a large number of experiments on UCM, AID and NWPU data set, it shows that the class average accuracy and overall accuracy of AKG-DBSS for the classification of invisible class scene can reach 98% and 59.56% respectively, the standard deviation is less than 6.91%, significantly better than other prior art methods, with good scalability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of scene recognition, and particularly relates to a double-branch zero-shot remote sensing scene classification method based on a knowledge graph. BACKGROUND

[0002] Semantic analysis and understanding of high-resolution remote sensing images as a whole to realize scene recognition is one of the landmark tasks in the field of computer vision. It has attracted extensive attention due to its great application potential in tasks such as city mapping to explain land distribution and utilization from the semantic level. However, due to the fact that a scene is an abstract concept with rich semantic information, a scene usually has a many-to-many correspondence with different geographical objects and local landscapes, which makes it difficult for traditional pixel-level or object-level target recognition methods to be applicable. Therefore, how to realize higher-level, more abstract and more discriminative scene interpretation is a challenging problem faced by the remote sensing scene classification task.

[0003] At present, deep learning models have shown excellent performance in scene classification due to their strong abstract representation capabilities. However, most existing deep learning-based methods belong to the supervised or semi-supervised learning category. These methods usually require sufficient labeled samples for training, otherwise they will have problems such as overfitting, which will significantly reduce the classification accuracy. In practical applications, due to the rich diversity of remote sensing scenes, and the accelerating process of global urbanization and the continuous emergence of new scene types, such methods often face the problem of lack of training samples or insufficient number of samples, which greatly reduces their usability.

[0004] To address the above challenges, some researchers have proposed few-shot learning (FSL) and zero-shot learning (ZSL) methods for remote sensing scene classification (RSSC). Among them, FSL, as an important branch of machine learning, aims to construct a machine learning model that can efficiently learn and accurately predict using only a small number of labeled samples. Compared with FSL, ZSL can eliminate the dependence on labeled samples of unseen classes, so it has stronger feasibility and applicability in the RSSC task.

[0005] As a special transfer learning classification method, ZSL infers the prototype representation of the unseen classes in the image feature space by transferring the knowledge of the seen class-labeled samples, so that the model has the ability to identify unseen class samples. The existing ZSL methods in the RSSC task generally use natural language models such as Word2vec, Bert, PLM, etc. Since these language models are trained based on a broad corpus, it is difficult to fully represent the relevance and difference between remote sensing scenes.

[0006] Therefore, some scholars have proposed a remote sensing scene knowledge graph (RSKG) to refine the scene. RSKG describes the concepts of the scene in symbolic form and reveals the relationship between them, and has shown strong knowledge modeling and reasoning application ability in the zero-shot scene classification task.

[0007] However, the existing knowledge graph still has the following four limitations in practical application: 1. The construction of the knowledge graph currently mainly depends on the manual interpretation of experts. This method is not only susceptible to differences in prior knowledge, making it difficult to guarantee its representativeness and reliability, but also not convenient for updating and expanding the knowledge graph; 2. The existing knowledge graph only qualitatively describes the association relationship using natural language, but lacks quantitative evaluation of the specific association degree, and fails to fully exploit the rich semantic information implied in the association relationship; 3. The existing knowledge graph only focuses on the association relationship between "scene-geographical object", but ignores the representative landscape composed of typical geographical objects within the scene. In fact, the geographical object composition and spatial distribution information within the local area, i.e. the landscape-level features, have been proven to have strong discriminability in scene classification applications; 4. The knowledge graph is only used for semantic vector generation, but ignores the supervision of the visual space construction process. In ZSL, the visual space has the same scene semantic correlation structure as the semantic space, and has been proven to achieve better classification results. SUMMARY

[0008] To solve the above technical problems, the present application provides a double-branch zero-shot remote sensing scene classification method based on a knowledge graph, comprising the following steps:

[0009] S1, obtain a remote sensing urban scene sample dataset, and select part of the classes as a training set and the rest as a test set; the training set samples include class information, and the test set samples do not include class information;

[0010] S2, construct a zero-shot recognition model AKG-DBSS, the zero-shot recognition model AKG-DBSS uses a ridge regression network as a base network and uses a visual space as an embedding space;

[0011] S3, in the zero sample recognition model, based on the pre-trained yolo-v5 model extracts the typical ground objects in the scene; analyze the spatial distribution of ground objects, and then construct the landscape description set in the local area, realize the automatic construction of scene-landscape-ground object three-level knowledge graph;

[0012] S4, in the zero sample recognition model, based on VGGnet-16 extracts global and local visual features; at the same time, the TransH model is used for representation learning of the knowledge graph, so as to obtain the semantic vector corresponding to each scene;

[0013] S5, in the zero sample recognition model, the scene level semantic correlation matrix Simg and the landscape level semantic correlation matrix Siml are extracted respectively;

[0014] S6, in the zero sample recognition model, the semantic vector is mapped to the global and local visual feature space constructed by VGGnet-16 by using a double-branch structure, and the mapping process is supervised based on Simg and Siml, the local and global two branches are weighted and fused, and the training set is used to train the zero sample recognition model;

[0015] S7, select multiple remote sensing scene images and input them into the trained zero sample recognition model, obtain the scene classification result, compare it with the actual category, and test the reliability of the trained zero sample recognition model;

[0016] S8, quantitative evaluation of the zero sample recognition model;

[0017] S9, input the unknown urban scene image into the zero sample recognition model, and obtain the category of the unknown urban scene image.

[0018] The further defined technical scheme of the application is:

[0019] Further, in step S3, the typical ground objects in the scene are extracted based on the pre-trained yolo-v5 model, including the following steps:

[0020] S3.1.1, the yolo-v5 model is pre-trained by using the public data set DOTA, so that the yolo-v5 model has the ability to identify 15 typical ground objects in the city, and the 15 typical ground objects are airplane, ship, storage tank, baseball field, tennis court, basketball court, ground runway, port, bridge, large vehicle, small vehicle, helicopter, ring intersection, football field and basketball court;

[0021] S3.1.2, obtain the detection result of the typical ground object corresponding to each image, record the {k, [(x max , y max ), (x min , y min), (X, Y)]}, respectively correspond to the land feature categories and the coordinates of the top left, bottom right, and center points of the land feature bounding boxes output by the YOLO-V5 model.

[0022] The aforementioned knowledge graph-based dual-branch zero-shot remote sensing scene classification method, in step S3, constructing a landscape description set within a local area includes the following steps:

[0023] S3.2.1 Select 9 spatial relationships R between typical land features r The spatial relationships are defined as being on the surface, containing, surrounding, aligned, nearby, adjacent, overlapping, intersecting, and far away. Based on this, for land features k and k′ belonging to different categories, the discrimination rules corresponding to each spatial relationship are specified by analyzing the characteristics of each spatial relationship and ensuring that different spatial relationships are mutually exclusive.

[0024] S3.2.2, Describing the index R through the degree of correlation CI Further quantitative evaluation of the strength of spatial relationships:

[0025]

[0026] Where H and W represent the height and width of the image, respectively; X k Y k With X k′ Y k′ Then, these represent the x-coordinate and y-coordinate values ​​of the center points of k and k′, respectively;

[0027] S3.2.3 In a certain scenario, if there are multiple spatial relationships between two types of land features, then only R is retained. CI Obtain the maximum value R CI_m A spatial relationship R corresponding to time r_m If multiple spatial relationships are obtained simultaneously, R CI_m Then, one of the spatial relationships is randomly retained; finally, a class of scenes corresponding to the spatial relationships is obtained. <k,R r_m R CI_m The set of local landscape descriptions constituted by k′>.

[0028] The aforementioned knowledge graph-based two-branch zero-shot remote sensing scene classification method, in step S3, uses the publicly available remote sensing scene dataset NWPU for testing. By extracting typical land features and constructing a set of local landscape descriptions, a three-level knowledge graph of scene-landscape-land feature is obtained. Specifically, a remote sensing scene includes multiple local landscapes; each local landscape includes multiple typical land features, spatial relationships between land features, and degree of association; different remote sensing scenes are associated through having the same land features.

[0029] The global visual feature set VG = [VG1, VG2,..., VG N ] is obtained finally, wherein the output of the last fully connected layer of the VGGnet-16 is taken as the global feature of the image, and a feature vector with a size of 1024x1 is output through a Linear layer. N The local visual feature set VL = [VL1, VL2,..., VL

[0030] In step S4 of the above-described knowledge graph-based double-branch zero-sample remote sensing scene classification method, the TransH model is used to convert the automatic scene-landscape-feature three-level knowledge graph ASLG-KG into a semantic vector set S = {s1, s2,..., s T T}, wherein s t (t = 1, 2,..., T) represents a semantic vector obtained by the t-th scene, and T represents the total number of scene categories, including visible categories and invisible categories.

[0031] In step S5 of the above-described knowledge graph-based double-branch zero-sample remote sensing scene classification method, the following steps are included:

[0032] S5.1, the scene-level semantic correlation Simg is calculated by the following formula:

[0033]

[0034] wherein s t and s t′ represent semantic vectors of two different scenes, and T represents the total number of scene categories, including visible categories and invisible categories.

[0035] S5.2, a landscape-level encoding matrix corresponding to a scene category is constructed:

[0036]

[0037] wherein num represents the number of elements included in the local landscape description set; for the i-th element, Q i , Q i and QR i are the one-hot encoding vectors corresponding to k, k' and R r_m , respectively, represents the splicing of vectors; and R CI_m and R r_mThe corresponding code vector is multiplied, so that the landscape-level code matrix is used to reflect the local spatial relationship contained in the scene and the corresponding correlation degree;

[0038] S5.3, calculate the landscape-level semantic correlation Siml between scenes by the following formula:

[0039] Siml = 1 T ·L t ·L t′ T ·1, (t!= t')

[0040] Wherein, L t and L t′ respectively represent the landscape code matrix of two different scenes, the dimension of which is p and q respectively, 1 on the left side of the formula represents a full 1 column vector with a dimension of p, and 1 on the right side of the formula represents a full 1 column vector with a dimension of q.

[0041] The aforementioned double-branch zero-sample remote sensing scene classification method based on a knowledge graph, step S6 specifically includes the following steps:

[0042] S6.1, multiply the semantic vector set S corresponding to the remote sensing scene with Simg and Siml respectively, and then map to the global feature space and the local feature space respectively through the mapping function;

[0043] S6.2, the mapping results of the local branch and the global branch are matched with the global and local visual features respectively;

[0044] S6.3, the local and global two branches are fused by weighted summation.

[0045] The aforementioned double-branch zero-sample remote sensing scene classification method based on a knowledge graph, in step S6, a ridge regression function is used as a basic function to jointly train the local and global mapping branches under the correlation supervision, and the following loss function is used to jointly train the mapping function:

[0046]

[0047] Wherein, W1 and W2 respectively correspond to the mapping functions of the global and local branches, Sg and Siml supervise the mapping process of the double branches, B is a batch of training data, and alpha is the weighted fusion weight of the global and local branches.

[0048] The aforementioned double-branch zero-sample remote sensing scene classification method based on a knowledge graph, in step S8, the zero-sample recognition model is quantitatively evaluated by overall accuracy, standard deviation, class average accuracy and confusion matrix indicators.

[0049] The beneficial effects of the present application are:

[0050] (1) In the present application, based on the extraction of typical features, a landscape model is constructed by analyzing the feature composition and spatial distribution information in the local area from the sample itself; then, in combination with the spatial position information of each feature in the scene, a quantitative description model of the correlation degree is proposed, so that the automatic construction of the "scene-landscape-feature" knowledge graph is realized; at the same time, the construction and updating of ASLG-KG do not need expert manual interpretation, so it has good scalability;

[0051] (2) In the present application, in combination with ASLG-KG, the embodiment designs a semantic-visual mapping oriented double-branch supervised module DBSS; the module constructs scene-level and landscape-level correlation matrices based on ASLG-KG, supervises the matching process of semantic vectors and visual features, so as to make the semantic correlation structure in the visual space consistent with the semantic space; finally, through effective weighted fusion of the scene-level and landscape-level mapping branches, the scene feature expression ability of the model is significantly enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 It is the overall flowchart of the present application;

[0053] Figure 2 It is the AKG-DBSS model network architecture diagram in the embodiment of the present application;

[0054] Figure 3 It is the ASLG-KG visualization diagram in the embodiment of the present application;

[0055] Figure 4 It is the double-branch mapping matching diagram under the correlation supervision in the embodiment of the present application;

[0056] Figure 5 It is the sample image diagram of three data sets in the embodiment of the present application, wherein (a) is the sample image diagram of UCM, (b) is the sample image diagram of AID, and (c) is the sample image diagram of NWPU;

[0057] Figure 6 It is the single-class scene classification confusion matrix diagram in the embodiment of the present application, wherein (a) is the single-class scene classification result diagram of UCM, (b) is the single-class scene classification result diagram of AID, and (c) is the single-class scene classification result diagram of NWPU;

[0058] Figure 7 It is the 0A diagram of common dimension vectors corresponding to different graphs in the embodiment of the present application;

[0059] Figure 8 It is the 0A diagram of different alpha values in the embodiment of the present application. DETAILED DESCRIPTION

[0060] The embodiment provides a dual-branch zero-shot remote sensing scene classification method based on a knowledge graph, as shown in the following formula (I): Figure 1 The method comprises the following steps:

[0061] S1, acquiring a remote sensing city scene sample data set, and selecting part of the classes as a training set and the rest as a test set from the data set; the training set samples comprise class information, and the test set samples do not comprise class information.

[0062] S2, constructing a zero-shot recognition model AKG-DBSS, namely an automatic knowledge graph and dual-branch semantic correlation supervision (AKG-DBSS), as shown in the following formula (II): Figure 2 The zero-shot recognition model AKG-DBSS adopts a ridge regression network as a basic network and adopts a visual space as an embedding space; the ridge regression network is helpful to reduce overfitting, and has shown good classification performance in a zero-shot learning (ZSL) task.

[0063] In the process of constructing an automatic "scene-landscape-ground object" three-level knowledge graph (ASLG-KG), first, typical ground objects in a scene are extracted, and the spatial distribution of the ground objects is analyzed to describe a landscape in a local region, so that the automatic "scene-landscape-ground object" three-level knowledge graph is constructed; in a dual-branch semantic-visual mapping supervision module (DBSS), a scene semantic vector S is obtained through TransH representation learning, and scene-level semantic correlation Simg and landscape-level semantic correlation Siml are extracted; the semantic vector is mapped to a global and local visual feature space constructed by a VGGnet-16 through a dual-branch structure, and the mapping process is supervised based on Simg and Siml, so that the visual space can fully retain the correlation structure of the semantic space; then, visual feature and semantic vector matching of the dual-branch is respectively performed; through effective weighted fusion of the scene-level and landscape-level mapping branches, a scene classification result is obtained.

[0064] S3, in the zero sample recognition model, based on the pre-trained yolo-v5 model extracts the typical features in the scene; analyze the spatial distribution of features, and then construct the landscape description set in the local area, realize the automatic construction of scene-landscape-feature three-level knowledge graph.

[0065] As shown in Figure 2 , first, the original image is extracted for typical features, which lays the foundation for subsequent construction of local landscape description set, based on the pre-trained yolo-v5 model extracts the typical features in the scene, including the following steps:

[0066] S3.1.1, the public dataset DOTA is used to pretrain yolo-v5 model, so that yolo-v5 model has the ability to identify 15 typical features in the city, 15 typical features are airplane, ship, storage tank, baseball field, tennis court, basketball court, ground runway, port, bridge, large vehicle, small vehicle, helicopter, ring intersection, football field and basketball court.

[0067] S3.1.2, the typical feature detection result corresponding to each image is obtained, and the {k, [(x max , y max ), (x min , y min ), (X, Y)]} of each feature is recorded, which respectively corresponds to the feature category and the coordinate values of the left upper corner, right lower corner and center point of the feature marking box output by yolo-v5 model.

[0068] On the basis of the extracted typical features, the spatial distribution information of the features in the scene is used to describe the local landscape, which includes the following steps:

[0069] S3.2.1, spatial relationship judgment: on the basis of referring to the existing knowledge graph, this embodiment selects nine kinds of spatial relationships R r , including surface, contain, surround, align, nearby, adjacent, coincide, cross and away; on this basis, for the features k and k' belonging to different categories, by analyzing the characteristics of each spatial relationship and ensuring the mutual exclusivity between different spatial relationships, the corresponding discrimination rules of each spatial relationship are specified, as shown in table 1 below.

[0070] Table 1 spatial relationship and discrimination rule table

[0071]

[0072] In table 1 above, k and k' represent different categories of features; (x max , y max ) and (x min , y min) are the coordinate values of the left upper corner and the right lower corner of the ground object label box output by the yolo-v5 model respectively, and S represents the number of pixels contained by the ground object.

[0073] S3.2.2, correlation degree extraction: the embodiment designs a correlation degree description index R CI to further quantitatively evaluate the strength of the spatial relationship:

[0074]

[0075] wherein H and W represent the height and width of the image respectively; X k , Y k and X k′ , Y k′ represent the horizontal coordinate value and the vertical coordinate value of the center point of k and k' respectively.

[0076] S3.2.3, local landscape description set construction: in a certain type of scene, if there are multiple spatial relationships between two types of ground objects, only the spatial relationship R CI that obtains the maximum value (denoted as R CI_m ) is retained, that is, the correlation degree is the strongest; if multiple spatial relationships obtain R r_m at the same time, one of them is randomly retained; finally, a local landscape description set composed of <k, R CI_m , R r_m , k'> corresponding to a type of scene is obtained. CI_m

[0077] Taking the classic public remote sensing scene data set NWPU as an example for testing, by carrying out typical ground object extraction and constructing a local landscape description set, the "scene-landscape-ground object" three-level knowledge graph ASLG-KG visualization result obtained is as shown in Figure 3 , which can be specifically represented as: a scene has multiple local landscapes; a local landscape is composed of multiple typical ground objects, spatial relationships between ground objects and correlation degrees; different scenes are associated through the same ground objects; in Figure 3 , the inclusion relationship between the scene and the landscape, and the landscape and the ground object is represented by a straight line, wherein the dashed line circle is a local landscape diagram, and the correlation R r_m and the correlation degree R CI_m between the two ground objects on both sides of the arrow in the landscape are displayed.

[0078] ​Therefore, the ASLG-KG can realize the automatic construction of the "scene-landscape-feature" three-level knowledge graph from the sample itself without relying on the manual interpretation of experts. It should be particularly pointed out that the pre-training DOTA data set and the NWPU data set used for testing in the embodiment are only for illustrating the construction process and effect of the ASLG-KG. In fact, the ASLG-KG is a general and scalable Remote Sensing Scene Knowledge Graphs (RSKG) construction method, and users can select a suitable data set according to the actual application requirements to construct or extend the ASLG-KG.

[0079] The DBSS designed in the embodiment mainly includes three parts of visual feature and semantic vector extraction, semantic correlation matrix extraction, and double-branch mapping under correlation supervision.

[0080] S4, visual feature and semantic vector extraction: the visual features are extracted from the global and local aspects to correspond to the scene-level and landscape-level semantic information in the ASLG-KG, respectively. In the embodiment, the output of the last fully connected layer of the VGGnet-16 is taken as the global feature of the image, and a feature vector with a size of 1024x1 is output through the Linear layer, and finally a global visual feature set VG=[VG1, VG2,..., VG N ] is obtained. In addition, the output of the last convolutional layer of the VGGnet-16 is taken as the local feature of the image, and a feature vector with a size of 1024x1 is output through the Linear layer, and finally a local visual feature set VL=[VL1, VL2,..., VL N ] is obtained.

[0081] In the semantic vector extraction, the TransH model is adopted to convert the ASLG-KG into a semantic vector set S={s1, s2,..., s T} due to the advantages of the translation invariance of the representation learning model TransH. Wherein s t (t=t, 2,..., T) represents the semantic vector obtained by the t-th scene, T represents the total number of scene categories, including visible categories and invisible categories.

[0082] S5, semantic correlation matrix extraction: scene-level semantic correlation matrix Simg and landscape-level semantic correlation matrix Siml are extracted, specifically including the following steps:

[0083] S5.1, the scene-level semantic correlation Simg is calculated by the following formula:

[0084]

[0085] Among them, s t and s t′ These represent the semantic vectors of two different scenarios, where T represents the total number of scenarios, including visible and invisible classes.

[0086] S5.2 Construct a landscape-level encoding matrix corresponding to a scene category:

[0087]

[0088] Where num represents the number of elements contained in the local landscape description set; for the i-th element, Q i Q i ′ and QR i They are k, k′ and R respectively. r_m The corresponding one-hot encoded vector, This represents the concatenation of vectors; by using R... CI_m With R r_m The corresponding encoding vectors are multiplied to make the landscape-level encoding matrix reflect the local spatial relationships contained in the scene and the corresponding degree of association.

[0089] S5.3 Calculate the landscape-level semantic correlation Siml between scenes using the following formula:

[0090] Siml=1 T ·L t ·L t′ T ·1,(t≠t′)

[0091] Among them, L t and L t′ Let p and q represent the landscape encoding matrices for two different scenes, respectively. The 1 on the left side of the formula represents a column vector of all 1s with dimension p, and the 1 on the right side of the formula represents a column vector of all 1s with dimension q. Finally, the correlation matrices Simg and Siml are output to represent the landscape-level semantic correlation and scene-level semantic correlation between any two scenes.

[0092] S6. Two-branch mapping under correlation supervision: The extracted Simg and Siml are used for supervision. The semantic vectors are mapped to the local and global embedding spaces through two branches and matched with visual features.

[0093] like Figure 4 As shown, step S6 specifically includes the following sub-steps:

[0094] S6.1, respectively multiply the semantic vector set S corresponding to the remote sensing scene with Simg and Siml, and then respectively map to the global feature space and the local feature space through the mapping function, so as to fully retain the semantic correlation structure contained in the ASLG-KG.

[0095] S6.2, the mapping results of the local branch and the global branch are matched with the global and local visual features respectively.

[0096] S6.3, the local branch and the global branch are fused by weighted summation, and the weight parameter a is set to 1.

[0097] On this basis, this paper adopts the ridge regression function with improved model generalization performance as the basis function, and then jointly trains the local and global mapping branches under the correlation supervision, and uses the following loss function to jointly train the mapping function:

[0098]

[0099] Wherein, W1 and W2 respectively correspond to the mapping function of the global branch and the local branch, Simg and Siml supervise the mapping process of the double branch, B is the batch size of the training data, and a is the weighted fusion weight of the global branch and the local branch.

[0100] S7, select multiple remote sensing scene images to input into the trained zero-shot recognition model, obtain the scene classification result, compare with the actual category, and test the reliability of the trained zero-shot recognition model.

[0101] The experiment adopts three groups of currently popular public remote sensing scene data sets, which are UC Merced (UCM), aerial image data set (AID) and NWPU-RESISC45 (NWPU). The reliability of the trained zero-shot recognition model is tested on the UCM, AID and NWPU data sets.

[0102] Among them, UCM is taken from the US Geological Survey National Map Urban Area Image Series, which provides typical urban scene labeling samples including intersection, dense_residential and 21 categories; AID is published by Huazhong University of Science and Technology and Wuhan University, which is a large-scale aerial image data set constructed by collecting samples from Google map images; NWPU is a public data set published by Northwestern Polytechnical University, which has significant differences in translation, spatial resolution and other aspects between different scene samples; the use of these three data sets helps to analyze the performance of the method from different angles, and the detailed parameters of each data set are shown in the following table 2.

[0103] Table 2 Comparison of experimental data sets

[0104]

[0105] To fully verify the performance of AKG-DBSS method, four advanced zero-shot remote sensing scene recognition methods are selected for comparison, which are Variational Autoencoder with Generative Adversarial Network (VAE-GAN), Multilevel Cross-Modal Feature Alignment method (MCFA), Local-Global Feature Fusion and Weight Mapping Loss method (LGFFWM) and Adaptive Relation Aware Network (ARAN). These methods are designed from different perspectives for the remote sensing scene recognition task, and the comparative experiments are helpful for the comprehensive and objective evaluation of AKG-DBSS.

[0106] Among them, VAE-GAN introduces GAN to learn the reconstruction quality measurement method suitable for VAE, and then proposes a cross-modal feature matching loss CMFM-Loss to promote the deep alignment of visual features and semantic features of each scene; MCFA proposes a multi-level feature alignment method to fully exploit the contrast relationship between the cross-modal features contained in the scene; LGFFWM improves the model's ability to distinguish different scenes by fusing local and global features, and on this basis, proposes a weight mapping loss (WM Loss) function to force the model to preferentially learn the visible class scenes with strong relevance to the unseen class scenes; ARAN proposes an adaptive relation aware network to learn the inter-class and intra-class relationships of different categories in the data set. These methods are designed from different perspectives for the remote sensing scene recognition task, and the comparative experiments are helpful for the comprehensive and objective evaluation of AKG-DBSS.

[0107] To ensure the fairness of the experiment, the experimental environment is built under the Ubuntu 16.04 system, and the Pytorch-1.3.1 framework is used, and the hardware configuration is an Nvidia GeForce RTX 2080ti GPU with 11GB RAM; the target recognition is based on the Yolo-v5 network, and the DOTA dataset is used for pre-training; wherein the img-size is set to 1024*1024, and 16 is used as the batch-size to train for 100 epochs; at the same time, the VGGnet is used to obtain a 1024-dimensional visual feature vector, and the TransH is used to obtain a 300-dimensional semantic feature vector; during the training process, the learning rate of the Adam optimizer is set to 0.00001, and the mini-batch size is set to 16.

[0108] Finally, in order to objectively compare with the existing methods, according to the above experimental settings, four different visible class / non-visible class division ratios are used in the experiment for UCM, AID and NWPU respectively; at the same time, it is ensured that there is no cross overlap between the visible class and the non-visible class, and the specific division is shown in Table 3; in addition, this embodiment shows some sample images on the three datasets as shown in Figure 5

[0109] Table 3 Data set visible class / non-visible class division table

[0110]

[0111] S8, the zero sample recognition model is quantitatively evaluated by overall accuracy (OA), standard deviation (SD), class average accuracy (CA) and confusion matrix (CM) indicators; wherein OA is a direct measure of the classification accuracy of the model on the entire dataset, as shown in the following formula:

[0112]

[0113] wherein N t and N f respectively represent the number of correctly classified and misclassified samples; SD reflects the dispersion degree of OA; CM explains the predicted class and actual class of the sample through the row and column, so it can analyze the positive and negative classification of different classes in detail.

[0114] In addition, CA provides an indicator to measure the average accuracy of the model in classifying various scene categories, as shown in the following formula:

[0115]

[0116] wherein, represents the number of correctly classified samples of the ith class, ​denotes the total number of samples of the i-th class.

[0117] The accuracy evaluation results of AKG-DBSS and comparative methods based on UCM, AID and NWPU three groups of data sets are shown in Tables 4-6 below. The underlined bold in the table represents the best result. The highest OA of AKG-DBSS in the three groups of data can reach 59.56%, 53.48% and 47.84% respectively, which are significantly better than other comparative methods.

[0118] Table 4 OA (%) and SD (%) of AKG-DBSS and state-of-the-art methods on UCM

[0119]

[0120] Table 5 OA (%) and SD (%) of AKG-DBSS and state-of-the-art methods on AID

[0121]

[0122] Table 6 OA (%) and SD (%) of AKG-DBSS and state-of-the-art methods on NWPU

[0123]

[0124] VAE_GAN, MCFA and ARAN all ignore the local features that are very discriminative in the scene recognition task during the training process. In the experiments of the three data sets, their overall accuracy is reduced by more than 4.04% compared with AKG-DBSS; LGFFWM, which is the second best, although it considers the double branch structure, it ignores the potential correlation between different scenes; compared with them, AKG-DBSS not only considers the fusion of local and global visual features, but also introduces the semantic correlation between scenes to supervise the mapping process, and its overall performance is relatively better in the experiments.

[0125] In the UCM dataset, the overall accuracy of AKG-DBSS is only 2.32% lower than that of LGFFWM at a visible class / non-visible class ratio of 10 / 11, and higher accuracy is achieved at the other three ratios; in the AID dataset experiment, although the OA of AKG-DBSS is better than that of the comparative method only at a visible class / non-visible class ratio of 25 / 5 and 15 / 15, the SD under the four visible class / non-visible class ratios all show the best effect, i.e., better stability; in the NWPU dataset experiment, although the OA of AKG-DBSS is 2.43% and 1.8% lower than that of LGFFWM at a visible class / non-visible class ratio of 30 / 15 and 20 / 25, respectively, the SD of AKG-DBSS is reduced by more than 0.07% at the four ratios; in summary, in the three experiments, AKG-DBSS shows better performance in overall accuracy and stability than other comparative methods.

[0126] Comparing the three datasets, the UCM dataset only contains 21 scene categories, and the number of samples per category is only 100, which is significantly less than the AID dataset and the NWPU dataset. Under this condition, AKG-DBSS and LGFFWM both use a double-branch structure, which improves the OA of VAE-GAN, MCFA, and ARAN by more than 10.09%, which also shows that the joint of global and local features has discriminative power in scene recognition tasks, especially for small sample applications.

[0127] The number of samples per scene category in the AID dataset is between 220 and 420, so there is an imbalance problem in the training process. Based on this dataset, AKG-DBSS improves the OA of the remaining comparative methods by more than 1.62%, so the semantic correlation supervision strategy introduced by AKG-DBSS can make the model pay more attention to the learning of "strongly related" samples, effectively alleviating the impact of the sample imbalance problem. However, when the proportion of visible class samples is low (e.g., 10 / 20), the semantic correlation of AKG-DBSS is difficult to fully express, resulting in a 1.75% decrease in OA compared to LGFFWM.

[0128] The NWPU dataset has 45 scene categories, and each scene has 700 samples, which helps the model to be trained more fully. The detection effect of AKG-DBSS and the comparative methods is generally better than that of the first two datasets, but AKG-DBSS still performs better.

[0129] To further analyze the classification performance of AKG-DBSS on each invisible class scene, the confusion matrix is adopted and combined with the CA index for experiments, wherein the UCM, AID, and NWPU three datasets are respectively set according to the mainstream division method, and the proportions of visible class and invisible class are set as 16 / 5, 25 / 5, 35 / 10, and the single-class scene classification results are as shown in Figure 6 .

[0130] As shown in Figure 6 , for UCM, the classification accuracy for "golf course", "medium-sized residential area", and "tank" is relatively high, all reaching more than 60%; for AID, the classification accuracy for "desert" and "industrial area" is all more than 90%, while "pond" and "forest" are easily misclassified; and for NWPU, "airplane", "basketball court", and "sparse residential area" are more easily accurately classified.

[0131] To further verify the effectiveness of the proposed ASLG-KG, the currently very popular remote sensing scene knowledge graph semantic representation remote sensing knowledge graph (SR-RSKG) is selected for comparison; the knowledge graph (KG) is artificially constructed based on expert knowledge, and specifically includes 70 scenes, 117 entity vectors, and 26 relationship vectors; in this embodiment, the 21, 30, and 45 scene semantic vectors common to ASLG-KG and SR-RSKG are selected, and then experiments are performed on the UCM, AID, and NWPU datasets; on this basis, to further discuss the influence of different semantic vector dimensions on OA, this embodiment compares and analyzes from three vector dimensions of 100, 300, and 500, and the same visible / invisible class division ratio described in the foregoing is adopted, and the specific evaluation results are as shown in Figure 7 .

[0132] As shown in Figure 7 , compared with SR-RSKG, the OA of ASLG-KG in the three datasets is increased by more than 6.3%, 4.6%, and 3.9% respectively; therefore, compared with the knowledge graph artificially constructed based on expert knowledge, ASLG-KG not only has high automation, but also shows stronger scene representation capability; in addition, with the gradual increase of vector dimension from 100, 300 to 500, the OA generally presents a trend of first rising and then falling; when the vector dimension is 300, the best classification performance is reached; we believe that when the dimension is low, the important feature information of the semantic vector is not fully expressed, and it is difficult to distinguish the subtle differences between semantics; and when the vector dimension is too high, redundant and interference information is generated in the semantic vector, resulting in a decrease in accuracy.

[0133] To verify the effectiveness of the DBSS double-branch strategy, the global and local branches are trained separately under the supervision of semantic correlation, and the obtained OAs are compared with the OA obtained by joint training of the two branches. The results are shown in Table 7, where represents embedding the module, and the underlined bold indicates the best result.

[0134] Table 7 DBSS double-branch strategy effectiveness analysis table

[0135]

[0136] It can be seen that similar conclusions are obtained for the three data sets: compared with the mapping of a global or local branch, the double-branch strategy in DBSS improves OA by more than 14.68%; this fully demonstrates that it is feasible and necessary to further introduce a local related mapping branch based on the global supervised mapping branch in the task of zero-shot remote sensing scene classification (Zero-shot RSSC), and the two have complementary effects.

[0137] To verify the effectiveness of the semantic correlation supervision in DBSS, scene-level semantic correlation (Simg) and landscape-level semantic correlation (Siml) are introduced separately or simultaneously under the double-branch strategy for supervision, and the results without introducing supervision are compared. The effectiveness is evaluated by OA, and the specific results are shown in Table 8, where represents embedding the module, and the underlined bold indicates the best result.

[0138] Table 8 correlation supervision effectiveness analysis table

[0139]

[0140] It can be seen that compared with not introducing supervision, introducing scene-level and landscape-level semantic correlation supervision separately or simultaneously can improve the classification accuracy of the three data sets by more than 1%, which fully demonstrates that introducing correlation supervision can make the visual space fully reflect the semantic correlation structure contained in ASLG-KG, and thus help to significantly improve OA; at the same time, it verifies that scene-level semantic correlation supervision has a greater impact on OA, while landscape-level semantic correlation plays a supplementary role.

[0141] In the construction of DBSS, the parameter α reflects the weight in the weighted fusion process of local and global branches. To analyze the influence of different α on the classification of the double-branch network, this embodiment sets α to increase by 0.5 each time, and takes OA as the evaluation index. The specific results are shown in Figure 8 .

[0142] As Figure 8As shown, with the increase of the value of a, the 0A obtained by the three data sets on AKG-DBSS generally presents a trend of first rising and then gradually falling; we believe that this is because at the beginning stage, the local branch supplements the expression of the global branch, so that the 0A presents an upward trend; then, when the proportion of the local branch is too high, it is easy to inhibit the expression of the global branch, so that the 0A presents a downward trend; in addition, the optimal or suboptimal effect is obtained in the three groups of experiments when a = 1, so this embodiment suggests setting a to 1 in actual application.

[0143] S9, inputting the unknown urban market scene image into the zero sample recognition model to obtain the category to which the unknown urban market scene belongs.

[0144] The embodiment proposes a double-branch zero sample remote sensing scene classification method AKG-DBSS based on a knowledge graph. Unlike the traditional manual labeling method, ASLG-KG can analyze the ground object composition and spatial distribution information in the local area to realize automatic construction of a "scene-landscape-ground object" three-level knowledge graph. On this basis, DBSS supervises the mapping of semantic vectors to visual space from global and local branches, so as to enable the visual space to fully reflect the correlation structure contained in the semantic space. Finally, the scene classification result is obtained by weighted fusion.

[0145] A large number of experiments on UCM, AID and NWPU data sets show that the 0A of AKG-DBSS for classifying invisible class scenes can reach 59.56%, and the standard deviation is less than 6.91%, which is significantly better than four advanced comparison methods. In addition, compared with the expert knowledge-based SR-RSKG, the proposed ASLG-KG can improve the accuracy by more than 4.8%. Finally, the double-branch fusion strategy in DBSS and the introduction of semantic correlation supervision can improve the 0A by more than 14.68% and 8.98%, respectively.

[0146] The ASLG-KG proposed in the embodiment can start from the sample itself, construct a landscape model by analyzing the ground object composition and spatial distribution information in the local area on the basis of typical ground object extraction, and further combine with the spatial position information of each ground object in the scene to propose a relationship and degree description (RDD) model, thereby realizing automatic construction of a "scene-landscape-ground object" knowledge graph. At the same time, the construction and updating of ASLG-KG do not require expert manual interpretation, so it has good scalability.

[0147] Combined with the ASLG-KG, a dual-branch supervised semantic-visual mapping module (DBSS) is designed in this embodiment; the module respectively constructs scene-level and landscape-level correlation matrices based on the ASLG-KG, and supervises the matching process of the semantic vector and the visual feature, so as to promote the consistency between the semantic correlation structure in the visual space and the semantic space; finally, through effective weighted fusion of the scene-level and landscape-level mapping branches, the scene feature expression capability of the model is significantly enhanced.

[0148] In addition to the above embodiments, the present application can have other implementation manners. Any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the protection scope required by the present application.

Claims

1. A knowledge graph-based two-branch zero-shot remote sensing scene classification method, characterized in that: Includes the following steps: S1. Obtain a remote sensing urban scene sample dataset, and select a portion of the categories as the training set, while the remainder is the test set; the training set samples include category information, while the test set samples do not. S2. Construct the zero-shot recognition model AKG-DBSS. The zero-shot recognition model AKG-DBSS uses a ridge regression network as the base network and visual space as the embedding space. S3. In the zero-shot recognition model, typical land features in the scene are extracted based on the pre-trained YOLO-V5 model; the spatial distribution of land features is analyzed, and then a set of landscape descriptions in the local area is constructed to realize the automated construction of a three-level knowledge graph of scene-landscape-land feature. S4. In the zero-shot recognition model, global and local visual features are extracted based on VGGnet-16; at the same time, the TransH model is used to learn the representation of the knowledge graph to obtain the semantic vectors corresponding to each scene. S5. In the zero-shot recognition model, extract the scene-level semantic relevance matrix Simg and the landscape-level semantic relevance matrix Siml, respectively. S6. In the zero-shot recognition model, a dual-branch structure is adopted to map the semantic vectors to the global and local visual feature spaces constructed by VGGnet-16 respectively, and the mapping process is supervised based on Simg and Siml. The zero-shot recognition model is trained by weighted fusion of the local and global branches and using the training set. S7. Select multiple remote sensing scene images and input them into the trained zero-shot recognition model to obtain scene classification results. Compare these results with the actual categories to perform a reliability test on the trained zero-shot recognition model. S8. Quantitatively evaluate the zero-shot identification model; S9. Input the unknown urban scene image into the zero-shot recognition model to obtain the category to which the unknown urban scene belongs; Step S3, constructing the landscape description set within the local area includes the following steps: S3.2.1 Select 9 spatial relationships R between typical land features r The spatial relationships are defined as being on the surface, containing, surrounding, aligned, nearby, adjacent, overlapping, intersecting, and far away. Based on this, for land features k and k′ belonging to different categories, the discrimination rules corresponding to each spatial relationship are specified by analyzing the characteristics of each spatial relationship and ensuring that different spatial relationships are mutually exclusive. S3.2.2, Describing the index R through the degree of correlation CI Further quantitative evaluation of the strength of spatial relationships: Where H and W represent the height and width of the image, respectively; X k Y k With X k′ Y k′ Then, these represent the x-coordinate and y-coordinate values ​​of the center points of k and k′, respectively; S3.2.3 In a certain scenario, if there are multiple spatial relationships between two types of land features, then only R is retained. CI Obtain the maximum value R CI_m A spatial relationship R corresponding to time r_m If multiple spatial relationships are obtained simultaneously, R CI_m Then, one of the spatial relationships is randomly retained; finally, a class of scenes corresponding to this relationship is obtained. <k,R r_m R CI_m The set of local landscape descriptions constituted by k′>.

2. The knowledge graph-based dual-branch zero-shot remote sensing scene classification method according to claim 1, characterized in that: In step S3, extracting typical features in the scene based on the pre-trained YOLO-V5 model includes the following steps: S3.1.

1. The YOLO-V5 model is pre-trained using the public dataset DOTA, enabling the YOLO-V5 model to recognize 15 typical urban features. The 15 typical urban features are airplanes, ships, storage tanks, baseball fields, tennis courts, basketball courts, ground runways, ports, bridges, large vehicles, small vehicles, helicopters, roundabouts, football fields, and basketball courts. S3.1.2 Obtain the typical feature detection results corresponding to each image, and record the {k, [(x]} of each feature. max y max ), (x min y min ), (X, Y)]}, respectively correspond to the land feature categories and the coordinates of the top left, bottom right, and center points of the land feature bounding boxes output by the YOLO-V5 model.

3. The knowledge graph-based dual-branch zero-shot remote sensing scene classification method according to claim 1, characterized in that: In step S3, the publicly available remote sensing scene dataset NWPU is used for testing. By extracting typical land features and constructing a set of local landscape descriptions, a three-level knowledge graph of scene-landscape-land feature is obtained. Specifically, a remote sensing scene includes multiple local landscapes; each local landscape includes multiple typical land features, spatial relationships between land features, and degree of association; different remote sensing scenes are associated with each other by having the same land features.

4. The knowledge graph-based dual-branch zero-shot remote sensing scene classification method according to claim 1, characterized in that: In step S4, visual features are extracted from both global and local perspectives. The output of the last fully connected layer of VGGnet-16 is used as the global feature of the image, and a feature vector of size 1024×1 is output through a Linear layer, finally obtaining the global visual feature set VG = [VG1, VG2, ..., VG...]. N The output of the last convolutional layer of VGGnet-16 is used as the local feature of the image. A feature vector of size 1024×1 is output through a Linear layer, ultimately yielding a local visual feature set VL = [VL1, VL2, ..., VL...]. N ].

5. The knowledge graph-based dual-branch zero-shot remote sensing scene classification method according to claim 4, characterized in that: In step S4, the Automated Scene-Landscape-Ground Feature Three-Level Knowledge Graph (ASLG-KG) is transformed into a semantic vector set S = {s1, s2, ..., s} using the TransH model. T }, where s t (t = 1, 2, ..., T) represents the semantic vector obtained for the t-th scene, where T represents the total number of scene categories, including visible and invisible categories.

6. The knowledge graph-based dual-branch zero-shot remote sensing scene classification method according to claim 1, characterized in that: Step S5 specifically includes the following sub-steps: S5.1 Calculate the scene-level semantic relevance Simg using the following formula: Among them, s t and s t′ These represent semantic vectors for two different scenarios, where T represents the total number of scenarios, including visible and invisible classes. S5.2 Construct a landscape-level encoding matrix corresponding to a scene category: L＝[l1，l2，l3,...l N ]， Where num represents the number of elements contained in the local landscape description set; for the i-th element, Q i Q i ′ and QR i They are k, k′ and R respectively. r_m The corresponding one-hot encoded vector, This represents the concatenation of vectors; by using R... CI_m With R r_m The corresponding encoding vectors are multiplied to make the landscape-level encoding matrix reflect the local spatial relationships contained in the scene and the corresponding degree of association; S5.3 Calculate the landscape-level semantic correlation Siml between scenes using the following formula: Siml=1 T ·L t ·L t′ T ·1,(t≠t′) Among them, L t and L t′ Let p and q represent landscape encoding matrices for two different scenes, respectively. The 1 on the left side of the formula represents a column vector of all 1s with dimension p, and the 1 on the right side of the formula represents a column vector of all 1s with dimension q.

7. The knowledge graph-based dual-branch zero-shot remote sensing scene classification method according to claim 1, characterized in that: Step S6 specifically includes the following sub-steps: S6.1 Multiply the semantic vector set S corresponding to the remote sensing scene with Simg and Siml respectively, and then map them to the global feature space and local feature space respectively through the mapping function; S6.

2. The local branch and the global branch respectively match the mapping results with the global and local visual features; S6.

3. Merge the local and global branches using a weighted summation method.

8. The knowledge graph-based dual-branch zero-shot remote sensing scene classification method according to claim 7, characterized in that: In step S6, the ridge regression function is used as the base function to jointly train the local and global mapping branches under correlation supervision. The following loss function is used to jointly train the mapping function: Where W1 and W2 correspond to the mapping functions of the global and local branches, respectively, Simg and Siml supervise the mapping process of the two branches, B is the training data batch, and α is the weighted fusion weight of the global and local branches.

9. The knowledge graph-based dual-branch zero-shot remote sensing scene classification method according to claim 1, characterized in that: In step S8, the zero-shot identification model is quantitatively evaluated using overall accuracy, standard deviation, class average accuracy, and confusion matrix index.

Images