A knowledge graph-based photo land use classification method and system

Abstract

The application discloses a kind of based on knowledge graph's photo land use classification method and system, wherein the method includes: constructing land use knowledge graph basic framework, making ground surface landscape sample dataset;Analysis ground surface landscape sample dataset, obtain scene, visual feature semantics and its corresponding relationship, and obtain land use knowledge graph in combination with basic framework;Collect land ground surface landscape photo, input semantic segmentation model to obtain the visual feature semantics and content proportion of photo, and carry out semantic matching with land use knowledge graph, obtain pure semantic combination;Pure semantic combination is input into multilayer perception model, and the land use type corresponding to each photo is output, and land use classification is completed.The application is matched and identified by constructing land use knowledge graph with land ground surface landscape photo to obtain land use type, can accurately identify the semantics irrelevant to photo and remove, improve scene signal-to-noise ratio, enhance the reliability and accuracy of classification result.

Description

TECHNICAL FIELD

[0001] The application relates to the field of land classification, and in particular to a photo land use classification method and system based on a knowledge graph. BACKGROUND

[0002] In order to timely grasp the land use change situation, the annual change survey, the semi-annual change survey and the daily change survey are carried out regularly in various places. Therefore, an intelligent method is urgently needed to realize the automatic interpretation of a large amount of land use real scene photos, so as to meet the needs of rapid and current land use survey and monitoring.

[0003] In recent years, deep learning technology has achieved remarkable results in image classification, target detection and image segmentation tasks in the field of computer vision. At present, the research on intelligent interpretation and classification of field evidence photos (or ground surface landscape photos) is also based on this, and the key problem is how to accurately identify the land cover feature semantics in the photos and how to map the overall scene features composed of various land cover feature semantics in the photos to the land use types. The initial research directly inputs the whole photo into a deep learning model for land use type classification, which is simple to operate and easy to implement, but the required sample size and training parameter amount are huge, and the influence of background noise is ignored, resulting in low classification accuracy. In order to reduce the interference of background noise, some researches segment the main scene in the photo through traditional machine learning technology and image processing technology and then classify it, which improves the classification accuracy to a certain extent, but the selection rule of the main scene in this method is too simple, it is difficult to obtain the real important features in the photo, and the effect is unstable. In view of this, some scholars have proposed a photo land use semantic analysis method based on saliency attention, but when facing very complex or irregular scenes, the accurate extraction of the salient region is still a challenge. On the other hand, although the above researches have paid attention to the influence of background noise on the extraction of photo feature semantics and the final land use classification accuracy and have adopted different technical means to control it, but when they remove the noise (irrelevant feature semantics), it is not targeted, and for different scene photos and different photos of the same scene, the removed noise is uncertain, which greatly limits the improvement of the photo land use classification accuracy.

[0004] In existing technologies, there is a significant difference in classification systems between land cover types directly obtained from visual coverage features and land use types indirectly inferred from human use features, making it difficult to further improve the accuracy of land use classification. Two bottlenecks hinder intelligent determination of land use types in photographs: First, land use types are classified based on human utilization of geographical scenes, not directly determined by the visual features of the scene. This results in many instances of "different types of the same object" and "different types of the same object," and the land use type of a scene is closely related to the semantic combination, distribution, and proportion of visible features in the scene. Second, the scenes depicted in the photographs are numerous, semantically complex, and noisy. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a knowledge graph-based method and system for land use classification based on photographs, which eliminates irrelevant semantic features of land photographs and improves the accuracy and reliability of land use classification.

[0006] Therefore, the technical solution adopted by the present invention is as follows:

[0007] This invention provides a knowledge graph-based photo-based land use classification method, the method comprising:

[0008] Construct a basic framework for a land use knowledge graph and create a sample dataset of land surface landscapes;

[0009] By analyzing the surface landscape sample dataset, we obtained the semantics of various scenes and visual features and their corresponding relationships, and combined them with the basic framework of land use knowledge graph to obtain a land use knowledge graph.

[0010] Collect land surface landscape photos, input them into a semantic segmentation model to obtain the visual feature semantics and content ratio of the photos, and perform semantic matching with a land use knowledge graph to obtain a pure semantic combination of land surface landscape photos.

[0011] The pure semantic combination of land surface landscape photos is input into a multilayer perceptual model, and the land use type corresponding to each photo is output to complete the land use classification.

[0012] According to the above scheme, the basic framework of the land use knowledge graph specifically includes first-level categories, second-level categories, and the relationships between each level of categories. The land use knowledge graph specifically includes first-level categories, second-level categories, third-level categories, and the relationships between each level of categories. Among them, the first-level category is the land use type to which the land cover scene type belongs, the second-level category is the land cover scene type, and the third-level category is the visual feature semantic type.

[0013] According to the above scheme, the constructed land use knowledge graph can be updated through incremental training and feedback mechanism, wherein the incremental training specifically refers to adding new scenes, visual feature semantics or corresponding relationships to the constructed land use knowledge graph when they exist; the feedback mechanism specifically refers to feeding back and adjusting the knowledge graph structure and the corresponding relationship between the scene and the visual feature semantics when the semantic combination cannot be mapped to the scene type.

[0014] According to the above scheme, the semantic segmentation model is preliminarily trained through an initial semantic segmentation sample set; the initial semantic segmentation sample set is obtained by manually collecting original land surface landscape photo sample data and labeling the original land surface landscape photo sample data by type.

[0015] According to the above scheme, the surface landscape sample data set is specifically obtained by the following method:

[0016] The test data set is obtained, and the preliminarily trained semantic segmentation model is inputted to obtain segmented test data;

[0017] The test data meeting the segmentation effect requirement is added to the surface landscape sample data set, and the test data not meeting the segmentation effect requirement is added to the initial semantic segmentation sample set; the semantic segmentation model is retrained, and the test data set is segmented and screened, and the training effect of the retrained semantic segmentation model is evaluated until the number of data in the surface landscape sample data set reaches a certain threshold and the training effect of the semantic segmentation model meets the requirement, thereby obtaining the final surface landscape sample data set and the trained semantic segmentation model.

[0018] According to the above scheme, the semantic matching with the land use knowledge graph specifically includes:

[0019] The trained semantic segmentation model is used to segment the land surface landscape photos to obtain the visual feature semantic composition of each photo and the corresponding content proportion, and the land use knowledge graph is matched to remove irrelevant semantics, thereby obtaining the pure semantic combination of each land surface landscape photo.

[0020] According to the above scheme, inputting the pure semantic combination of the land surface landscape photo into the multi-layer perception model specifically includes:

[0021] The pure semantic combination of each land surface landscape photo is converted into an input feature vector and inputted into the multi-layer perception model.

[0022] The multi-layer perception model calculates the land use type probability distribution corresponding to each input feature vector, selects the land use type with the maximum probability as the output result, and completes the land use type classification of the land surface landscape photo.

[0023] The multi-layer perception model is trained by a loss function calculated according to a weighted sum of a cross-entropy loss and a knowledge graph constraint loss, and the knowledge graph constraint loss is calculated according to a prediction probability of each land use type by the multi-layer perception model and an association strength of each land use type and a visual feature semantic.

[0024] The application further provides a photo land use classification system based on a knowledge graph, which comprises:

[0025] A framework modeling module is configured to build a land use knowledge graph basic framework and make a land surface landscape sample data set.

[0026] A graph modeling module is configured to analyze the land surface landscape sample data set to obtain various scenes, visual feature semantics and corresponding relationships, and combine the land use knowledge graph basic framework to obtain a land use knowledge graph.

[0027] A matching module is configured to collect land surface landscape photos, input the photos into a semantic segmentation model to obtain visual feature semantics and content proportions of the photos, and perform semantic matching with the land use knowledge graph to obtain a pure semantic combination of the land surface landscape photos.

[0028] A classification module is configured to input the pure semantic combination of the land surface landscape photos into a multi-layer perception model to output land use types corresponding to each photo, thereby completing land use classification.

[0029] According to the above scheme, the classification module is specifically configured to convert the pure semantic combination of each land surface landscape photo into an input feature vector and input the input feature vector into the multi-layer perception model; the multi-layer perception model calculates a land use type probability distribution of the land surface landscape corresponding to each input feature vector, selects a land use type with the largest probability as an output result, and completes land use type classification of the land surface landscape photo.

[0030] The application further provides a computer storage medium having a computer program stored therein and executable by a processor, wherein the computer program performs the photo land use classification method based on a knowledge graph.

[0031] The application has the following beneficial effects: the application makes a land surface landscape sample data set and builds a land use knowledge graph, integrates scenes, visual feature semantics and corresponding relationships into the knowledge graph, and performs matching and recognition on the land surface landscape photos to be classified based on the land use knowledge graph, thereby obtaining corresponding land use types, accurately recognizing and removing semantics irrelevant to the scenes of the photos, improving the scene signal-to-noise ratio, and enhancing the reliability and accuracy of the land use type classification result.

[0032] Further, the present application realizes continuous optimization and improvement of the knowledge graph by incrementally training and feedback mechanism, and further optimizes the land use classification result.

[0033] Further, the present application trains the multi-layer perception model by a loss function calculated according to the weighted sum of the cross-entropy loss and the knowledge graph constraint loss, can integrate the prior knowledge stored in the knowledge graph into the training of the classification model in the form of the knowledge constraint loss term, makes the model learn the rules from the data while also complying with the prior knowledge, and enhances the reliability of the classification result of the multi-layer perception model. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a method flowchart of the photo land use classification method based on the knowledge graph of the embodiment of the present application;

[0035] Figure 2 is a ground surface landscape sample data set construction flowchart of the embodiment of the present application;

[0036] Figure 3 is a ground surface landscape sample data set example graph of the embodiment of the present application;

[0037] Fig. 4 (a) is a schematic diagram of the relationship details of a certain three-level node of the land use knowledge graph constructed by the embodiment of the present application;

[0038] Fig. 4 (b) is a schematic diagram of the relationship details of a certain two-level node of the land use knowledge graph constructed by the embodiment of the present application;

[0039] Figure 5 is a technology roadmap of the embodiment of the present application;

[0040] Figure 6 is a schematic diagram of the embodiment of the present application;

[0041] Figure 7 is an effect evaluation image of different scene types in the embodiment of the present application; wherein (a) represents a feature-rich scene type image 1; (b) represents a feature-rich scene type image 2; (c) represents a feature-rich scene type image 3; (d) represents a feature-rich scene type image 4; (e) represents a feature-rich scene type image 5; (f) represents a background composition simple scene type image 1; (g) represents a background composition simple scene type image 2; (h) represents a background composition simple scene type image 3;

[0042] Figure 8 is a semantic segmentation effect comparison graph of the embodiment of the present application; wherein (a) represents Figure 7 the semantic segmentation effect comparison of (a); (b) represents Figure 7Semantic segmentation effect comparison in (b); (c) represents Figure 7 Semantic segmentation effect comparison in (c); (d) represents Figure 7 Semantic segmentation effect comparison in (d); (e) represents Figure 7 Semantic segmentation effect comparison in (e); (f) represents Figure 7 Semantic segmentation effect comparison in (f); (g) represents Figure 7 Semantic segmentation effect comparison in (g); (h) represents Figure 7 Semantic segmentation effect comparison in (h). DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0044] The embodiment of the present application provides a photo land use classification method based on a knowledge graph, as shown in the figure, the method comprises the following steps: Figure 1

[0045] S1, a land use knowledge graph basic framework is constructed, and a ground surface landscape sample data set is made.

[0046] S2, the ground surface landscape sample data set is analyzed to obtain various scenes, visual feature semantics and corresponding relationships, and a land use knowledge graph is obtained in combination with the land use knowledge graph basic framework.

[0047] S3, land surface landscape photos are collected, input into a semantic segmentation model to obtain visual feature semantics and content proportion of the photos, and are subjected to semantic matching with the land use knowledge graph to obtain a pure semantic combination of the land surface landscape photos.

[0048] S4, the pure semantic combination of the land surface landscape photos is input into a multi-layer perception model, and a land use type corresponding to each photo is output, and land use classification is completed.

[0049] ​Specifically, the land use knowledge graph basic framework specifically includes a first class, a second class, and a relationship between the first class and the second class, wherein the first class is specifically a land use type to which a land cover scene type belongs, and the second class is specifically a land cover scene type. The land use knowledge graph includes a first class, a second class, and a third class, wherein the third class is a visual feature semantic type. That is, the land use knowledge graph basic framework is constructed in two layers, one layer is a first class of land use types to which a land cover scene type belongs, and the other layer is a second class of land cover scene types, which is used to add a third class and a relationship between the third class and the second class to the framework according to a land landscape sample data set, and form a land use knowledge graph. In this embodiment, the knowledge graph is constructed by referring to the classification standard of the third national land survey, which is divided into a first class, a second class, and a basic type in turn; wherein the first class is a final land use type, the second class is an intermediate transition classification scene type, and the basic type is a visual feature semantic type that can be obtained by a human eye through a photo; for example, a photo of a chicken farm, the basic type (visual feature semantic type) refers to chickens, chicken coops, chicken feed troughs, and bare land surfaces in the photo, the second class refers to the scene type of the photo, that is, a chicken farm, and the first class refers to the land use type to which the chicken farm belongs: facility agricultural land.

[0050] Specifically, the semantic segmentation model is preliminarily trained through an initial semantic segmentation sample set; the initial semantic segmentation sample set is obtained by manually collecting original land surface landscape photo sample data and labeling the original land surface landscape photo sample data by type.

[0051] Specifically, the land surface landscape sample data set is prepared based on the basic framework, specifically including:

[0052] A test data set is obtained and input into the preliminarily trained semantic segmentation model to obtain segmented test data;

[0053] The test data that meets the segmentation effect requirement is added to the land surface landscape sample data set, and the test data that does not meet the segmentation effect requirement is added to the initial semantic segmentation sample set to increase the data similar in feature semantics to the test data, the semantic segmentation model is retrained, and the test data set is used for segmentation and screening, and the training effect of the retrained semantic segmentation model is evaluated, until the number of data in the land surface landscape sample data set reaches a certain threshold and the training effect of the semantic segmentation model meets the requirement, to obtain a final land surface landscape sample data set and a trained semantic segmentation model.

[0054] The flowchart for constructing the land surface landscape sample data set is shown in Figure 2 In addition, as shown in Figure 3 , it is an example of the land surface landscape sample data set constructed by the embodiment of the present application.

[0055] Specifically, the ground surface landscape sample data set is analyzed, the types of each type of ground surface coverage scene involved in the data set and the corresponding visual feature semantic types are counted, the content proportion of each visual feature semantic type is counted, and the pixel frequency of each visual feature semantic type under each scene type is counted. Combined with professional knowledge, the necessary and optional visual feature semantic types of each scene and the relationship between each scene type and each visual feature semantic type are summarized; each scene, each visual feature semantic and the corresponding relationship are stored in the knowledge graph to obtain the land use knowledge graph.

[0056] Specifically, based on the above description, the land use knowledge graph constructed in the embodiment contains 85 entities and 229 relationships. Fig. 4(a) shows the details of a certain three-level node relationship of the land use knowledge graph (taking a chicken farm as an example). The facility farmland (green bottom circle) is a first-level class, representing the land use type to which the ground surface coverage scene type belongs. The chicken farm (light blue bottom circle) is a second-level class, representing the ground surface coverage scene type. The blue bottom circle is a third-level class, such as a chicken coop, representing a visual feature semantic type. Among them, the scene-visual feature semantic relationship type includes sufficient and selectable. Fig. 4(b) shows the details of a certain two-level node relationship of the land use knowledge graph. The pink bottom circle is a land use type first-level class, such as arable land. The green bottom circle is a land use type second-level class, such as paddy field, both of which represent the land use type to which the ground surface coverage scene type belongs. For example, taking arable land as an example, referring to the three adjustment work classification, the fact triple “land use type”-contains->“arable land” can be obtained. According to expert knowledge and general knowledge, and the arable land related scenes contained in the data set constructed in this paper, the fact triple “lotus field”-belongs to->“paddy field” and “oilseed rape field”-belongs to->“dry land” can be obtained. According to the statistical analysis results of the visual feature semantics of each scene, the triple such as “oilseed rape field (visual feature semantic type)”-sufficient->“oilseed rape field (scene type)” and “forest land (visual feature semantic type)”-selectable->“oilseed rape field (scene type)” can be obtained. Among them, sufficient and selectable measure the importance of a certain visual feature semantic to a certain scene type. In this example, the photo scene type is oilseed rape field, which means that there must be a field planting oilseed rape (sufficient) in the photo, and other visual features may or may not appear in the photo (optional). The example shows that in the construction of the land use knowledge graph, in addition to considering the actual situation of the research task, the field standard, expert knowledge and general knowledge are also actively combined, which makes the constructed knowledge graph have good applicability and interpretability in the task.

[0057] In addition, the constructed land use knowledge graph can be updated through incremental training and a feedback mechanism. The incremental training dynamically integrates semantic information in the newly acquired photo sample dataset into the knowledge graph by incremental training of a deep learning model. For newly discovered entities or relationships, they are added to the knowledge graph after expert verification. The feedback mechanism specifically feeds back when some semantic combinations cannot be accurately mapped to land use types, adjusts the structure or semantic mapping rules of the knowledge graph, and thus realizes continuous optimization and improvement of the knowledge graph, thereby providing reliable prior knowledge support for photo land use classification.

[0058] Specifically, the semantic matching with the land use knowledge graph specifically includes:

[0059] The trained semantic segmentation model is used to segment the land surface landscape photos to obtain the visual feature semantic composition of each photo and the content proportion of each semantic, and the land use knowledge graph is matched to remove irrelevant semantics to obtain the pure semantic combination of each land surface landscape photo.

[0060] The content proportion of each semantic refers to the area proportion of each visual feature semantic obtained by the semantic segmentation model. Irrelevant semantics are semantics that have no connection with the main feature semantics in the land use knowledge graph. For example, the semantic segmentation and statistics of a photo labeled as a corn field are {sky: 0.3, forest land: 0.2, corn field: 0.4, grassland: 0.1}. The land use knowledge graph is queried for the corn field scene, and it is found that the sky has no connection with the corn field, that is, it is neither sufficient nor optional for the corn field scene type. At this time, the sky is considered as irrelevant semantics, and the irrelevant semantics can be removed.

[0061] Specifically, inputting the pure semantic combination of the land surface landscape photo into the multi-layer perception model specifically includes:

[0062] The pure semantic combination of each land surface landscape photo is converted into an input feature vector and input into the multi-layer perception model.

[0063] The multi-layer perception model calculates the land use type probability distribution corresponding to each input feature vector, selects the land use type with the maximum probability as the output result, and completes the land use type classification of the land surface landscape photo.

[0064] The technical roadmap of the embodiment is as shown in Figure 5As shown in the figure, the multi-layer perception model is one of the base models of the deep learning model, and can effectively perform complex pattern recognition and classification tasks through its multi-layer structure and nonlinear activation function. In this embodiment, the multi-layer perceptron is selected as the method for distinguishing and determining the type, and the selection is based on the following: first, the multi-layer perceptron has characteristics suitable for multi-classification tasks, the output layer can flexibly configure multiple neurons, and each neuron corresponds to a specific type, and thus can directly output the probability value of the sample belonging to each classification, which greatly facilitates the multi-classification decision of land use; second, the model can effectively deal with complex classification boundary problems and solve the problem of similar between classes and different within classes often occurring in land use classification tasks. Among many classification tasks, the type boundary of data points often presents a very complex nonlinear form. The multi-layer perceptron can map the input features to a new feature space by virtue of its hidden layer and nonlinear activation function, and in this space, the originally complex classification boundary is relatively simplified, thereby significantly improving the accuracy of data classification; third, the multi-layer perceptron also has a certain generalization ability. Under the premise of reasonable setting of the network structure (such as the number of hidden layer neurons) and training parameters (such as learning rate, regularization coefficient, etc.) of the multi-layer perceptron, the model can deeply learn the general rules contained in the data, rather than just mechanically memorizing the details of the training data. Figure 6 As shown in the figure, it is an implementation scheme diagram of the model training process and the prediction process of the embodiment of the application.

[0065] Specifically, the type number of the multi-layer perception model used in this embodiment is C, the input feature dimension is D, and given an input sample , the model output logits vector , and then the type probability distribution is obtained by the softmax function: and

[0066]

[0067] Wherein: X represents an D-dimensional real vector, wherein is a multi-dimensional vector representation; represents the mapping function of the multi-layer perceptron model with X as the input sample; represents that the logits vector output by the model is a C-dimensional real vector; represents the probability value of the model predicting the jth type, which measures the possibility of the model predicting that the sample belongs to the jth type under the given input, and the value is between (0~1), and the sum of the probabilities of all types is 1; represents the original output (logits) of the jth type. is the corresponding one for all types (from type 1 to type C) is summed up, where is the original output of the kth type.

[0068] During the training process, the model maps the input feature vector to the output probability vector by means of the forward propagation mechanism, where each element represents the probability of the sample belonging to a specific land use type. Then, the loss function value is calculated according to the predicted output probability and the real type label, and the model's weight and bias parameters are dynamically adjusted through the backpropagation algorithm combined with the gradient descent optimization method, aiming to minimize the loss value. The basic multilayer perceptron generally only uses the cross-entropy loss function, and on this basis, the improved multilayer perceptron classification model total loss is the weighted sum of the cross-entropy loss and the knowledge graph constraint loss , and the strength of the constraint term is controlled by the hyperparameter λ:

[0069]

[0070] By setting a reasonable value of the hyperparameter λ, the model can simultaneously consider and balance data learning and knowledge guidance.

[0071] where the specific expressions of the cross-entropy loss and the knowledge graph constraint loss are as follows:

[0072] For multi-classification problems, the formula of the cross-entropy loss function can be expressed as:

[0073]

[0074] where the real label is a one-hot vector, represented as , where only the value of the th position is 1, indicating that the sample belongs to the th class. represents the value of the th class in the real label, represents the probability of the model predicting the th class, and Σ represents the summation over all types.

[0075] For a dataset of samples, the cross-entropy loss is usually averaged:

[0076]

[0077] where represents whether the th sample belongs to the th class, The model represents the first The sample belongs to the first The predicted probability of a class. The smaller the value of the cross-entropy loss function (the goal of the loss function is to maximize the predicted probability of the correct type), the smaller the difference between the model's prediction and the true label, meaning the better the model's performance.

[0078] Additionally, for each input sample Knowledge graph constraint loss term Defined as:

[0079]

[0080] in, This represents the association matrix of a knowledge graph, where C is the number of types and D is the feature dimension. Pointer matrix The Middle line, number Column elements represent types With features The strength of the association; This represents the feature vector of the input sample. From the feature vector X of the input sample, the first... 1D feature values; The model represents the first The predicted probabilities for each type (output by the softmax function).

[0081] In this embodiment, the design of the knowledge graph loss term follows the importance of sample patterns while ensuring that the model prediction conforms to prior knowledge. For example, when a certain important land feature appears, the prediction probability of the relevant land use scenario type cannot be too low.

[0082] In this embodiment of the invention, the classification dataset consists of a series of records, each record covering the semantic composition of the visual features of a photograph scene and the proportion of each visual feature semantic in the photograph, with each record corresponding to one sample. Each sample is transformed into an input feature vector and input into a multilayer perceptron model. This model uses a forward propagation mechanism to map the input feature vector into an output probability vector, where each element represents the probability that the sample belongs to a specific land use type. The model's weights and bias parameters are dynamically adjusted using a backpropagation algorithm combined with gradient descent optimization to minimize the difference between the predicted output probability and the true type label (this difference is quantified and evaluated using a loss function). After multiple iterations of training, the model gradually learns the complex nonlinear mapping relationship between the input feature vector and the land use type label, thereby constructing a classification model capable of accurately identifying the land use type of a photograph scene.

[0083] likeFigure 7 , Figure 8 The image shown illustrates the implementation results of the knowledge graph-based photo land use classification method in this embodiment. This embodiment uses several high-performing models (DeepLabV3+, HRNetv2, PSPNet, TransUNet, U-HRNet) from recent segmentation tasks for training and testing on the aforementioned landscape photo sample dataset. Subjective and objective accuracy evaluations are used to measure and compare the segmentation prediction performance of each model on the test set, thereby selecting the optimal model for subsequent research. Figure 7 The image shown is used as input to the model for comparison. Figure 8 The image shows a comparison of the model performance. It can be seen that U-HRNet has the best overall segmentation performance, followed by HRNet and DeepLab V3+, with TransUNet and PSPNet performing slightly worse. Additionally:

[0084] For large-area features in the photograph that have significant differences in characteristics, regular shapes, and simple boundaries, the segmentation results of the models selected in this study are quite ideal, such as... Figure 8 (f), (g), (h);

[0085] For irregular features in photos with small feature differences or in complex backgrounds, multiple models are prone to misclassification or missed detection. For example, DeepLabV3+ and TransUNet cannot distinguish between them. Figure 8 In the (g) field, the irrigation ditch below the field, while PSPNet and HRNetv2 can distinguish between the ditch and the field, misclassifies it as other land cover types. PSPNet and TransUNet cannot distinguish it. Figure 8 The chicken flock in (b) is completely predicted and segmented;

[0086] For small features and obscured features in the photo, such as Figure 8 In (a), the flock of sheep in the upper right corner is obscured by a fence. Figure 8 The yellow chicken feed trough in (b) of the text. Figure 8 In (f), the cultivated layer and low houses in the far right of the middle of the middle are basically unable to be identified and predicted by DeepLabV3+, PSPNet and HRNetv2. The recognition and prediction effect of TransUNet is also very limited. Only U-HRNet can accurately predict and segment them.

[0087] U-HRNet still demonstrates the most outstanding performance in terms of the fineness of feature boundary segmentation in the image, such as... Figure 8 As shown in the rightmost column of the visualization results, U-HRNet can not only accurately identify and predict each target feature, but its segmentation boundaries are also very detailed, while other models show large-area segmentation boundary deviations.

[0088] In addition, as shown in the following table, the accuracy evaluation results of each semantic segmentation model are compared as follows:

[0089] Table 1 Comparison of accuracy evaluation results of each semantic segmentation model

[0090]

[0091] When objectively evaluating the performance of each semantic segmentation model, we use Overall Accuracy (overall accuracy) and Overall Mean IoU (average intersection over union) as two core evaluation indicators in semantic segmentation tasks to measure the performance of the model from different angles. Overall Accuracy (overall accuracy) measures the proportion of correctly classified pixels to the total number of pixels. It can intuitively reflect the overall accuracy of the model, but it has limitations when the sample is imbalanced. Overall Mean IoU (average intersection over union) calculates the intersection (precision) and union (recall) ratio of the predicted region and the true region for each type, and then averages the Mean IoU of all types. This indicator considers the prediction of each type and better reflects the balance of the model's performance on each type. A higher Mean IoU means that the model not only has a high overall accuracy, but also performs well on each type, especially on difficult-to-classify types.

[0092] Based on the above description, analyzing the accuracy evaluation results of each semantic segmentation model in Table 1, the overall accuracy of all segmentation models selected in the experiment on the test set samples is more than 80%, which indicates that they have good segmentation ability for the experimental data set of this study. Among them, HRNetv2 and U-HRNet have better overall accuracy than other models due to the advantage of multi-scale parallel feature fusion. The Overall Mean IoU of each model on the test set is significantly different, and the Overall Mean IoU of each model on the test set is significantly different. Combined with the confusion matrix of each model, it can be seen that the performance of each model on each type is different. For example, the performance of HRNetv2 on the "building" type is better than other models, while the performance of U-HRNet on the "vegetation" type is better than other models. This shows that the performance of each model on each type is different, and the performance of each model on each type is different. Figure 8In the analysis, the Overall Mean IoU of TransUNet is only 0.6639, which may be related to its advantage in dealing with complex background and long-distance dependence, but not enough fine-grained feature extraction for some types in multi-type segmentation task. The Overall Mean IoU of PSPNet is 0.7202, which performs better in capturing global context information, but slightly insufficient in dealing with high-resolution details, affecting the overall intersection over union. DeepLabV3+ has an advantage in capturing multi-scale context information and restoring spatial details, but the segmentation accuracy on some specific types is not high enough, affecting the overall performance indicator, Overall Mean IoU is 0.7225. HRNet and U-HRNet can effectively capture rich spatial information and details by maintaining high-resolution feature maps through parallel multi-scale convolution, thus showing higher Overall Accuracy and Overall Mean IoU in multi-type segmentation tasks.

[0093] Therefore, in the embodiment of the present application, the U-HRNet (U-shaped High-Resolution Network) model is selected as the semantic segmentation model of the photo land use to perform semantic segmentation on the photo and extract the visible feature semantic types and corresponding contents in each photo. The U-HRNet combines the advantages of the U-Net encoder-decoder structure and the HRNet multi-scale parallel feature fusion, and its U-shaped structure is composed of multiple stages, each stage containing feature maps of different resolutions, which effectively captures the context information and multi-scale features of the image through the encoder-decoder structure. In addition, U-HRNet adds more stages in the main network, gradually upsamples the feature maps and fuses them with the features of the previous stage, so that the feature maps with strong semantic representation can be combined with low-level high-resolution features earlier. The semantic representation head integrates features of different resolutions through pooling operation and channel splicing, and converts multi-resolution feature maps into final prediction output. U-HRNet has innovative features such as dynamic calculation allocation, multi-stage feature fusion and compatibility expansion by redesigning the network architecture, and becomes an efficient solution for dense prediction tasks such as semantic segmentation and depth estimation. Experimental verification shows that the model has high segmentation accuracy and detailed segmentation boundary, which can effectively meet the task requirements of this paper and provide high-quality semantic feature support for subsequent land use classification.

[0094] In addition, in order to verify the effectiveness of the method proposed in the present application, the embodiment uses different methods to classify land use on a new set of test sample photo data sets, matches the classification results with the artificial classification results, and calculates the accuracy of each method, so as to compare the classification accuracy of other photo land use type discrimination methods with the method proposed in the present application.

[0095] Table 2 Accuracy comparison of different photo land use classification methods

[0096]

[0097] As shown in Table 2, the embodiment compares and analyzes the performance of four different methods in the land use type determination task. Method one directly classifies the whole photo using InceptionV4 network, and the accuracy is 75.75%. Although this method is simple to operate, the classification accuracy is limited in complex scenes due to background noise interference. Method two introduces Mean shift image processing technology to pre-segment the photo to suppress the influence of background noise, and extracts the three largest features as the main scene to input InceptionV4 for classification, and the accuracy is improved to 88.25%, which is increased by 12.5% compared with method one. However, this method relies too much on dominant features and ignores the contribution of weak features. In addition, the Mean shift method is based on low-level features such as color, texture and edge of pixels for manual design segmentation, which leads to poor stability of segmentation effect and limits the further improvement of classification accuracy. Method three uses U-HRNet to perform pixel-by-pixel semantic segmentation on the photo, extracts as many feature semantics as possible, and uses a multi-layer perceptron (MLP) to determine the land use type based on the feature semantic combination and its content. The accuracy of this method on the test set is 90.5%, which is improved compared with the previous two methods. However, although method three makes full use of all extracted features, it does not specifically remove the influence of background noise. In addition, the misjudgment pixels in the U-HRNet semantic segmentation result introduce additional noise, which further limits the improvement of classification accuracy.

[0098] In view of the limitations of the above methods, method four proposed by the application introduces a land use knowledge graph based on method three. Through the knowledge representation and reasoning ability of the knowledge graph, this method can effectively filter irrelevant feature semantic information in the photo and reduce the interference of background noise. Experimental results show that the classification accuracy of method four on the test set is 96.25%, which is significantly better than the previous three methods. This shows that the semantic filtering combined with the knowledge graph can effectively improve the accuracy of land use type determination, and provides an efficient solution for land use classification in complex scenes.

[0099] In addition, the embodiment of the application also provides a photo land use classification system based on a knowledge graph, which is used for the photo land use classification method based on a knowledge graph described in the embodiment of the application. The system comprises:

[0100] A framework modeling module is configured to build a basic framework of a land use knowledge graph and make a ground surface landscape sample data set.

[0101] a graph construction module configured to analyze a ground surface landscape sample data set to obtain various types of scenes, visual feature semantics, and corresponding relationships thereof, and to obtain a land use knowledge graph in combination with a basic framework of a land use knowledge graph;

[0102] a matching module configured to collect ground surface landscape photos, input the photos into a semantic segmentation model to obtain visual feature semantics and content proportions of the photos, and perform semantic matching with the land use knowledge graph to obtain pure semantic combinations of the ground surface landscape photos;

[0103] a classification module configured to input the pure semantic combinations of the ground surface landscape photos into a multi-layer perception model to output land use types corresponding to the photos, and complete land use classification.

[0104] The classification module is specifically configured to convert the pure semantic combinations of the ground surface landscape photos into input feature vectors, and input the input feature vectors into the multi-layer perception model; the multi-layer perception model calculates land use type probability distributions corresponding to the input feature vectors, selects a land use type with the largest probability as an output result, and completes land use type classification of the ground surface landscape photos.

[0105] The various modules or mechanisms of the system are mainly used to implement the various steps of the above method embodiments, and will not be described here.

[0106] In addition, the embodiment of the present application also provides a computer storage medium, which stores a computer program executable by a processor, and the computer program executes the land use classification method of the photos based on the knowledge graph according to any one of the above.

[0107] The land use classification method of the photos based on the knowledge graph and the system provided by the embodiment of the present application can effectively improve the accuracy and interpretability of the classification result by making a ground surface landscape sample data set and constructing a land use knowledge graph, constructing a land use knowledge graph storing scenes, visual feature semantics, and corresponding relationships thereof, removing irrelevant semantic noise in each scene based on the knowledge graph, and integrating prior knowledge stored in the knowledge graph into the training of the classification model in the form of a knowledge constraint loss term, so that the model learns the rules from the data while also following the prior knowledge.

[0108] Further, the embodiment of the present application dynamically updates the constructed land use knowledge graph through incremental training and feedback mechanisms, realizes continuous optimization and improvement of the knowledge graph, and further optimizes the land use classification result.

[0109] It should be noted that, according to the needs of implementation, each step / component described in the present application can be split into more steps / components, or two or more steps / components or part operations of the steps / components can be combined into a new step / component, to achieve the purpose of the present application.

[0110] The size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0111] It should be understood that, for those skilled in the art, improvements or changes can be made according to the above description, and all these improvements and changes shall fall within the protection scope of the appended claims of the present application.

Claims

1. A knowledge graph based photo land use classification method, characterized in that, The method comprises: constructing a land use knowledge graph basic framework and making a ground surface landscape sample data set; analyzing the ground surface landscape sample data set to obtain various types of scenes, visual feature semantics and corresponding relationships, and combining the land use knowledge graph basic framework to obtain a land use knowledge graph; collecting land surface landscape photos, inputting the photos into a semantic segmentation model to obtain the visual feature semantics and content proportion of the photos, and performing semantic matching with the land use knowledge graph to obtain a pure semantic combination of the land surface landscape photos; inputting the pure semantic combination of the land surface landscape photos into a multilayer perception model to output the corresponding land use type of each photo, thereby completing land use classification; wherein the multilayer perception model is trained by a loss function calculated according to a weighted sum of cross-entropy loss and knowledge graph constraint loss; Wherein, the loss function of the multi-layer perception model can be expressed as: λ represents a hyperparameter that controls the strength of the constraint term; Among them, cross-entropy loss ;in Indicates the first Does the _ sample belong to the _ ... kind, The model represents the first The sample belongs to the first The predicted probability of a class, where N represents the number of samples; Knowledge graph constraint loss ; wherein , denotes a knowledge graph correlation matrix, C is the number of types, D is the feature dimension, points to the element in the matrix , the first row, the first column, represents the association strength of the type and the feature ; denotes the feature vector of the input sample, is the value of the first dimension feature in the feature vector X of the input sample; denotes the prediction probability of the model for the first type.

2. The knowledge graph-based photo land use classification method of claim 1, wherein, The land use knowledge graph basic framework specifically includes first-level categories, second-level categories and relationships between categories at each level, and the land use knowledge graph specifically includes first-level categories, second-level categories, third-level categories and relationships between categories at each level; wherein the first-level category is specifically a land use type to which a ground surface coverage scene type belongs, the second-level category is specifically a ground surface coverage scene type, and the third-level category is a visual feature semantic type.

3. The knowledge graph based photo land use classification method of claim 1, wherein, The constructed land use knowledge graph can be updated through incremental training and a feedback mechanism, wherein the incremental training specifically involves adding new scenes, visual feature semantics or corresponding relationships to the constructed land use knowledge graph when they exist; and the feedback mechanism specifically involves feeding back and adjusting the knowledge graph structure and the corresponding relationship between scenes and visual feature semantics when a semantic combination cannot be mapped to a scene type.

4. The knowledge graph-based photo land use classification method of claim 1, wherein, The semantic segmentation model is preliminarily trained through an initial semantic segmentation sample set; and the initial semantic segmentation sample set is obtained by manually collecting original land surface landscape photo sample data and labeling the original land surface landscape photo sample data by type.

5. The knowledge graph-based photo land use classification method of claim 4, wherein, The ground surface landscape sample data set is specifically obtained by: obtaining a test data set and inputting the preliminarily trained semantic segmentation model to obtain segmented test data; selecting test data meeting the segmentation effect requirement to add to the ground surface landscape sample data set, adding data similar in feature semantics to test data not meeting the segmentation effect requirement to the initial semantic segmentation sample set, retraining the semantic segmentation model, and using the test data set to segment and select test data and evaluate the training effect of the retrained semantic segmentation model until the number of data in the ground surface landscape sample data set reaches a certain threshold and the training effect of the semantic segmentation model meets the requirement, thereby obtaining a final ground surface landscape sample data set and a trained semantic segmentation model.

6. The knowledge graph-based photo land use classification method of claim 5, wherein, The semantic matching with the land use knowledge graph specifically comprises: segmenting the land surface landscape photos using the trained semantic segmentation model to obtain the visual feature semantic composition of each photo and the corresponding content proportion of each semantic, and matching with the land use knowledge graph to remove irrelevant semantics, thereby obtaining a pure semantic combination of each land surface landscape photo.

7. The knowledge graph-based photo land use classification method of claim 1, wherein, Inputting the pure semantic combination of the land surface landscape photos into the multilayer perception model specifically comprises: converting the pure semantic combination of each land surface landscape photo into an input feature vector and inputting it into the multilayer perception model; The multi-layer perception model calculates a land use type probability distribution of a land surface landscape corresponding to each input feature vector, selects a land use type with the largest probability as an output result, and completes land use type classification of the land surface landscape photo.

8. A knowledge graph based photo land use classification system, characterized in that, The system comprises: a framework modeling module configured to build a land use knowledge graph basic framework and make a land surface landscape sample data set; a graph modeling module configured to analyze the land surface landscape sample data set, obtain various scenes, visual feature semantics, and corresponding relationships, and obtain a land use knowledge graph in combination with the land use knowledge graph basic framework; a matching module configured to collect land surface landscape photos, input the photos into a semantic segmentation model to obtain visual feature semantics and content proportion of the photos, and perform semantic matching with the land use knowledge graph to obtain a pure semantic combination of the land surface landscape photos; a classification module configured to input the pure semantic combination of the land surface landscape photos into a multi-layer perception model to output a land use type corresponding to each photo, and complete land use classification; wherein the multi-layer perception model is trained by using a loss function calculated according to a weighted sum of a cross-entropy loss and a knowledge graph constraint loss; Wherein, the loss function of the multi-layer perception model can be expressed as: λ represents a hyperparameter that controls the strength of the constraint term; Among them, cross-entropy loss ;in Indicates the first Does the _ sample belong to the _ ... kind, The model represents the first The sample belongs to the first The predicted probability of a class, where N represents the number of samples; Knowledge graph constraint loss ; wherein , denotes a knowledge graph correlation matrix, C is the number of types, D is the feature dimension, points to the element in the matrix , the th row, the th column, represents the association strength of the type and the feature ; denotes the feature vector of the input sample, is the value of the th feature in the feature vector X of the input sample; denotes the prediction probability of the model for the th type.

9. The knowledge graph-based photo land use classification system of claim 8, wherein, the classification module is specifically configured to convert the pure semantic combination of each land surface landscape photo into an input feature vector and input the input feature vector into the multi-layer perception model; the multi-layer perception model calculates a land use type probability distribution of a land surface landscape corresponding to each input feature vector, selects a land use type with the largest probability as an output result, and completes land use type classification of the land surface landscape photo.

10. A computer storage medium, characterized in that, The computer program stored in the memory can be executed by the processor, and the computer program performs the land use classification method of the photo based on the knowledge graph according to any one of claims 1-7.

Images