Knowledge graph construction method and system for multi-modal cultivated land data

Abstract

The invention relates to a knowledge graph construction method and system for multi-modal cultivated land data. The method comprises the following steps: constructing a cultivated land protection field knowledge system; entities, relations and attributes are extracted and stored in a knowledge graph, corresponding description information is obtained and input into an incomplete multi-modal data fusion algorithm model based on deep semantic matching for data feature fusion, cultivated land data in different modals are mined, the knowledge graph is supplemented and adjusted, and the knowledge graph of the multi-modal cultivated land data is constructed. Entities, relationships and attributes are extracted and stored in the knowledge graph, so that the constructed knowledge graph can be ensured to be more accurate, the coverage field is wider, and the application of the knowledge graph is more diversified and accurate; data feature fusion is carried out in an incomplete multi-modal data fusion algorithm model, so that the heterogeneous problem in the knowledge graph is effectively solved; the cultivated land data in different modes are fully mined, and the knowledge graph can become more powerful in tasks such as self-complementation and intelligent question and answer.

Description

Technical Field

[0001] The present invention relates to the technical field of constructing a knowledge graph of cultivated land data, and in particular to a method and system for constructing a knowledge graph of multimodal cultivated land data. Background Art

[0002] While the economy is developing rapidly, the phenomenon of non-cultivated land such as agricultural land, forest land, and grassland being transferred to cultivated land has also occurred from time to time, exacerbating the reduction of cultivated land area and the decline of cultivated land quality. The situation of cultivated land protection is very serious, and advanced methods are urgently needed to strengthen cultivated land protection, ensure the utilization of cultivated land resources, and coordinate cultivated land protection and economic development. To this end, multiple departments have built a series of cultivated land protection platforms and measures, such as permanent basic farmland division, cultivated land resource quality classification, soil type map division, soil fertility protection and improvement, soil environmental quality investigation and monitoring, and "three lines and one list" ecological environment zoning control system. However, these platforms and statistical reports are developed for a certain business. They are scattered, disordered, isolated, and have a low degree of informatization, weak expression ability, and lack of reasoning ability. They cannot meet the requirements for cultivated land protection under the new situation. As a highly interconnected semantic network, the knowledge graph can convert massive unstructured farmland remote sensing images, policy texts and semi-structured statistical tables into standardized, unified, reliable and effective structured knowledge, breaking the problem of isolated farmland protection data, providing support for farmland management and decision-making, and improving the quality of farmland and the level of farmland protection.

[0003] In the context of big data and the Internet of Things, data sources are becoming increasingly rich and data structures are becoming increasingly complex. The knowledge understanding and organizational expression of multi-source heterogeneous data has become a particularly critical issue in the process of building knowledge graphs. At present, many knowledge graphs developed by research institutions at home and abroad are based on large-scale multi-source heterogeneous data. These knowledge graphs not only have multimodal and heterogeneous data types, but also have multi-source data sources. The following table shows some knowledge graphs built based on multi-source heterogeneous data:

[0004]

[0005]

[0006] According to the above knowledge graph, the purpose of building a knowledge graph based on multi-source heterogeneous data is to build a multimodal knowledge graph and integrate heterogeneous data with the knowledge graph. Most knowledge graphs are built based on static data, and there is little research on farmland protection and land.

[0007] Therefore, there is a lack of a unified knowledge management mechanism in the field of farmland protection, and data is stored in a scattered and disordered state, making it difficult to use it efficiently. Summary of the invention

[0008] Based on this, in order to solve the above technical problems, a knowledge graph construction method and system for multimodal cultivated land data are provided. Planning and management are carried out through the knowledge graph, knowledge is effectively combined with the cultivated land protection system, and knowledge is used to drive business, making the management of cultivated land protection data more personalized and intelligent.

[0009] A method for constructing a knowledge graph of multimodal cultivated land data, the method comprising:

[0010] Determine the need for farmland protection, collect farmland protection data based on the need for farmland protection, and construct a knowledge system in the field of farmland protection based on the farmland protection data;

[0011] Acquire multimodal cultivated land data from the cultivated land protection field knowledge system, extract entities, relationships, and attributes from the multimodal cultivated land data based on the multimodal cultivated land data and geographic information, and store the entities, relationships, and attributes in a knowledge graph to obtain a preliminary knowledge graph;

[0012] According to the preliminary knowledge graph, each description information corresponding to the entity, relationship, and attribute is obtained, and each description information is input into an incomplete multimodal data fusion algorithm model based on deep semantic matching to perform data feature fusion, so as to obtain a knowledge graph after feature fusion;

[0013] Based on the knowledge graph after the feature fusion, the cultivated land data in different modalities are mined, and the knowledge graph is supplemented and adjusted according to each of the mined cultivated land data to complete the construction of the knowledge graph of multimodal cultivated land data.

[0014] In one embodiment, the determining of farmland protection needs, collecting farmland protection data based on the farmland protection needs, and constructing a farmland protection domain knowledge system based on the farmland protection data include:

[0015] Obtaining a needs analysis report, and determining farmland protection needs based on the needs analysis report;

[0016] According to the farmland protection needs, farmland protection data is collected, and entities are extracted based on the farmland protection data to form a multi-dimensional and layered farmland protection field knowledge system.

[0017] In one embodiment, the method further comprises:

[0018] Determine the top-level concepts of the domain knowledge graph, and decompose and refine them layer by layer based on the top-level concepts to obtain a concept classification system with a hierarchical structure;

[0019] Adding entity objects extracted from the farmland protection data to the concept classification system;

[0020] The entity objects are summarized to obtain underlying concepts and form the knowledge system in the field of farmland protection.

[0021] In one embodiment, the multimodal cultivated land data is obtained from the cultivated land protection field knowledge system, entities, relationships, and attributes of the multimodal cultivated land data are extracted based on the multimodal cultivated land data and geographic information, and the entities, relationships, and attributes are stored in a knowledge graph to obtain a preliminary knowledge graph, including:

[0022] Dividing the multimodal cultivated land data into structured data, semi-structured data and unstructured data;

[0023] Determine the input information containing structured information in the structured data, identifiable marked data and extract entities, relationships and attributes;

[0024] Extracting entities, relationships, and attributes from the semi-structured and unstructured data using automated techniques;

[0025] The extracted entities, relationships, and attributes are stored in the knowledge graph to obtain a preliminary knowledge graph.

[0026] In one embodiment, the extracting entities, relationships, and attributes of the multimodal cultivated land data based on the multimodal cultivated land data and geographic information includes:

[0027] Obtaining pre-defined extraction rules, performing text analysis on the multimodal cultivated land data and geographic information, and obtaining analysis text description features;

[0028] According to the analysis text description features, text matching is performed on the multimodal cultivated land data according to the extraction rules to complete entity extraction.

[0029] In one embodiment, the extracting entities, relationships, and attributes of the multimodal cultivated land data based on the multimodal cultivated land data and geographic information includes:

[0030] Acquire label data, annotate the multimodal farmland data according to the label data, determine word boundaries in the text, and obtain classification features;

[0031] Based on the classification features, a BiLSTM model is used for training to extract sentence features, and forward and backward hidden state results in the multimodal cultivated land data are combined according to the sentence features;

[0032] The output of the BiLSTM model is used as the input of CRF to form a BiLSTM-CRF structure, and sequence labeling is performed based on the BiLSTM-CRF structure to complete entity extraction.

[0033] In one embodiment, the extracting entities, relationships, and attributes of the multimodal cultivated land data based on the multimodal cultivated land data and geographic information includes:

[0034] Extracting dynamic change information, inclusion relationships, and expression forms from the multimodal cultivated land data;

[0035] Calculate the occurrence frequency of each language pattern based on the dynamic change information, inclusion relationship, and expression form;

[0036] The language pattern corresponding to the highest frequency of occurrence is matched with the entity text to complete the relationship extraction.

[0037] In one embodiment, the method further comprises:

[0038] Utilize multimodal high-level semantic design to build an incomplete multimodal data fusion algorithm model based on deep semantic matching;

[0039] Determine the spatial geometric characteristics of the incomplete multimodal data fusion algorithm model, and design a modal local invariant graph regularization factor based on the spatial geometric characteristics to couple the multimodal deep sharing features and the original modal features;

[0040] The incomplete multimodal data fusion algorithm model is optimized and updated through an optimization algorithm to obtain high-level semantic shared features of the multimodal data.

[0041] In one embodiment, the method further comprises:

[0042] The graph database Neo4j is used to store the constructed knowledge graph of multimodal cultivated land data;

[0043] Each entity in the knowledge graph is stored through the nodes in the graph database Neo4j, and the edges between the nodes are established according to the association relationship between the entities.

[0044] A knowledge graph construction system for multimodal cultivated land data, the system comprising:

[0045] A farmland protection domain knowledge system construction module is used to determine farmland protection needs, collect farmland protection data based on the farmland protection needs, and construct a farmland protection domain knowledge system based on the farmland protection data;

[0046] A knowledge extraction module is used to obtain multimodal cultivated land data from the cultivated land protection field knowledge system, extract entities, relationships, and attributes of the multimodal cultivated land data based on the multimodal cultivated land data and geographic information, and store them in a knowledge graph to obtain a preliminary knowledge graph;

[0047] A knowledge fusion module is used to obtain various descriptive information corresponding to the entities, relationships, and attributes according to the preliminary knowledge graph, and input each of the descriptive information into an incomplete multimodal data fusion algorithm model based on deep semantic matching to perform data feature fusion, so as to obtain a knowledge graph after feature fusion;

[0048] The graph construction module is used to mine the cultivated land data in different modes based on the knowledge graph after the feature fusion, and to supplement and adjust the knowledge graph according to the mined cultivated land data, so as to complete the construction of the knowledge graph of multimodal cultivated land data.

[0049] The above-mentioned knowledge graph construction method and system for multimodal cultivated land data extracts entities, relationships, and attributes of the multimodal cultivated land data based on multimodal cultivated land data and geographic information and stores them in the knowledge graph, which can ensure that the constructed knowledge graph is more accurate, covers a wider range of fields, and has more diverse and precise applications; by fusing data features in an incomplete multimodal data fusion algorithm model, the heterogeneous problem in the knowledge graph can be effectively solved; by fully mining the cultivated land data in different modes, the knowledge graph can become more powerful in tasks such as self-completion and intelligent question and answer; planning and management are carried out through the knowledge graph, knowledge is effectively combined with the cultivated land protection system, and knowledge is used to drive the business, making the management of cultivated land protection data more personalized and intelligent. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 An application environment diagram of a method for constructing a knowledge graph of multimodal farmland data in one embodiment;

[0051] Figure 2 A schematic diagram of a flow chart of a method for constructing a knowledge graph of multimodal farmland data in one embodiment;

[0052] Figure 3 A schematic diagram of a technical route for constructing a knowledge system in the field of cultivated land protection in one embodiment;

[0053] Figure 4 A schematic diagram of a knowledge concept model in the field of farmland protection in an embodiment;

[0054] Figure 5 A diagram of a rule-based farmland protection terminology vocabulary type annotation block diagram in one embodiment;

[0055] Figure 6 A flow chart of the BiLSTM and CRF algorithm models in one embodiment;

[0056] Figure 7 A schematic diagram of a farmland protection entity extraction process in an embodiment;

[0057] Figure 8Schematic diagram of an incomplete multimodal data fusion architecture based on deep semantic matching in one embodiment;

[0058] Fig. 9 A schematic diagram of an unsupervised multimodal co-learning architecture of related and unrelated features in one embodiment;

[0059] Fig.10 A schematic diagram of a knowledge service system framework for farmland protection based on a knowledge graph in one embodiment;

[0060] Fig.11 A structural block diagram of a knowledge graph construction system for multimodal farmland data in one embodiment;

[0061] Fig.12 It is a structural block diagram of a village and town land big data knowledge service system in one embodiment;

[0062] Fig.13 A schematic diagram of a farmland protection business and a comprehensive knowledge graph in an embodiment;

[0063] Fig.14 A schematic diagram of the inclusion relationship between relevant terms and standard specifications in the field of cultivated land protection in an embodiment;

[0064] Fig.15 A schematic diagram of knowledge extraction from a knowledge graph in one embodiment;

[0065] Fig.16 A schematic diagram showing a visualization of knowledge extraction results of a knowledge graph in one embodiment;

[0066] Fig.17 A schematic diagram of knowledge fusion of a knowledge graph in an embodiment;

[0067] Fig.18 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] The knowledge graph construction method of multimodal cultivated land data provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Figure 1As shown, the application environment includes a computer device 110. The computer device 110 can determine the demand for farmland protection, collect farmland protection data based on the demand for farmland protection, and build a knowledge system in the field of farmland protection based on the farmland protection data; the computer device 110 can obtain multimodal farmland data from the knowledge system in the field of farmland protection, extract entities, relationships, and attributes of the multimodal farmland data based on the multimodal farmland data and geographic information, and store them in the knowledge graph to obtain a preliminary knowledge graph; the computer device 110 can obtain various description information corresponding to the entities, relationships, and attributes based on the preliminary knowledge graph, input each description information into the incomplete multimodal data fusion algorithm model based on deep semantic matching to perform data feature fusion, and obtain a knowledge graph after feature fusion; the computer device 110 can mine farmland data in different modes based on the knowledge graph after feature fusion, and supplement and adjust the knowledge graph according to the mined farmland data, and complete the knowledge graph of multimodal farmland data. Among them, the computer device 110 can be, but not limited to, various personal computers, laptops, smart phones, robots, unmanned aerial vehicles, tablet computers and other devices.

[0070] In one embodiment, Figure 2 As shown, a method for constructing a knowledge graph of multimodal cultivated land data is provided, comprising the following steps:

[0071] Step 202, determine the demand for farmland protection, collect farmland protection data based on the demand for farmland protection, and construct a farmland protection field knowledge system based on the farmland protection data.

[0072] Among them, the farmland protection data, namely the relevant data on farmland protection, mainly comes from laws and regulations, standards and specifications, rules and documents, technical regulations, literature, farmland protection evaluation reports, farmland quality evaluation databases, field industry websites, Baidu Encyclopedia, and project results materials.

[0073] Since farmland protection is relatively complex, it covers a wide range of knowledge and disciplines. Figure 3 As shown, we first need to determine the demand for farmland protection, and then determine the demand analysis report based on the demand for farmland protection. On the basis of the demand analysis report, we classify and summarize the farmland protection knowledge needs of managers in different positions and in different business fields through on-site surveys, expert interviews, literature review and other methods. Then, based on the demand analysis results, combined with the industry's existing knowledge classification system, terminology in standards and specifications, etc., we form a multi-dimensional hierarchical concept system, and then combine it with professional experience and knowledge to expand the non-class relationship between entities, focusing on the structural decomposition of the farmland protection knowledge system from the dimensions of farmland protection business processes, application objects, etc., so as to identify the concept system of farmland protection as comprehensively as possible.

[0074] Specifically, in one embodiment, a method for constructing a knowledge graph of multimodal farmland data is provided, which may also include a process of constructing a knowledge system in the field of farmland protection. The specific process includes: obtaining a demand analysis report, and determining farmland protection needs based on the demand analysis report; collecting farmland protection data based on the farmland protection needs, and extracting entities based on the farmland protection data to form a multi-dimensional and layered knowledge system in the field of farmland protection.

[0075] In this embodiment, ontology modeling can be performed to study the relationship expression between multi-level and same-level objects in farmland protection, as well as the composition of elements such as object type and function; the differences in farmland protection ontology data at multiple levels such as structure, expression, and semantic expression are realized based on the unified description of ontology concepts to achieve the mapping of ontology models.

[0076] Among them, ontology modeling methods include TOVE method, METHONTOLOGY method, Uschold method, KACTUS engineering method, SENSUS method, DEF5 method, seven-step method and domain ontology construction method based on thesaurus, etc. Among them, the seven-step method is a method for constructing domain ontology, which includes seven steps: determining the scope of domain ontology, reusing existing ontologies, listing terms in the domain, defining classes and class hierarchical relationships, defining class attributes, defining attribute facets, and filling in instances.

[0077] In another embodiment, a method for constructing a knowledge graph of multimodal cultivated land data is provided, which may also include a process of constructing a knowledge system in the field of cultivated land protection. The specific process includes: determining the top-level concepts of the domain knowledge graph, and decomposing and refining the top-level concepts layer by layer to obtain a concept classification system with a hierarchical structure; adding entity objects extracted from cultivated land protection data to the concept classification system; summarizing the entity objects to obtain the underlying concepts to form a knowledge system in the field of cultivated land protection.

[0078] Among them, this embodiment can adopt the seven-step ontology modeling method, using a top-down method that combines the top-down and bottom-up methods. Figure 3 As shown in the figure, we can first determine the top-level concepts of the domain knowledge graph, and then form a hierarchical concept classification system (Taxonomy) by decomposing and refining it layer by layer. On this basis, we can add corresponding entity objects to enrich the knowledge content. The bottom-up method is to summarize the existing entities to form the bottom-level concepts, and then summarize them layer by layer to form the upper-level concepts. Ontology-based concept system modeling is a typical top-down modeling method, and it is also the most commonly used modeling method for knowledge graphs. Figure 3As shown in the figure, in combination with the knowledge needs in the field of farmland protection, we first conduct entity extraction based on high-quality and easily structured data such as professional literature, standards and specifications, policies and systems in the field of farmland protection, and initially form a multi-dimensional and layered framework system for the field of farmland protection.

[0079] In this embodiment, specifically, when constructing the ontology model, the model layer of the farmland protection knowledge graph is designed mainly according to the knowledge needs in the field of farmland protection to describe the concepts, scopes and constraints, including concept design, relationship design and concept attribute design, to form a multi-dimensional hierarchical concept system and construct an ontology model of the knowledge graph in the field of farmland protection.

[0080] Among them, Figure 4 As shown in the figure, the comprehensive knowledge system of cultivated land protection includes basic knowledge, business knowledge and subject knowledge. Basic knowledge includes natural objects, social objects and subjects involved in cultivated land protection; business knowledge is divided into land development and remediation, land use planning, cultivated land status survey, etc. according to business type; subject knowledge includes concepts, principles, laws, methods, technical means, etc. in the field of cultivated land protection and utilization, usually in the form of standards, literature books, technical regulations, etc. The classification and content attributes of each cultivated land protection field in the comprehensive knowledge system of cultivated land protection are shown in the following table:

[0081]

[0082] Step 204, obtain multimodal cultivated land data from the knowledge system of cultivated land protection field, extract entities, relationships and attributes of the multimodal cultivated land data based on the multimodal cultivated land data and geographic information, and store them in the knowledge graph to obtain a preliminary knowledge graph.

[0083] Among them, combining professional experience and knowledge, focusing on business needs related to farmland protection, and based on the knowledge system of farmland protection, before realizing information integration and semantic interaction of farmland protection knowledge graph, it is necessary to study knowledge extraction technology to extract information and calculate relationships for various structured, semi-structured and unstructured data.

[0084] In one embodiment, a method for constructing a knowledge graph of multimodal cultivated land data is provided, which may also include a process of knowledge extraction, and the specific process includes: dividing the multimodal cultivated land data into structured data, semi-structured data, and unstructured data; determining the input information containing structured information in the structured data, identifiable marked data, and extracting entities, relationships, and attributes; using automatic technology to extract entities, relationships, and attributes from semi-structured and unstructured data; storing the extracted entities, relationships, and attributes in the knowledge graph to obtain a preliminary knowledge graph.

[0085] Specifically, knowledge extraction operations can be divided into two categories: structured data extraction only processes input data with structured information, especially data containing identifiable tags; for semi-structured and unstructured data, automatic technology is used to extract knowledge, including entity extraction, relationship extraction, and attribute extraction. Considering that the data may be missing, abnormal, inconsistent in dimension, etc., it is necessary to clean and verify the data; the quality of cultivated land protection data is improved through operations such as filling missing values, removing redundancy, and comparative verification to meet the input data requirements for building a knowledge graph, which is mainly completed by using the data analysis package Panda in Python.

[0086] In one embodiment, a knowledge graph construction method for multimodal cultivated land data provided may also include a process of extracting cultivated land protection knowledge entities based on rules, and the specific process includes: obtaining pre-established extraction rules, performing text analysis on multimodal cultivated land data and geographic information, and obtaining analysis text description features; based on the analysis text description features, performing text matching on the multimodal cultivated land data according to the extraction rules to complete entity extraction.

[0087] Specifically, rule-based entity extraction is to first formulate extraction rules, then perform text matching, and extract information that meets the rules as entities. It is generally aimed at unstructured data with clear description characteristics, such as terms in standards and technical regulations related to farmland protection and utilization. After extracting the terms according to the rules, these terms are classified and labeled, and then the terms are expanded through inclusion relationships, genus relationships, equivalence relationships, etc.

[0088] Specifically, in this embodiment, the rule-based farmland protection terminology vocabulary type annotation block diagram is as follows: Figure 5 As shown in the figure, it mainly targets unstructured data with relatively clear description features, such as the terms in the standards and technical regulations related to the protection, development and remediation of cultivated land. By analyzing the text description features, manually constructing matching rules, and then automatically extracting them using the program. Entity knowledge elements such as organizational structures, technical methods, planning types, natural objects, and engineering objects provide semantic support for the acquisition of standard specification knowledge by establishing connections with the terminology in the standards and specifications, and are also the link between the relevant knowledge such as standards and specifications and technical regulations. First, terminology is extracted from the standards and specifications and laws and regulations in the field of cultivated land protection and management, and then these terminology is classified and annotated, and on this basis, professional field entity transformation is carried out. Among them, the extraction of terminology is based on the terminology interpretation in the standards and specifications, and the type is annotated according to the dictionary and rules, and then the terminology is expanded through the genus relationship, inclusion relationship, equivalence relationship, etc.

[0089] Among them, the standard specification generally provides a unified explanation of the terms that need to be explained at the beginning, or has a separate chapter to define the terms to explain the meaning of specific words used in the standard specification. For example, based on the extraction rules constructed according to the term features, the following terms can be extracted: "cultivated land quality", "positioning monitoring", "cultivated land management"; the explanation of cultivated land quality is "the ability to meet the continuous output and quality safety of agricultural products composed of cultivated land fertility, soil health status and field infrastructure".

[0090] In one embodiment, a knowledge graph construction method for multimodal cultivated land data provided may also include a process of cultivated land protection entity recognition based on deep learning, and the specific process includes: obtaining labeled data, annotating the multimodal cultivated land data according to the labeled data, determining the boundaries of words in the text, and obtaining classification features; training using a BiLSTM model based on the classification features, extracting sentence features, and combining forward and backward hidden state results in the multimodal cultivated land data according to the sentence features; using the output of the BiLSTM model as the input of the CRF to form a BiLSTM-CRF structure, and performing sequence labeling based on the BiLSTM-CRF structure to complete entity extraction.

[0091] Since rule-based methods require manual construction of rules, they have high accuracy when processing small-scale knowledge graphs and are difficult to expand on a large scale. For technical regulations, planning texts, reports, literature, etc. related to the protection and utilization of cultivated land, which are mostly unstructured data, a bidirectional long short-term memory neural network (BiLSTM) and a conditional random field (CRF) model are used to extract cultivated land protection and utilization entities. BiLSTM includes forward and backward sequence training, which learns the contextual information of the sequence. CRF learns the dependency information between labels and can add constraints to the labels predicted by the BiLSTM layer. The combination of the two can make full use of their respective advantages. The specific process is as follows: Figure 6 shown.

[0092] Due to the lack of annotated corpus resources in the field of cultivated land protection, the terminology extracted by rules is used to organize relevant entries and perform manual text annotation, and to construct an annotation set and a special dictionary for cultivated land protection and utilization. The recognition corpus of cultivated land protection entities is mainly unstructured object description information. First, an annotation set is formulated based on the corpus sample, and a label is defined for each element in the sample sequence according to the characteristics of cultivated land protection and utilization. Based on this, the preprocessed unstructured text is annotated to determine the boundaries of words in the text and determine the classification features for entity recognition; then the BiLSTM model is used for training, and sentence features are extracted by saving the context information of the whole sentence, and the hidden state results of the forward and backward directions in the text are combined; finally, the model output of BiLSTM is passed to CRF as input to form a BiLSTM-CRF structure, and sequence annotation is performed using context information. In the model training, a method based on word vector combination is used, and the automatically acquired dictionary is used to match sentences, and the input character sequence and all potential words are encoded.

[0093] Specifically, Figure 7 As shown in the figure, firstly, a label set is formulated based on the corpus sample, and a label is defined for each element in the sample sequence according to the feature. Based on this, the preprocessed unstructured text is annotated to determine the boundaries of words in the text and determine the classification features for entity recognition. Then, the BiLSTM model is used for training to extract sentence features by saving the context information of the entire sentence and combining the hidden state results of the forward and backward directions in the text. Finally, the output of the BiLSTM model is passed to the CRF as input, and the context information is used for sequence labeling.

[0094] In one embodiment, a knowledge graph construction method for multimodal cultivated land data provided may also include a process for extracting cultivated land protection relationships based on pattern matching, and the specific process includes: extracting dynamic change information, inclusion relationships, and expression forms in multimodal cultivated land data; calculating the frequency of occurrence of each language pattern based on the dynamic change information, inclusion relationships, and expression forms; matching the language pattern corresponding to the highest frequency of occurrence with the entity text to complete relationship extraction.

[0095] In rule-based term extraction, some clauses in the standard specifications contain vocabulary related to the terms, and there are proprietary names with inclusion relationships and equivalence relationships, such as "arable land quality information: soil site conditions, natural attributes, health conditions, farmland construction and management and other data related to arable land quality, original records, etc. and related dynamic change information, expression forms include: text, numbers, symbols, graphics, images, videos, etc.", the inclusion relationship between "arable land quality information" and "soil site conditions", "natural attributes", "health conditions", and "farmland construction and management" can be extracted; "text, numbers, symbols, graphics, images, videos" are the "expression forms" of arable land quality information; for unstructured text data, the pattern matching method is used to summarize the language patterns with high frequency according to the characteristics of the data in the field of arable land protection, and then match the entity text with the pattern.

[0096] In this embodiment, a method based on pattern matching to extract farmland protection relations is used to extract farmland protection related data, and unstructured data such as technical regulations, planning texts, reports, and literature materials for farmland protection are manually annotated. 80% of the data is used as a training set and 20% as a validation set, and 1518 entities and 1716 relationships are obtained. The entity and relationship data are stored in the Neo4j database. F1 is used to evaluate the entity recognition effect. The calculation method formula is: Among them, TP is a correct positive example, FP is a wrong positive example, and FN is a wrong negative example; the labeling accuracy is shown in the following table:

[0097]

[0098] In this embodiment, the content of the farmland protection knowledge graph is shown in the following table:

[0099]

[0100]

[0101] Step 206, obtain various descriptive information corresponding to entities, relationships, and attributes based on the preliminary knowledge graph, input various descriptive information into an incomplete multimodal data fusion algorithm model based on deep semantic matching to perform data feature fusion, and obtain a knowledge graph after feature fusion.

[0102] The variability and diversity of cultivated land protection data make knowledge representation difficult. Knowledge fusion is an effective way to solve this problem. The focus is to unify the knowledge obtained from various knowledge sources and find the potential true value to complete the construction of the knowledge graph.

[0103] Due to the heterogeneity of knowledge graphs, the same entity may exist in different forms of graphs. On the other hand, the knowledge of the same entity may be distributed in several graphs. In order to deal with the heterogeneity problem and integrate the knowledge in different knowledge graphs, it is necessary to perform alignment technology, that is, to match entities in different knowledge graphs. In practical applications, multimodal data usually has high-dimensional attribute characteristics. By fusing the relevant features of multimodal data, the multimodal high-dimensional features are mapped to a low-dimensional unified feature sharing space, which can promote the mutual complementation and sharing of knowledge between modalities, and then mine the hidden semantic information of multimodal data. Most of the existing multimodal data feature learning methods only focus on the learning of shared features between modalities. They convert the high-dimensional attributes of each modality into a low-dimensional shared subspace through transformation, projection and other techniques.

[0104] In one embodiment, a knowledge graph construction method for multimodal cultivated land data is provided, which may also include a knowledge fusion process, and the specific process includes: using multimodal high-level semantics to design and construct an incomplete multimodal data fusion algorithm model based on deep semantic matching; determining the spatial geometric characteristics of the incomplete multimodal data fusion algorithm model, and designing a modal local invariant graph regularization factor based on the spatial geometric features, coupling the multimodal deep shared features and the original modal features; optimizing and updating the incomplete multimodal data fusion algorithm model through an optimization algorithm to obtain high-level semantic shared features of the multimodal data.

[0105] Different from the existing methods, this embodiment proposes an incomplete multimodal data fusion method based on deep semantic matching based on the deep feature integration characteristics of the deep learning network. The incomplete multimodal data fusion architecture based on deep semantic matching is as follows: Figure 8 As shown, by utilizing the correlation of multimodal high-level semantics, a unified deep learning model that integrates modal private deep networks and modal shared features is designed; in addition, based on the spatial geometric characteristics of the modalities, a modal local invariant graph regularization factor is designed to couple multimodal deep shared features and original modal features; finally, the entire model is optimized and updated using optimization algorithms such as coordinate descent and back propagation to obtain high-level semantic shared features of multimodal data. Experimental results show that the deep incomplete multimodal data fusion algorithm proposed in this embodiment can effectively learn the fusion features of incomplete multimodal data.

[0106] In this embodiment, an unsupervised multi-modal related and unrelated feature co-learning mechanism is also used to uniformly learn the modality private features and modality shared features. Fig. 9As shown in the figure, by separating private features, we can better learn the shared features of inter-modal data, and thus obtain more accurate multimodal data fusion results; experimental results show that the unsupervised multimodal feature co-learning algorithm can effectively learn modal shared features, while completing the dimensionality reduction of high-dimensional multimodal data attributes and improving data fusion accuracy.

[0107] In summary, in this embodiment, due to the lack of a unified knowledge management mechanism in the field of farmland protection, data is in a scattered and disordered storage state, making it difficult to use it efficiently. Through planning and management through knowledge graphs, knowledge is effectively combined with the farmland protection system, and knowledge is used to drive business, making the management of farmland protection data more personalized and intelligent.

[0108] Step 208, based on the knowledge graph after feature fusion, mine the cultivated land data in different modes, and supplement and adjust the knowledge graph according to the mined cultivated land data to complete the construction of the knowledge graph of multimodal cultivated land data.

[0109] In the big data of cultivated land, some cultivated land information may have rich picture information but lack structured attribute description, while another type of cultivated land information may have rich structured graph data. By fully mining the knowledge in the above different modalities, the knowledge graph can be made more powerful in tasks such as self-completion and intelligent question and answer. In this embodiment, by using methods such as knowledge extraction, knowledge fusion, knowledge representation, and knowledge reasoning, a full-process multimodal cultivated land protection knowledge graph is constructed by fully mining and associating knowledge in different modalities, thereby improving the information retrieval capability in cultivated land protection.

[0110] In one embodiment, a method for constructing a knowledge graph of multimodal cultivated land data is provided, and the process may also include: using the graph database Neo4j to store the constructed knowledge graph of multimodal cultivated land data; storing each entity in the knowledge graph through the nodes in the graph database Neo4j, and establishing edges between nodes based on the association relationship between each entity.

[0111] Based on the construction of the knowledge graph in the field of cultivated land protection, a knowledge service system for cultivated land protection big data based on the knowledge graph is developed, which can provide knowledge navigation, intelligent search and other functions in the field of cultivated land protection. The graph database Neo4j is used to store the constructed cultivated land protection knowledge graph. Among them, the basic structure of Neo4j is a graph, which contains nodes, attributes and relationships, and they are all stored independently. When storing the cultivated land protection knowledge graph in Neo4j, the nodes store the entities in the knowledge graph, and the attribute information of the entities is stored in the attributes of the nodes. The edges between the nodes are established through the association relationship between the entities. When defining nodes, labels can be added to the nodes, and the nodes can be organized together using labels to reflect the role of the nodes in the knowledge graph. The attribute information of the node exists in the form of key-value pairs, corresponding to the attribute name and attribute value in the knowledge graph. The number of attribute information is not restricted, which is different from the requirements in the relational database. The information of the edge contains three parts: the starting node, the target node and the relationship name, corresponding to the entity and relationship in the triple, and the edge can also have attributes.

[0112] In one embodiment, a method for constructing a knowledge graph of multimodal farmland data is provided, which can be applied to Fig.10 In the framework of the farmland protection knowledge service system based on knowledge graph shown in the figure, the knowledge service platform and users are effectively combined to support the innovative application of farmland protection knowledge such as knowledge retrieval, knowledge recommendation, and knowledge generation. Fig.10 As shown, the knowledge service system framework for farmland protection based on knowledge graph can include data collection, knowledge extraction, knowledge fusion, knowledge discovery, and knowledge application. Each section realizes different functions and cooperates with each other.

[0113] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0114] In one embodiment, Fig.11 As shown, a knowledge graph construction system for multimodal cultivated land data is provided, including: a cultivated land protection field knowledge system construction module 1110, a knowledge extraction module 1120, a knowledge fusion module 1130 and a graph construction module 1140, wherein:

[0115] The farmland protection domain knowledge system construction module 1110 is used to determine the farmland protection demand, collect farmland protection data based on the farmland protection demand, and construct the farmland protection domain knowledge system based on the farmland protection data;

[0116] The knowledge extraction module 1120 is used to obtain multimodal cultivated land data from the knowledge system of cultivated land protection, extract entities, relationships and attributes of the multimodal cultivated land data based on the multimodal cultivated land data and geographic information, and store them in the knowledge graph to obtain a preliminary knowledge graph;

[0117] The knowledge fusion module 1130 is used to obtain various description information corresponding to entities, relationships, and attributes according to the preliminary knowledge graph, and input various description information into the incomplete multimodal data fusion algorithm model based on deep semantic matching to perform data feature fusion, so as to obtain a knowledge graph after feature fusion;

[0118] The graph construction module 1140 is used to mine the cultivated land data in different modalities based on the knowledge graph after feature fusion, and to supplement and adjust the knowledge graph according to the mined cultivated land data, so as to complete the construction of the knowledge graph of multimodal cultivated land data.

[0119] In one embodiment, the farmland protection domain knowledge system construction module 1110 is also used to obtain a demand analysis report and determine the farmland protection needs based on the demand analysis report; collect farmland protection data based on the farmland protection needs, and extract entities based on the farmland protection data to form a multi-dimensional and layered farmland protection domain knowledge system.

[0120] In one embodiment, the farmland protection domain knowledge system construction module 1110 is also used to determine the top-level concepts of the domain knowledge graph, and to decompose and refine the top-level concepts layer by layer to obtain a concept classification system with a hierarchical structure; to add entity objects extracted from farmland protection data to the concept classification system; to summarize the entity objects to obtain the bottom-level concepts and form a farmland protection domain knowledge system.

[0121] In one embodiment, the knowledge extraction module 1120 is also used to divide multimodal farmland data into structured data, semi-structured and unstructured data; determine the input information containing structured information in the structured data, identifiable marked data and extract entities, relationships and attributes; use automatic technology to extract entities, relationships and attributes from semi-structured and unstructured data; store the extracted entities, relationships and attributes in the knowledge graph to obtain a preliminary knowledge graph.

[0122] In one embodiment, the knowledge extraction module 1120 is also used to obtain pre-established extraction rules, perform text analysis on multimodal farmland data and geographic information, and obtain analysis text description features; based on the analysis text description features, text matching is performed on the multimodal farmland data according to the extraction rules to complete entity extraction.

[0123] In one embodiment, the knowledge extraction module 1120 is also used to obtain labeled data, annotate the multimodal farmland data according to the labeled data, determine the boundaries of words in the text, and obtain classification features; train the BiLSTM model based on the classification features, extract sentence features, and combine the forward and backward hidden state results in the multimodal farmland data according to the sentence features; use the output of the BiLSTM model as the input of the CRF to form a BiLSTM-CRF structure, and perform sequence labeling based on the BiLSTM-CRF structure to complete entity extraction.

[0124] In one embodiment, the knowledge extraction module 1120 is also used to extract dynamic change information, inclusion relationships, and expression forms in multimodal farmland data; calculate the frequency of occurrence of each language pattern based on the dynamic change information, inclusion relationships, and expression forms; match the language pattern corresponding to the highest frequency of occurrence with the entity text to complete relationship extraction.

[0125] In one embodiment, the knowledge fusion module 1130 is also used to utilize multimodal high-level semantic design to construct an incomplete multimodal data fusion algorithm model based on deep semantic matching; determine the spatial geometric characteristics of the incomplete multimodal data fusion algorithm model, and design a modal local invariant graph regularization factor based on the spatial geometric features to couple the multimodal deep shared features and the original modal features; optimize and update the incomplete multimodal data fusion algorithm model through an optimization algorithm to obtain high-level semantic shared features of the multimodal data.

[0126] In one embodiment, the graph construction module 1140 is also used to store the constructed knowledge graph of multimodal farmland data using the graph database Neo4j; each entity in the knowledge graph is stored through the nodes in the graph database Neo4j, and edges between nodes are established based on the association relationship between each entity.

[0127] In one embodiment, a knowledge-based farmland protection service system is constructed based on knowledge graph. Specifically, the village and town land big data knowledge service system is as follows: Fig.12As shown in the figure, through knowledge display, knowledge navigation, intelligent search and other methods, it provides multi-level fine-grained knowledge such as standards, terminology, and business processes, thereby solving the problem of lack of association and organization of data in the field of cultivated land protection and the difficulty of efficient use of relevant knowledge, and providing technical support for knowledge organization and management in the field of cultivated land protection. The search results are displayed in a visual form, and the resources pointed to by the results of knowledge retrieval are browsed, such as policy documents, training courseware, special materials, scientific research results, etc.; the knowledge engine helps users complete online big data analysis, and at the same time, for specific subject areas and specific topics, a customized and process-based knowledge chain is formed to help users complete in-depth data mining, and then obtain valuable knowledge, providing users with intelligent knowledge services.

[0128] System function interface such as Fig.12 As shown in the figure, the knowledge overview can include a visual display of the overall picture of the farmland protection knowledge map; the knowledge classification part provides multi-level and multi-type knowledge according to comprehensive categories (land use status classification, land management law classification, terminology, standards and specifications, etc.), business categories (construction planning, land development and remediation, etc.), and subject categories (technical methods, papers, etc.); knowledge query can include cross-domain query and retrieval services for farmland protection knowledge. Among them, farmland protection business and comprehensive knowledge maps are shown in the figure. Fig.13 As shown, Fig.13 It intuitively displays the 12 first-level categories, 73 second-level categories, and the category relationship between the first-level and second-level categories in the national standard GB / T 21010-2017 "Land Use Status Classification"; uses the mainstream neo4j database to enter the sorted data into the database, links to the Neo4j Browser, and uses the Spring-data-neo4j package to convert the database data into java objects. The inclusion relationship between the 125 related terms in the field of cultivated land protection and the 49 standard specifications is as follows Fig.14 shown.

[0129] Specifically, in this embodiment, the information extraction function is displayed as follows: Fig.15 , 16 As shown, Fig.15 As shown in the figure, enter a paragraph in the search box, and entities and relationships can be extracted through the knowledge extraction algorithm. Then, the results of knowledge extraction can be used to search the constructed knowledge graph. If new entities and relationships are found, they will be added to the graph. If similar entities and relationships already exist, the entities of knowledge fusion are aligned to determine the similarity, so as to determine whether to update to the knowledge graph. In the process, the existing neo4j database is combined for knowledge fusion, such as Fig.17 shown.

[0130] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.18 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for constructing a knowledge graph of multimodal cultivated land data is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0131] Those skilled in the art will understand that Fig.18 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, steps of a method for constructing a knowledge graph of multimodal cultivated land data are implemented.

[0133] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a method for constructing a knowledge graph of multimodal cultivated land data are implemented.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0135] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for constructing a knowledge graph of multimodal cultivated land data, characterized in that: The method comprises: Determine the need for farmland protection, collect farmland protection data based on the need for farmland protection, and construct a knowledge system in the field of farmland protection based on the farmland protection data; Acquire multimodal cultivated land data from the cultivated land protection field knowledge system, extract entities, relationships, and attributes from the multimodal cultivated land data based on the multimodal cultivated land data and geographic information, and store the entities, relationships, and attributes in a knowledge graph to obtain a preliminary knowledge graph; According to the preliminary knowledge graph, each description information corresponding to the entity, relationship, and attribute is obtained, and each description information is input into an incomplete multimodal data fusion algorithm model based on deep semantic matching to perform data feature fusion, so as to obtain a knowledge graph after feature fusion; Based on the knowledge graph after the feature fusion, the cultivated land data in different modalities are mined, and the knowledge graph is supplemented and adjusted according to each of the mined cultivated land data to complete the construction of the knowledge graph of multimodal cultivated land data.

2. The method for constructing a knowledge graph of multimodal cultivated land data according to claim 1, characterized in that: The determining of the farmland protection demand, collecting farmland protection data based on the farmland protection demand, and constructing a farmland protection field knowledge system based on the farmland protection data include: Obtaining a needs analysis report, and determining farmland protection needs based on the needs analysis report; According to the farmland protection needs, farmland protection data is collected, and entities are extracted based on the farmland protection data to form a multi-dimensional and layered farmland protection field knowledge system.

3. The method for constructing a knowledge graph of multimodal cultivated land data according to claim 2, characterized in that: The method further comprises: Determine the top-level concepts of the domain knowledge graph, and decompose and refine them layer by layer based on the top-level concepts to obtain a concept classification system with a hierarchical structure; Adding entity objects extracted from the farmland protection data to the concept classification system; The entity objects are summarized to obtain underlying concepts and form the knowledge system in the field of farmland protection.

4. The method for constructing a knowledge graph of multimodal cultivated land data according to claim 1, characterized in that: The method of obtaining multimodal cultivated land data from the cultivated land protection field knowledge system, extracting entities, relationships, and attributes of the multimodal cultivated land data based on the multimodal cultivated land data and geographic information, and storing the entities, relationships, and attributes in a knowledge graph to obtain a preliminary knowledge graph includes: Dividing the multimodal cultivated land data into structured data, semi-structured data and unstructured data; Determine the input information containing structured information in the structured data, identifiable marked data and extract entities, relationships and attributes; Extracting entities, relationships, and attributes from the semi-structured and unstructured data using automated techniques; The extracted entities, relationships, and attributes are stored in the knowledge graph to obtain a preliminary knowledge graph.

5. The method for constructing a knowledge graph of multimodal cultivated land data according to claim 4, characterized in that: The extracting entities, relationships and attributes of the multimodal cultivated land data based on the multimodal cultivated land data and geographic information includes: Obtaining pre-defined extraction rules, performing text analysis on the multimodal cultivated land data and geographic information, and obtaining analysis text description features; According to the analysis text description features, text matching is performed on the multimodal cultivated land data according to the extraction rules to complete entity extraction.

6. The method for constructing a knowledge graph of multimodal cultivated land data according to claim 4, characterized in that: The extracting entities, relationships and attributes of the multimodal cultivated land data based on the multimodal cultivated land data and geographic information includes: Acquire label data, annotate the multimodal farmland data according to the label data, determine word boundaries in the text, and obtain classification features; Based on the classification features, a BiLSTM model is used for training to extract sentence features, and forward and backward hidden state results in the multimodal cultivated land data are combined according to the sentence features; The output of the BiLSTM model is used as the input of CRF to form a BiLSTM-CRF structure, and sequence labeling is performed based on the BiLSTM-CRF structure to complete entity extraction.

7. The method for constructing a knowledge graph of multimodal cultivated land data according to claim 4, characterized in that: The extracting entities, relationships and attributes of the multimodal cultivated land data based on the multimodal cultivated land data and geographic information includes: Extracting dynamic change information, inclusion relationships, and expression forms from the multimodal cultivated land data; Calculate the occurrence frequency of each language pattern based on the dynamic change information, inclusion relationship, and expression form; The language pattern corresponding to the highest frequency of occurrence is matched with the entity text to complete the relationship extraction.

8. The method for constructing a knowledge graph of multimodal cultivated land data according to claim 1, characterized in that: The method further comprises: Utilize multimodal high-level semantic design to build an incomplete multimodal data fusion algorithm model based on deep semantic matching; Determine the spatial geometric characteristics of the incomplete multimodal data fusion algorithm model, and design a modal local invariant graph regularization factor based on the spatial geometric characteristics to couple the multimodal deep sharing features and the original modal features; The incomplete multimodal data fusion algorithm model is optimized and updated through an optimization algorithm to obtain high-level semantic shared features of the multimodal data.

9. The method for constructing a knowledge graph of multimodal cultivated land data according to claim 1, characterized in that: The method further comprises: The graph database Neo4j is used to store the constructed knowledge graph of multimodal cultivated land data; Each entity in the knowledge graph is stored through the nodes in the graph database Neo4j, and the edges between the nodes are established according to the association relationship between the entities.

10. A knowledge graph construction system for multimodal cultivated land data, characterized in that: The system comprises: A farmland protection domain knowledge system construction module is used to determine farmland protection needs, collect farmland protection data based on the farmland protection needs, and construct a farmland protection domain knowledge system based on the farmland protection data; A knowledge extraction module is used to obtain multimodal cultivated land data from the cultivated land protection field knowledge system, extract entities, relationships, and attributes of the multimodal cultivated land data based on the multimodal cultivated land data and geographic information, and store them in a knowledge graph to obtain a preliminary knowledge graph; A knowledge fusion module is used to obtain various descriptive information corresponding to the entities, relationships, and attributes according to the preliminary knowledge graph, and input each of the descriptive information into an incomplete multimodal data fusion algorithm model based on deep semantic matching to perform data feature fusion, so as to obtain a knowledge graph after feature fusion; The graph construction module is used to mine the cultivated land data in different modes based on the knowledge graph after the feature fusion, and to supplement and adjust the knowledge graph according to the mined cultivated land data, so as to complete the construction of the knowledge graph of multimodal cultivated land data.

Images