Deep learning-based farmland protection text knowledge extraction method and system

Abstract

The application relates to a deep learning-based farmland protection text knowledge extraction method and system. The method comprises the following steps: performing data preprocessing based on a hierarchical clustering algorithm and an information entropy-based data cleaning method to obtain to-be-extracted multi-modal data; processing text data in the to-be-extracted multi-modal data by fusing a rule priori and an inflation gate convolutional neural network; processing remote sensing images in the to-be-extracted multi-modal data by a graph neural network; obtaining geographic information in the to-be-extracted multi-modal data and constructing a relationship between images and texts; constructing an unstructured farmland text semantic extraction network to perform entity recognition, and extracting entity-attribute-value triplets and entity-relation-attribute information triplets according to entity information and farmland attribute information. The use of the inflation gate convolutional neural network and the rule priori for entity extraction and relationship construction can realize efficient and accurate relationship extraction and attribute extraction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cultivated land protection text knowledge extraction, in particular to a cultivated land protection text knowledge extraction method and system based on deep learning. BACKGROUND

[0002] Cultivated land protection data has various forms, including text, images, multimedia data, sensor data and other different modal data, and has the characteristics of complex form, various modalities and large data volume. These data sources are scattered, and there are great differences in the aspects of expression method, knowledge dimension and data structure. There is no fixed form of expression, and the writing is highly professional, so it is difficult to apply existing models to extract the knowledge of cultivated land protection text content.

[0003] In the field of knowledge extraction, in 2019, Liu et al. connected the entities in the knowledge graph through the SameAs predicate, and implemented relationship reasoning between different resources. In 2020, Wang et al. proposed a multi-modal graph Richpedia, which uses a rule-based relationship extraction template to generate multi-modal semantic relationships between image entities with the help of hyperlink information in Wikipedia image descriptions. In 2022, Bai C proposed a new model, i.e. e-commerce knowledge extraction based on multi-modal machine reading comprehension (EKE-MRC), to better extract e-commerce product attributes.

[0004] However, traditional artificial intelligence neural network methods have achieved good knowledge extraction results under the driving of large-scale and high-quality data, but the extraction effect is not satisfactory in professional fields, and the model training and deployment require high computing power. Therefore, the data extracted by traditional cultivated land protection text knowledge extraction methods often lacks correlation and organization, and relevant knowledge is difficult to be efficiently utilized. SUMMARY

[0005] Therefore, in order to solve the above technical problems, a cultivated land protection text knowledge extraction method and system based on deep learning are provided, which can realize efficient and accurate relationship extraction and attribute extraction, thereby providing data, technology and service support for cultivated land protection and management.

[0006] A cultivated land protection text knowledge extraction method based on deep learning, the method comprising:

[0007] Obtaining multi-modal cultivated land protection data, and using a data cleaning method based on hierarchical clustering algorithm and information entropy to preprocess the multi-modal cultivated land protection data to obtain multi-modal data to be extracted;

[0008] The text data in the to-be-extracted multi-modal data is processed by fusing a rule priori and an inflation gate convolutional neural network, and a feature vector is extracted to obtain entity information of the text.

[0009] The remote sensing image in the to-be-extracted multi-modal data is processed by a graph neural network to obtain entity information of the image; geographic information in the to-be-extracted multi-modal data is obtained, a relationship between the image and the text is constructed according to spatial information in the geographic information; and cultivated land attribute information is determined according to the geographic information.

[0010] An unstructured cultivated land text semantic extraction network Bert-BiLSTM-CRF is constructed, entity recognition is performed on the to-be-extracted multi-modal data by using the unstructured cultivated land text semantic extraction network, and entity-attribute-attribute value triplets are extracted according to the entity information and the cultivated land attribute information; entity-relation-attribute information triplets are extracted according to the entity information, the relationship, and the cultivated land attribute information.

[0011] According to the entity-attribute-attribute value triplets and the entity-relation-attribute information triplets, a text knowledge six-tuple directed graph for cultivated land protection is constructed.

[0012] In one of the embodiments, multi-modal cultivated land protection data is obtained, and a data cleaning method based on a hierarchical clustering algorithm and information entropy is used to pre-process the multi-modal cultivated land protection data, including:

[0013] Remote sensing images, text information, and geographic information are collected from unstructured text as multi-modal cultivated land protection data, and the multi-modal cultivated land protection data is stored in a database;

[0014] When knowledge extraction is performed, the multi-modal cultivated land protection data is extracted from the database;

[0015] The multi-modal cultivated land protection data is cleaned and format-converted by using the data cleaning method based on the hierarchical clustering algorithm and the information entropy.

[0016] In one of the embodiments, the text data in the to-be-extracted multi-modal data is processed by fusing a rule priori and an inflation gate convolutional neural network, and a feature vector is extracted to obtain entity information of the text, including:

[0017] The text data in the to-be-extracted multi-modal data is processed by fusing a rule priori and an inflation gate convolutional neural network, and a feature vector is extracted to obtain entity information of the text, including:

[0018] The processed text data is input into an inflation gate convolutional layer in the inflation gate convolutional neural network, and a feature vector is extracted by the inflation gate convolutional layer.

[0019] The dependency relationship in the text data is captured through the self-attention mechanism in the dilated gate convolutional neural network.

[0020] The entity information of the text is obtained based on the feature vector and the dependency relationship.

[0021] In one of the embodiments, the method further comprises:

[0022] The word-mixed vector is obtained according to the text data in the multi-modal data to be extracted, and the word-mixed vector is input into the dilated gate convolutional neural network.

[0023] The subject in the multi-modal data to be extracted is identified through the relationship capturing by the self-attention mechanism in the dilated gate convolutional neural network and the verification by combining the prior knowledge.

[0024] In one of the embodiments, the method further comprises:

[0025] The position vector is obtained according to the geographic information in the multi-modal data to be extracted, and the position vector is input into the dilated gate convolutional neural network.

[0026] The object in the multi-modal data to be extracted is identified through the relationship capturing by the self-attention mechanism in the dilated gate convolutional neural network and the feature extraction by the dilated gate convolutional layer, and the verification by combining the prior knowledge.

[0027] In one of the embodiments, the unstructured farmland text semantic extraction network is used for entity recognition of the multi-modal data to be extracted, and an entity-attribute-attribute value triple is extracted according to the entity information and the farmland attribute information, including:

[0028] The word dependency relationship in the multi-modal data to be extracted is captured through the multi-layer encoder of Bert in the unstructured farmland text semantic extraction network Bert-BiLSTM-CRF.

[0029] The context analysis of the text semantic of the multi-modal data to be extracted is performed through the BiLSTM part in the unstructured farmland text semantic extraction network Bert-BiLSTM-CRF, the semantic structure of the farmland protection text is identified, and the text feature is extracted according to the semantic structure.

[0030] The entity-attribute-attribute value triple of the farmland protection is extracted according to the word dependency relationship and the text feature through the unstructured farmland text semantic extraction network Bert-BiLSTM-CRF.

[0031] In one of the embodiments, the unstructured farmland text semantic extraction network Bert-BiLSTM-CRF is trained using an end-to-end method.

[0032] In one of the embodiments, the method further comprises:

[0033] Collecting unstructured data of farmland protection, and manually annotating the unstructured data;

[0034] Dividing the annotated data into a training set and a validation set, obtaining entities and relationships, and storing them into a database;

[0035] Verifying the unstructured farmland text semantic extraction network Bert-BiLSTM-CRF using the validation set, and evaluating the entity recognition effect using F1.

[0036] A farmland protection text knowledge extraction system based on deep learning, comprising:

[0037] A data processing module, configured to acquire multi-modal farmland protection data, and pre-process the multi-modal farmland protection data using a data cleaning method based on hierarchical clustering algorithm and information entropy to obtain to-be-extracted multi-modal data;

[0038] A first extraction module, configured to process text data in the to-be-extracted multi-modal data by fusing rule priori and dilated gate convolutional neural network, and extract feature vectors to obtain entity information of the text;

[0039] A second extraction module, configured to process remote sensing images in the to-be-extracted multi-modal data by a graph neural network to obtain entity information of the images, acquire geographic information in the to-be-extracted multi-modal data, construct a relationship between the images and the text according to spatial information in the geographic information, and determine farmland attribute information according to the geographic information;

[0040] A knowledge extraction module, configured to construct an unstructured farmland text semantic extraction network Bert-BiLSTM-CRF, perform entity recognition on the to-be-extracted multi-modal data using the unstructured farmland text semantic extraction network, and extract entity-attribute-attribute value triplets according to the entity information and the farmland attribute information, and extract entity-relation-attribute information triplets according to the entity information, the relationship, and the farmland attribute information;

[0041] A directed graph construction module, configured to construct a text knowledge six-tuple directed graph of farmland protection according to the entity-attribute-attribute value triplets and the entity-relation-attribute information triplets.

[0042] In one of the embodiments, the data processing module is further configured to collect remote sensing images, text information, and geographic information from unstructured text as multi-modal cultivated land protection data, and store the multi-modal cultivated land protection data in a database; when performing knowledge extraction, the multi-modal cultivated land protection data is extracted from the database; and a data cleaning method based on hierarchical clustering algorithm and information entropy is used to clean and format the multi-modal cultivated land protection data.

[0043] The cultivated land protection text knowledge extraction method and system based on deep learning can effectively eliminate invalid and redundant data interference on subsequent text knowledge extraction by using the data cleaning method based on hierarchical clustering algorithm and information entropy; the entity extraction and relationship construction are performed using the rule priori and dilated gate convolutional neural network, so that the knowledge can be mutually supplemented and inherited, efficient and accurate relationship extraction and attribute extraction can be realized, and data, technology and service support can be provided for cultivated land protection and management. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 An application environment diagram of the cultivated land protection text knowledge extraction method based on deep learning in one embodiment;

[0045] Figure 2 A flowchart of the cultivated land protection text knowledge extraction method based on deep learning in one embodiment;

[0046] Figure 3 A model framework diagram of the rule priori and dilated gate convolutional neural network in one embodiment;

[0047] Figure 4 A knowledge extraction method diagram of multi-modal cultivated land big data in one embodiment;

[0048] Figure 5 A Bert-BiLSTM-CRF network structure diagram in one embodiment;

[0049] Figure 6 An interface diagram of knowledge extraction results in one embodiment;

[0050] Figure 7 A structural block diagram of the cultivated land protection text knowledge extraction system based on deep learning in one embodiment;

[0051] Figure 8 An internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0053] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe the extraction module, but these extraction modules are not limited by these terms. These terms are only used to distinguish the first extraction module from another extraction module. For example, without departing from the scope of the present application, the first extraction module can be called the second extraction module, and similarly, the second extraction module can be called the first extraction module. The first extraction module and the second extraction module are both extraction modules, but they are not the same extraction module.

[0054] The deep learning-based farmland protection text knowledge extraction method provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 As shown in Figure 1 The application environment includes a computer device 110. The computer device 110 can obtain multi-modal farmland protection data, and pre-process the multi-modal farmland protection data by using a data cleaning method based on a hierarchical clustering algorithm and information entropy to obtain to-be-extracted multi-modal data. The computer device 110 can process the text data in the to-be-extracted multi-modal data by fusing rule priori and dilated gate convolutional neural network, and extract a feature vector to obtain entity information of the text. The computer device 110 can process the remote sensing image in the to-be-extracted multi-modal data by using a graph neural network to obtain entity information of the image. The geographic information in the to-be-extracted multi-modal data is obtained, the relationship between the image and the text is constructed according to the spatial information in the geographic information, and the farmland attribute information is determined according to the geographic information. The computer device 110 can construct an unstructured farmland text semantic extraction network Bert-BiLSTM-CRF, use the unstructured farmland text semantic extraction network to perform entity recognition on the to-be-extracted multi-modal data, and extract entity-attribute-attribute value triplets according to the entity information and the farmland attribute information. According to the entity information, the relationship, and the farmland attribute information, entity-relation-attribute information triplets are extracted. The computer device 110 can construct a text knowledge six-tuple directed graph of farmland protection according to the entity-attribute-attribute value triplets and the entity-relation-attribute information triplets. The computer device 110 can be, but is not limited to, various personal computers, notebook computers, smart phones, robots and the like.

[0055] In one embodiment, as shown in Figure 2 A deep learning-based farmland protection text knowledge extraction method is provided, including the following steps:

[0056] At step 202, the multi-modal cultivated land protection data is acquired, and a data cleaning method based on a hierarchical clustering algorithm and information entropy is used to pre-process the multi-modal cultivated land protection data to obtain multi-modal data to be extracted.

[0057] The multi-modal cultivated land protection data can include remote sensing images, text information, geographic information, and other data related to cultivated land protection. The computer device can use a data cleaning method based on a hierarchical clustering algorithm and information entropy to clean and format the multi-modal cultivated land protection data, so as to extract the entity information thereof.

[0058] In one embodiment, the deep learning-based cultivated land protection text knowledge extraction method can further include a data preprocessing process, which specifically includes: collecting remote sensing images, text information, and geographic information from unstructured text as multi-modal cultivated land protection data, and storing the multi-modal cultivated land protection data in a database; when performing knowledge extraction, extracting the multi-modal cultivated land protection data from the database; and using a data cleaning method based on a hierarchical clustering algorithm and information entropy to clean and format the multi-modal cultivated land protection data.

[0059] The unstructured text such as cultivated land protection-related policies and regulations and land data reports is the core of cultivated land protection knowledge learning. In this embodiment, data collection and database establishment can be completed to facilitate data processing.

[0060] At step 204, the text data in the multi-modal data to be extracted is processed by fusing a rule priori and an expanded gate convolutional neural network, and a feature vector is extracted to obtain entity information of the text.

[0061] The cultivated land protection entity extraction and relationship construction model with the fused rule priori and the expanded gate convolutional neural network can be designed in the computer device, and the output unit receptive field can be increased by the expanded convolution.

[0062] In one embodiment, the deep learning-based cultivated land protection text knowledge extraction method can include a data processing process by fusing a rule priori and an expanded gate convolutional neural network, which specifically includes: the expanded gate convolutional neural network performs character and word segmentation processing on the text data in the multi-modal data to be extracted to obtain processed text data; the processed text data is input into an expanded gate convolutional layer in the expanded gate convolutional neural network, and a feature vector is extracted by the expanded gate convolutional layer; a self-attention mechanism in the expanded gate convolutional neural network is used to capture the dependency relationship in the text data; and entity information of the text is obtained based on the feature vector and the dependency relationship.

[0063] Specifically, as Figure 3As shown, in one embodiment, the provided deep learning-based farmland protection text knowledge extraction method can further include a process of identifying a subject, and the specific process includes: obtaining a word-mixed vector from text data in the multi-modal data to be extracted, and inputting the word-mixed vector into an expanded gate convolutional neural network; capturing relationships through a self-attention mechanism in the expanded gate convolutional neural network, and verifying in combination with prior knowledge to identify the subject in the multi-modal data to be extracted.

[0064] In one embodiment, as Figure 3 shown, the provided deep learning-based farmland protection text knowledge extraction method can further include a process of identifying an object, and the specific process includes: obtaining a location vector from geographic information in the multi-modal data to be extracted, and inputting the location vector into an expanded gate convolutional neural network; capturing relationships through a self-attention mechanism in the expanded gate convolutional neural network, and extracting features through an expanded gate convolutional layer, verifying in combination with prior knowledge to identify the object in the multi-modal data to be extracted.

[0065] As Figure 3 shown, in this embodiment, after the computer device obtains the multi-modal data to be extracted, the expanded gate convolutional neural network model first performs word and character processing on the text data in the multi-modal data to be extracted, and then inputs the text data in the training set into the expanded gate convolutional layer to extract feature vectors. When processing the text data, the text context width can be expanded to extract the global information of the text, and thus the text semantics can be better understood. The formula for processing data by the expanded gate one-dimensional convolutional layer can be represented as: y = σ (Wx + b) * x ; where Conv1D(·) represents one-dimensional convolution, x represents the vector sequence to be processed, σ(·) represents a gating function.

[0066] Step 206, processing the remote sensing image in the multi-modal data to be extracted through the graph neural network to obtain entity information of the image; obtaining geographic information in the multi-modal data to be extracted, constructing the relationship between the image and the text according to the spatial information in the geographic information; and determining the farmland attribute information according to the geographic information.

[0067] In order to fully utilize the rich information structure brought by multi-modal data, the entities, relationships, and attributes are extracted in a differentiated manner, as Figure 4 shown. Specifically, for remote sensing images, text information, and geographic information, convolutional neural networks and graph neural networks are used for extraction, respectively, so that more suitable feature information can be obtained. In addition, the spatial information contained in the geographic information data is fully utilized to accurately construct the relationship between the image and the text. Based on the geographic location information, the farmland information in the image and the text can also be obtained, which provides accurate data support for farmland protection.

[0068] Step 208: Construct the unstructured farmland text semantic extraction network Bert-BiLSTM-CRF, use the unstructured farmland text semantic extraction network to perform entity recognition on the multimodal data to be extracted, and extract entity-attribute-attribute value triples based on entity information and farmland attribute information; extract entity-relationship-attribute information triples based on entity information, relation, and farmland attribute information.

[0069] To address the diverse formats and land attribute types of farmland documents, and the lack of a unified farmland attribute extraction model in the field of farmland protection, a Bert-BiLSTM-CRF network for unstructured farmland text semantic extraction was constructed. In one embodiment, the Bert-BiLSTM-CRF network was trained using an end-to-end method, such as... Figure 5 As shown, the constructed unstructured farmland text semantic extraction network Bert-BiLSTM-CRF can include several parts: a Bert module, a BiLSTM module, and a CRF module.

[0070] In one embodiment, a deep learning-based method for extracting knowledge from farmland protection text may further include a data processing procedure using the Bert-BiLSTM-CRF unstructured farmland text semantic extraction network. Specifically, this procedure includes: capturing word dependencies within sentences in the multimodal data to be extracted using the multi-layer encoder of the Bert-BiLSTM-CRF network; performing contextual semantic parsing of the multimodal data to be extracted using the BiLSTM part of the Bert-BiLSTM-CRF network to identify the semantic structure of the farmland protection text and extracting text features based on the semantic structure; and extracting entity-attribute-attribute value triples for farmland protection based on word dependencies and text features using the Bert-BiLSTM-CRF network.

[0071] Specifically, such as Figure 5 As shown, the pre-trained BERT multi-layer encoder can capture word dependencies within sentences. By parsing the semantic context of the text, it can identify the semantic structure of farmland protection texts and extract text features covering long-distance semantics. Secondly, an end-to-end method is used to train the BERT-BiLSTM-CRF network, which can perform word segmentation and semantic recognition on farmland protection texts. Words are segmented according to Chinese grammar, supplemented by farmland protection entities and concepts. By establishing long-distance dependencies between farmland protection entities and sentences, and simultaneously utilizing natural language understanding technology for semantic understanding and textual intent recognition, entity-attribute-attribute value triples for farmland protection are extracted.

[0072] In step 210, a text knowledge six-element directed graph for farmland protection is constructed according to the entity-attribute-attribute value triplets and the entity-relation-attribute information triplets.

[0073] The computer device can construct a text knowledge six-element directed graph for farmland protection according to the entity-attribute-attribute value triplets and the entity-relation-attribute information triplets T =(E,R,A,V,T r ,T a ), wherein E={e1,e2,…,e |e|}, R={r1,r2,…,r |R|}, A={a1,a2,…,a |A|}, V={v1,v2,…,v |V|} respectively denote the sets of farmland protection text knowledge entities, relations, attributes and values, T r ={(h,r,t)|h,r∈E,r∈R}, T a ={(e,a,v)|e∈E,a∈A,v∈V} are the sets of relation triplets and attribute triplets.

[0074] In one embodiment, the deep learning-based farmland protection text knowledge extraction method provided can further include a process of evaluating the entity recognition effect, and the specific process includes: collecting unstructured data of farmland protection, and manually labeling the unstructured data; dividing the labeled data into a training set and a validation set, obtaining entities and relations and storing them in a database; verifying the unstructured farmland text semantic extraction network Bert-BiLSTM-CRF using the validation set, and evaluating the entity recognition effect using F1.

[0075] Using the deep learning-based farmland protection text knowledge extraction method provided in the present application to extract farmland protection related data, for farmland protection technical regulations, planning texts, reports, literature and other unstructured data, manually label, use 80% as the training set and 20% as the validation set, obtain 1518 entities and 1716 relations, store the entities and relations data in the Neo4j database; evaluate the entity recognition effect using F1. The calculation method formula can be expressed as: Where TP is the correct positive example, FP is the false positive example, and FN is the false negative example. The labeling accuracy is shown in the following table:

[0076] Labeling accuracy of various entity types

[0077]

[0078] In the embodiment, after the farmland protection related data is extracted by using the farmland protection text knowledge extraction method based on deep learning provided in the application, the corresponding knowledge extraction display interface is as shown in Figure 6 The inputted multi-modal data to be extracted can be "land economics is derived from land basic terminology", and the extraction result is "land", "economics", "derived from", "land", "basic", and "terminology"; wherein, the subject is land economics; the relationship is derived from; and the other subject is land terminology.

[0079] It should be understood that, although each step in the above flowchart is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise explicitly stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the above flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0080] In one embodiment, as shown in Figure 7 A farmland protection text knowledge extraction system based on deep learning is provided, comprising a data processing module 710, a first extraction module 720, a second extraction module 730, a knowledge extraction module 740, and a directed graph construction module 750, wherein:

[0081] The data processing module 710 is configured to acquire multi-modal farmland protection data, and pre-process the multi-modal farmland protection data by using a data cleaning method based on a hierarchical clustering algorithm and information entropy, to obtain multi-modal data to be extracted;

[0082] The first extraction module 720 is configured to process the text data in the multi-modal data to be extracted by fusing rule priori and dilated gate convolutional neural network, and extract a feature vector to obtain entity information of the text;

[0083] The second extraction module 730 is configured to process the remote sensing image in the multi-modal data to be extracted by using a graph neural network, to obtain entity information of the image; acquire geographic information in the multi-modal data to be extracted, construct the relationship between the image and the text according to the spatial information in the geographic information; and determine farmland attribute information according to the geographic information;

[0084] The knowledge extraction module 740 is configured to construct an unstructured farmland text semantic extraction network Bert-BiLSTM-CRF, perform entity recognition on the multi-modal data to be extracted by using the unstructured farmland text semantic extraction network, and extract entity-attribute-attribute value triplets according to the entity information and the farmland attribute information; and extract entity-relation-attribute information triplets according to the entity information, the relation, and the farmland attribute information.

[0085] The directed graph construction module 750 is configured to construct a text knowledge six-element directed graph of farmland protection according to the entity-attribute-attribute value triplets and the entity-relation-attribute information triplets.

[0086] In an embodiment, the data processing module 710 is further configured to collect remote sensing images, text information, and geographic information from unstructured text as multi-modal farmland protection data, and store the multi-modal farmland protection data in a database; when performing knowledge extraction, extract the multi-modal farmland protection data from the database; and perform data cleaning and format conversion processing on the multi-modal farmland protection data by using a data cleaning method based on a hierarchical clustering algorithm and information entropy.

[0087] In an embodiment, the first extraction module 720 is further configured to expand a gate convolutional neural network to perform character and word segmentation on the text data in the multi-modal data to be extracted, to obtain processed text data; input the processed text data into an expansion gate convolutional layer in the expansion gate convolutional neural network, extract a feature vector through the expansion gate convolutional layer; capture a dependency relationship in the text data through a self-attention mechanism in the expansion gate convolutional neural network; and obtain entity information of the text based on the feature vector and the dependency relationship.

[0088] In an embodiment, the first extraction module 720 is further configured to obtain a word-character mixed vector according to the text data in the multi-modal data to be extracted, input the word-character mixed vector into the expansion gate convolutional neural network, capture a relationship through the self-attention mechanism in the expansion gate convolutional neural network, and verify in combination with prior knowledge to identify a subject in the multi-modal data to be extracted.

[0089] In an embodiment, the first extraction module 720 is further configured to obtain a position vector according to the geographic information in the multi-modal data to be extracted, input the position vector into the expansion gate convolutional neural network, capture a relationship through the self-attention mechanism in the expansion gate convolutional neural network, and extract features through the expansion gate convolutional layer, verify in combination with prior knowledge, and identify an object in the multi-modal data to be extracted.

[0090] In one embodiment, the knowledge extraction module 740 is further configured to capture word dependency relationships within sentences in the multi-modal data to be extracted by a multi-layer encoder of Bert in the unstructured cultivated land text semantic extraction network Bert-BiLSTM-CRF; perform context analysis of text semantics on the multi-modal data to be extracted by the BiLSTM part in the unstructured cultivated land text semantic extraction network Bert-BiLSTM-CRF, identify the semantic structure of the cultivated land protection text, and extract text features according to the semantic structure; and extract entity-attribute-attribute value triplets of cultivated land protection according to the word dependency relationships and the text features by the unstructured cultivated land text semantic extraction network Bert-BiLSTM-CRF.

[0091] In one embodiment, the unstructured cultivated land text semantic extraction network Bert-BiLSTM-CRF is trained using an end-to-end method.

[0092] In one embodiment, the knowledge extraction module 740 is further configured to collect unstructured data of cultivated land protection, and manually annotate the unstructured data; divide the annotated data into a training set and a validation set, obtain entities and relationships, and store them in a database; use the validation set to verify the unstructured cultivated land text semantic extraction network Bert-BiLSTM-CRF, and evaluate the entity recognition effect using F1.

[0093] In one embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in Figure 8 The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a cultivated land protection text knowledge extraction method based on deep learning. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, trackball, or touchpad arranged on the shell of the computer device. The input device can also be an external keyboard, touchpad, or mouse, etc.

[0094] Those skilled in the art can understand that, Figure 8The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0095] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the deep learning based farmland protection text knowledge extraction method when executing the computer program.

[0096] In one embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program is executed by a processor to implement the steps of the deep learning based farmland protection text knowledge extraction method.

[0097] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium and can include the processes of the above-mentioned embodiments when executed. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0098] The technical features of the above embodiments can be combined in any way. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0099] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific and detailed manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A deep learning-based farmland protection text knowledge extraction method, characterized in that, The method comprises: acquiring multi-modal cultivated land protection data, and pre-processing the multi-modal cultivated land protection data by using a data cleaning method based on a hierarchical clustering algorithm and information entropy to obtain multi-modal data to be extracted; processing text data in the multi-modal data to be extracted by fusing rule priori and an expanded gate convolutional neural network, and extracting a feature vector to obtain entity information of the text, comprising: performing character and word processing on the text data in the multi-modal data to be extracted by the expanded gate convolutional neural network to obtain processed text data; inputting the processed text data into an expanded gate convolutional layer in the expanded gate convolutional neural network, and extracting a feature vector through the expanded gate convolutional layer; capturing a dependency relationship in the text data through a self-attention mechanism in the expanded gate convolutional neural network; obtaining entity information of the text based on the feature vector and the dependency relationship; obtaining a word-mixed vector from the text data in the multi-modal data to be extracted, and inputting the word-mixed vector into the expanded gate convolutional neural network; capturing a relationship through the self-attention mechanism in the expanded gate convolutional neural network, and verifying in combination with priori knowledge to identify a subject in the multi-modal data to be extracted; processing remote sensing images in the multi-modal data to be extracted by a graph neural network to obtain entity information of the images; acquiring geographic information in the multi-modal data to be extracted, constructing a relationship between the images and the text according to spatial information in the geographic information; and determining cultivated land attribute information according to the geographic information; constructing an unstructured cultivated land text semantic extraction network Bert-BiLSTM-CRF, performing entity recognition on the multi-modal data to be extracted by using the unstructured cultivated land text semantic extraction network, and extracting entity-attribute-attribute value triplets according to the entity information and the cultivated land attribute information; extracting entity-relation-attribute information triplets according to the entity information, the relationship between the images and the text, and the cultivated land attribute information; constructing a text knowledge six-tuple directed graph of cultivated land protection according to the entity-attribute-attribute value triplets and the entity-relation-attribute information triplets. 2.The deep learning-based cultivated land protection text knowledge extraction method according to claim 1, characterized in that, acquiring multi-modal cultivated land protection data, and pre-processing the multi-modal cultivated land protection data by using a data cleaning method based on a hierarchical clustering algorithm and information entropy, comprising: collecting remote sensing images, text information, and geographic information from unstructured text as multi-modal cultivated land protection data, and storing the multi-modal cultivated land protection data in a database; when performing knowledge extraction, extracting the multi-modal cultivated land protection data from the database; cleaning the multi-modal cultivated land protection data by using the data cleaning method based on the hierarchical clustering algorithm and the information entropy, and performing format conversion processing. 3.The deep learning-based cultivated land protection text knowledge extraction method according to claim 1, characterized in that, The method further comprises: inputting a position vector obtained from the geographic information in the multi-modal data to be extracted into the expanded gate convolutional neural network. The relationship is captured through the self-attention mechanism in the dilated gate convolutional neural network, and the feature is extracted through the dilated gate convolutional layer, the prior knowledge is verified, and the object in the to-be-extracted multi-modal data is identified. 4.The deep learning-based cultivated land protection text knowledge extraction method of claim 1, wherein, The non-structured cultivated land text semantic extraction network is used for entity recognition on the to-be-extracted multi-modal data, and entity-attribute-attribute value triplets are extracted according to the entity information and cultivated land attribute information, including: The multi-layer encoder of the Bert in the non-structured cultivated land text semantic extraction network Bert-BiLSTM-CRF captures the word dependency relationship inside the sentence in the to-be-extracted multi-modal data; The BiLSTM part in the non-structured cultivated land text semantic extraction network Bert-BiLSTM-CRF performs context analysis on the text semantics of the to-be-extracted multi-modal data, identifies the semantic structure of the cultivated land protection text, and extracts text features according to the semantic structure; The non-structured cultivated land text semantic extraction network Bert-BiLSTM-CRF uses an end-to-end method to complete the training. 5.The deep learning-based cultivated land protection text knowledge extraction method according to claim 4, characterized in that, The method further comprises: 6.The deep learning-based cultivated land protection text knowledge extraction method according to claim 1, characterized in that, Collecting non-structured data of cultivated land protection, and manually labeling the non-structured data; Divide the labeled data into a training set and a validation set, obtain entities and relationships, and store them in a database; The validation set is used to verify the non-structured cultivated land text semantic extraction network Bert-BiLSTM-CRF, and the F1 evaluation is used to evaluate the entity recognition effect. The system comprises: 7.A deep learning-based farmland protection text knowledge extraction system, characterized in that, A data processing module is configured to obtain multi-modal cultivated land protection data, and pre-process the multi-modal cultivated land protection data using a data cleaning method based on a hierarchical clustering algorithm and information entropy to obtain to-be-extracted multi-modal data; A first extraction module is configured to process text data in the to-be-extracted multi-modal data by fusing rule priori and a dilated gate convolutional neural network, and extract feature vectors to obtain entity information of the text, including: the dilated gate convolutional neural network processes the text data in the to-be-extracted multi-modal data by character and word, to obtain processed text data; the processed text data is input into a dilated gate convolutional layer in the dilated gate convolutional neural network, and the dilated gate convolutional layer extracts feature vectors; the self-attention mechanism in the dilated gate convolutional neural network captures the dependency relationship in the text data; the entity information of the text is obtained based on the feature vectors and the dependency relationship; a word-character mixed vector is obtained according to the text data in the to-be-extracted multi-modal data, and the word-character mixed vector is input into the dilated gate convolutional neural network; the self-attention mechanism in the dilated gate convolutional neural network captures the relationship, and verifies the subject in the to-be-extracted multi-modal data by combining the prior knowledge; ​ The second extraction module is configured to process remote sensing images in the multi-modal data to be extracted by using a graph neural network to obtain entity information of the images; obtain geographic information in the multi-modal data to be extracted, construct a relationship between the images and the text according to spatial information in the geographic information; and determine cultivated land attribute information according to the geographic information; The knowledge extraction module is configured to construct an unstructured cultivated land text semantic extraction network Bert-BiLSTM-CRF, perform entity recognition on the multi-modal data to be extracted by using the unstructured cultivated land text semantic extraction network, and extract entity-attribute-attribute value triplets according to the entity information and the cultivated land attribute information; extract entity-relation-attribute information triplets according to the entity information, the relationship between the images and the text, and the cultivated land attribute information; The directed graph construction module is configured to construct a text knowledge six-element directed graph of cultivated land protection according to the entity-attribute-attribute value triplets and the entity-relation-attribute information triplets. 8.The deep learning-based cultivated land protection text knowledge extraction system according to claim 7, wherein, The data processing module is further configured to collect remote sensing images, text information, and geographic information from unstructured text as multi-modal cultivated land protection data, and store the multi-modal cultivated land protection data in a database; when performing knowledge extraction, extract the multi-modal cultivated land protection data from the database; and perform data cleaning and format conversion processing on the multi-modal cultivated land protection data by using a data cleaning method based on a hierarchical clustering algorithm and information entropy.

Images