Wheat disease and insect pest multi-modal question and answer method fusing multi-modal knowledge graph
By constructing a wheat disease and pest question-and-answer system based on a multimodal knowledge graph, the problem of insufficient flexibility of existing systems in multimodal question-and-answer scenarios is solved, enabling accurate diagnosis and control of wheat diseases and pests, and improving control efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-30
- Publication Date
- 2026-04-10
AI Technical Summary
Existing knowledge graph question answering systems lack flexibility and applicability when facing multimodal question answering scenarios, making it difficult to effectively combine images and question context for accurate disease and pest diagnosis and control guidance.
This paper proposes a multimodal question-answering method for wheat diseases and pests that integrates a multimodal knowledge graph. By acquiring a multimodal dataset and constructing a multimodal question-answering model, the method combines the knowledge graph and the question-answering model to achieve multimodal question answering for wheat diseases and pests. This includes multimodal feature extraction, entity category and intent classification, and topic entity recognition using a heterogeneous attention mechanism and a noise-weighted averaging method.
It has improved the flexibility and accuracy of wheat disease and pest diagnosis and control, reduced yield losses caused by diseases and pests, and provided more comprehensive and flexible control guidance.
Smart Images
Figure CN121833867A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of knowledge graph, in particular to a wheat pest and disease multi-modal question answering method based on multi-modal knowledge graph. BACKGROUND
[0002] Pest and disease is one of the major hazards affecting crop growth, and is also an important factor restricting the development of agricultural industrialization and the economic benefits of agriculture. According to the research estimates of the United Nations Food and Agriculture Organization, the annual loss rate of food crops caused by agricultural pests and diseases is as high as 40%. Therefore, effective prevention and control of crop pests and diseases is a key link to reduce food loss and promote modern efficient agriculture. Timely diagnosis and control of agricultural pests and diseases must be based on sufficient knowledge of agricultural pest and disease control. Therefore, agricultural pest and disease knowledge is the basis for rapid, accurate and efficient prevention and control of crop pests and diseases, and has important practical significance for promoting food production and income, and ensuring agricultural production and food security.
[0003] Knowledge question answering based on knowledge graph (KGQA) provides a new perspective and important means for quickly and accurately obtaining crop pest and disease knowledge. On the one hand, knowledge graph (KG) is essentially a semantic network with strong semantic processing and interconnection organization capabilities, which can integrate multi-source heterogeneous fragmented knowledge; on the other hand, knowledge question answering, as one of the typical application scenarios of knowledge graph, can receive user requirements described in natural language form, and automatically obtain accurate answers through steps such as question analysis, knowledge acquisition, knowledge reasoning and answer generation. It is an important form of new generation information retrieval system and an important paradigm for precise human-computer interaction. Knowledge graph and its knowledge engine technology have become the infrastructure of artificial intelligence systems. Therefore, combining intelligent question answering technology to realize intelligent knowledge question answering of agricultural pests and diseases is of important academic significance and application value for pest and disease diagnosis and prevention decision-making, improving pest and disease prevention efficiency and accuracy, and thus reducing food loss and ensuring food security.
[0004] Currently, multi-hop reasoning methods for knowledge graphs can be broadly categorized into two types: semantic parsing-based methods and neural reasoning-based methods. The former uses templates and annotation information to parse the question, transforming it into a graph query statement, and then executes this query to obtain the answer. The latter extracts corresponding features from the question and answer, designs a scoring function to evaluate the semantic relevance between candidate answers and the question, and finally selects the candidate answer with the highest score as the final answer. While these methods have achieved some success, they only focus on pure text question answering and are not suitable for multimodal question answering scenarios that require combining images and question context. That is, for a given image, different question contexts lead to different answers, which obviously results in insufficient flexibility and applicability of the question answering system. Summary of the Invention
[0005] In order to solve the above-mentioned technical problems, this application proposes the following technical solution: Firstly, embodiments of this application provide a multimodal question-answering method for wheat diseases and pests that integrates a multimodal knowledge graph, including: Obtain a multimodal wheat disease and pest question-and-answer dataset; By linking and integrating heterogeneous knowledge from different data sources, a multimodal wheat disease and pest knowledge graph can be constructed. A multimodal question-and-answer model for wheat diseases and pests was constructed and trained using the multimodal wheat disease and pest question-and-answer dataset. A wheat disease and pest diagnosis and treatment system was constructed by combining a modal wheat disease and pest knowledge graph with a trained multimodal wheat disease and pest question-answering model.
[0006] In one possible implementation, obtaining the multimodal wheat disease and pest question-and-answer dataset includes: The acquired text modality and image modality data are preprocessed by cleaning, encoding, deduplication, and removal of irrelevant samples to obtain the original corpus, which is then stored in a txt file. Categorize according to subject entity type and intent type; The image is labeled with the subject entity and entity category corresponding to the question context, and the corresponding intent category is labeled according to the question. This process is repeated for each original multimodal question-and-answer sample until all samples are labeled. The labeled samples are then formatted and exported in JSON format to complete the construction of the multimodal wheat disease and pest question-and-answer dataset.
[0007] In one possible implementation, the classification according to the subject entity category and the intent category includes: The obtained multimodal wheat disease and pest question-and-answer corpus was divided into ten entity categories according to the subject entity category: disease, pest, crop, pesticide, pathogen, insect stage / cycle, part, variety, symptoms, and others. The intention categories are divided into 22 types: harmful crops, aliases, pests, affected parts, distribution areas, crop varieties, periods of damage, control agents, family and genus, insect life stages / cycles, inclusions, control periods, symptoms, control methods, characteristics of damage, morphological features, life habits, brief introduction, disease patterns, diseases, pathogens / transmission vectors, and others.
[0008] In one possible implementation, the step of associating and integrating heterogeneous knowledge from different data sources to construct a multimodal wheat disease and pest knowledge graph includes: Using various types of structured, semi-structured, and unstructured data as data sources, after cleaning and noise reduction preprocessing, the structured data is stored in a MySQL database, while the semi-structured and unstructured data is stored in txt files. This technique extracts entities and their attribute relationships from unstructured and semi-structured data to form structured triples. Structured data can be directly converted into triples according to mapping rules. Semi-structured data is directly extracted from entities and their types by training the named entity recognition model BiLSTM-CRF and mapped into triples according to predefined rules. Unstructured data is extracted by training the entity relationship joint extraction model CalRel to simultaneously identify entities and their corresponding relationships to form triples. Image modal data is regarded as attribute entities and attribute triples are constructed with corresponding entities in the form of links. The extracted triples are then fused at the pattern layer; The Neo4j database is used to store the fused triples, and a multimodal wheat disease and pest knowledge graph is constructed. The multimodal wheat disease and pest knowledge graph can be queried and visualized using Cypher statements.
[0009] In one possible implementation, constructing a multimodal question-answering model for wheat diseases and pests, and training it using the multimodal wheat disease and pests question-answering dataset, includes: Multimodal feature extraction is performed to extract visual and linguistic features from the problem; The topic entity category is dynamically identified based on the context in order to further narrow the search space of the multimodal knowledge graph. A heterogeneous attention mechanism is adopted to fully integrate visual features with the context information of the question, and entity category classification is achieved based on an entity category classifier. Under entity category constraints, and combined with the constructed multimodal wheat disease and pest knowledge graph, thematic entities in the image are identified based on the question context; Intents are classified based on their probability distribution and quantity. The training was conducted using the constructed multimodal wheat disease and pest question-and-answer dataset.
[0010] In one possible implementation, the multimodal feature extraction, which extracts visual and linguistic features from the problem, includes: For image modal input Visual features are extracted using the Vision Transformer model. ; Text Modal Input Language features are generated using the pre-trained BERT model. , is defined as: In the formula, and These represent the methods used to extract visual features. and language features Network parameters, These are the parameters for the pre-trained model.
[0011] In one possible implementation, the dynamic identification of topic entity categories based on context, in order to further narrow the search space of the multimodal knowledge graph, employs a heterogeneous attention mechanism to fully integrate visual features with question context information, and implements entity category classification based on an entity category classifier, including: In the formula, These represent the query, key, and value after transformation by the fully connected layer, respectively. ,express The importance of each character is determined by its value; the higher the value, the greater the correlation between the current character and the visual features. This represents the language features after fusing visual information. Visual information is dynamically integrated through weighted summation. A residual mechanism is introduced to mitigate network degradation. Layer normalization (LN) is introduced to enhance network stability. A gating mechanism is also introduced to filter useful features and maintain the model's nonlinearity. In the formula, For trainable weights, For bias, select As an activation function ; This represents a gating factor between 0 and 1. The weight set that constitutes the weight set; This represents the element-wise multiplication of two vectors. The final learned text features are then used; an attention mechanism is then employed to... Dynamic aggregation is used to represent entity categories of fixed size. Finally, a fully connected network is used to classify entity categories: .
[0012] In one possible implementation, the step of identifying topic entities in an image based on the question context, under entity category constraints and in conjunction with a constructed multimodal wheat disease and pest knowledge graph, includes: The visual representation of entities in the knowledge graph is obtained using the noise-weighted averaging method: In the formula, The first entity represents the third The visual representation of each image; This indicates the number of images corresponding to an entity. The noise value is represented using Euclidean distance. The input image is represented using a Euclidean distance metric. The similarity between the image representations in the knowledge graph is used to determine the entity corresponding to the image representation with the highest similarity in the knowledge graph, which is the best matching topic entity.
[0013] In one possible implementation, the intent classification based on intent probability distribution and intent quantity includes: The intent context semantics are obtained by sequentially employing an attention mechanism and a fully connected network. and intention probability distribution ,in Total number of intent categories; Then, an intent counting network is built to obtain the number of intents in the question. ; Finally, from Select before The broad intent category serves as the final intent result: .
[0014] In one possible implementation, the training in conjunction with the constructed multimodal wheat disease and pest question-answering dataset includes: Cross-entropy loss is used for entity category extraction. Binary cross-entropy loss is used for subject entity recognition. Intent counting uses binary cross-entropy loss. Intent recognition uses cross-entropy loss. The objective function is obtained by training the dataset using the constructed multimodal wheat disease and pest question-and-answer dataset: .
[0015] In this embodiment, a multimodal feature extraction module extracts feature representations of different modalities. Based on this, a topic entity recognition branch is designed in conjunction with a multimodal knowledge graph. Entity categories are extracted to narrow the search space within the multimodal knowledge graph. Under entity category constraints, a visual similarity matching method based on noise values is used to achieve topic entity recognition. An intent recognition branch is also designed, employing an attention mechanism and a fully connected network to achieve intent classification, ultimately realizing multimodal question answering for wheat diseases and pests. This effectively solves the multimodal question answering problem for wheat diseases and pests in this complex scenario, providing more comprehensive, flexible, and accurate guidance for the diagnosis and control of wheat diseases and pests, thereby improving the effectiveness of wheat disease and pest control and reducing wheat yield losses caused by diseases and pests. Attached Figure Description
[0016] Figure 1 A flowchart illustrating a multimodal question-answering method for wheat diseases and pests that integrates a multimodal knowledge graph, provided as an embodiment of this application; Figure 2 This is a schematic diagram of knowledge graph construction provided in an embodiment of this application; Figure 3 This is a schematic diagram of the crop pest and disease entity provided in the embodiments of this application; Figure 4 This is a schematic diagram illustrating some examples of relational crop disease and pest data provided in the embodiments of this application; Figure 5 A schematic diagram illustrating a data layer example provided in this application embodiment; Figure 6 This is a partial visualization of the crop disease and pest knowledge graph provided in the embodiments of this application; Figure 7 This is a schematic diagram of a multimodal question-and-answer model for wheat diseases and pests provided in an embodiment of this application; Figure 8 This is a schematic diagram illustrating a multimodal question-and-answer example provided in an embodiment of this application. Detailed Implementation
[0017] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.
[0018] See Figure 1 The wheat disease and pest multimodal question answering method integrating multimodal knowledge graphs in this embodiment includes: S101, Obtain the multimodal wheat disease and pest question-and-answer dataset.
[0019] Data Acquisition and Preprocessing: The multimodal wheat disease and pest question-and-answer dataset comes from text modal and image modal data from agricultural input investment websites, China Agricultural Technology Extension Information Service Platform, China Crop Germplasm Information Network, and professional electronic books. After preprocessing such as cleaning, encoding, deduplication, and removal of irrelevant samples, the raw corpus is stored in a txt file.
[0020] Category Classification: This includes subject entity classification and intent classification. The obtained multimodal wheat disease and pest question-and-answer corpus is classified into ten entity categories: disease, pest, crop, pesticide, pathogen, insect stage / cycle, part, variety, symptoms, and others. Intents are classified into twenty-two intent types: affected crop, synonym, pest, affected part, distribution area, crop variety, period of damage, control pesticide, family and genus, insect stage / cycle, inclusion, control period, symptoms, control methods, damage characteristics, morphological characteristics, life habits, brief introduction, disease development pattern, disease, pathogen / vector, and others.
[0021] Topic entity and intent category annotation: The topic entities and entity categories corresponding to the image are annotated according to the question context, and the corresponding intent categories are annotated according to the question. This process is repeated for each original multimodal question answering sample until all samples are annotated. The annotated samples are then formatted and exported in JSON format to complete the construction of the multimodal wheat disease and pest question answering dataset, providing data support for the training of the S3 wheat disease and pest multimodal question answering model.
[0022] S102, link and integrate heterogeneous knowledge from different data sources to construct a multimodal wheat disease and pest knowledge graph.
[0023] Knowledge graphs are essentially semantic networks based on graph structures, capable of linking fragmented, heterogeneous knowledge from multiple sources, thereby supporting knowledge retrieval and question answering. Generally, knowledge graphs can be divided into a schema layer and a data layer. This embodiment will describe in detail the construction process of a crop disease and pest knowledge graph from aspects such as data sources, schema layer construction, data layer construction, and knowledge storage and fusion.
[0024] See Figure 2 The knowledge graph construction process in this embodiment is presented.
[0025] S1021 uses various structured, semi-structured, and unstructured data as data sources. After cleaning and noise reduction preprocessing, the structured data is stored in a MySQL database, while the semi-structured and unstructured data are stored in txt files.
[0026] Data is the foundation for knowledge graph construction. To enrich knowledge about crop diseases and pests as much as possible, this embodiment identifies multiple heterogeneous data sources based on existing work, as detailed in Table 1.
[0027] Table 1. Information on Crop Disease and Pest Data Sources As shown in Table 1, the data used in this embodiment comes from diverse sources. Based on different data types, the datasets can be divided into authoritative unstructured data, primarily in print, such as "Chinese Crop Diseases and Pests (Third Edition)", and semi-structured data, in web pages with certain arrangement rules, such as the Crop Diseases, Pests and Weeds Image and Text Database and the China Crop Germplasm Information Network. In addition, there are semi-structured and unstructured data obtained from literature resource databases such as CNKI and Wanfang Data, as well as portal websites such as Baidu Encyclopedia and the China Agricultural Technology Extension Information Platform. "Chinese Crop Diseases and Pests (Third Edition)" covers 1514 crop diseases and pests, involving knowledge in various fields such as damage, symptoms, pathogens, morphological characteristics, life habits, and control techniques, and is highly authoritative in the industry. CNKI and Wanfang Data are important domestic literature resource databases containing rich literature resources related to crop diseases and pests, providing strong data support for the construction of a crop disease and pest knowledge graph. The crop disease, pest, and weed database is categorized by plant type, covering 2,525 types of diseases and pests across 111 crops in four main categories: fruit trees, melons, vegetables, and agricultural crops. The China Crop Germplasm Information Network covers 1,619 types of crop diseases and pests, offering a rich variety of pests and diseases, all presented in web-based formats, with most data being semi-structured, making it easier to collect and utilize. In addition, this embodiment also supplements the database with data from other portals such as the Plant Protection Technology Information Network, the China Pesticide Information Network, and the Agricultural Inputs Investment Network.
[0028] According to their different functions, the above corpora can be divided into the following four categories: (1) Named entity recognition corpora. In this embodiment, literature abstracts crawled from CNKI and Wanfang Data Platform are selected as named entity recognition corpora. Their advantages are that the data volume is large, the quality is high, and they have a certain degree of professionalism and contain rich crop pest and disease entities. In addition, in order to improve the diversity of text expression, open domain data such as Baidu Encyclopedia are also incorporated; (2) Entity relation extraction corpora. In this embodiment, unstructured text in Baidu Encyclopedia and some literature abstracts are selected as the original corpora, and a crop pest and disease entity relation extraction dataset is constructed on this basis; (3) Crop pest and disease question and answer corpora. In this embodiment, the question and answer data of China Agricultural Technology Extension Information Platform are selected as the data source, and some question and answer data from Agricultural Input Investment Network and Plant Protection Technology Information Network are introduced to construct a crop pest and disease intent recognition and slot filling dataset. (4) Crop pest and disease knowledge graph construction corpora. In addition to the above-mentioned corpus, this embodiment uses "Chinese Crop Diseases and Pests (Third Edition)" as a reference, focusing on the image and text database of diseases, pests and weeds, and supplementing it with other corpora to construct a crop disease and pest knowledge graph.
[0029] The schema layer defines the constraints and specifications that the data layer should adhere to. It is a further abstraction and constraint of entities, attributes, and their relationships within the data layer, and is generally managed using ontology. An ontology is a conceptual template for a structured knowledge base. A knowledge base formed through an ontology not only has a strong hierarchical structure but also low redundancy. Therefore, the construction process of the schema layer is the same as the ontology construction process. Ontology construction can be done in two ways: top-down and bottom-up. Top-down construction refers to using a structured data source to first define a top-level relational ontology and then integrate entities into it. Bottom-up construction refers to directly extracting knowledge from the underlying data and integrating the extracted entities, relationships, or attributes with high confidence into the knowledge base. To ensure the completeness of the domain ontology and improve the quality and accuracy of knowledge, this embodiment uses a combination of top-up and bottom-down methods to construct a crop pest and disease domain ontology.
[0030] This embodiment primarily relies on the crop disease, pest, and weed image database and "Chinese Crop Diseases and Pests (Third Edition)". Based on the analysis of existing crop disease and pest knowledge graph construction corpora and related literature, and under the guidance of domain experts, a preliminary model of the crop disease and pest ontology was established using the Protégé ontology construction tool. This model is used to clarify the concepts and relationships within the crop disease and pest ontology, laying the foundation for the next step of constructing the crop disease and pest ontology. Figure 3 As shown, the crop disease and pest ontology constructed in this embodiment includes 10 relatively typical categories: disease and pest, crop, geographical location, variety, biological taxonomy, pesticide, microorganism, control method, part / organ, and journal. Among them, the disease and pest category is the core category of the crop disease and pest ontology construction, mainly including two subcategories: diseases and pests. The disease and pest category is associated with the crop category through the relationship of "harming crops", with the pesticide category through the relationship of "control pesticides", with the control method category through the attribute relationship of "control methods", with the journal category through the relationship of "research literature", with the biological taxonomy category through the attribute relationship of "belonging to", and with the geographical location category through the relationship of "distribution area". Its subcategory "diseases" is associated with the microorganism category through the relationship of "pathogens" and with the part / organ category through the relationship of "harming parts". "Pests" is associated with the part / organ category through the relationship of "containing" or "harming parts". The crop category is associated with the part / organ category through the relationship of "containing". In addition, variety and part / organ categories describing their own attributes are also added. There may be a "transmission vector" relationship between pests and diseases.
[0031] Furthermore, to further illustrate the attributes of each entity class and its relationships with other entities, as shown in Tables 2 and 3, this embodiment focuses on the pest and disease entity class as an example, listing the sets of relationships with other entity classes and its own attribute set. The domain and range are used to constrain the scope of the entity class. For example, for the "pesticide" relationship, the subject can only be pests and diseases, while the object is the pesticide.
[0032] Table 2. Information on the relationship between pests and diseases and other types of entities. Table 3. Information on Pest and Disease Attributes S1022 is a technique for extracting entities and their attribute relationships from unstructured and semi-structured data to form structured triples. Structured data can be directly converted into triples according to mapping rules. Semi-structured data is directly extracted from entities and their types by training the named entity recognition model BiLSTM-CRF and mapped into triples according to predefined rules. Unstructured data is extracted from entities and their corresponding relationships by training the entity relationship joint extraction model CalRel to simultaneously identify entities and their corresponding relationships to form triples. Image modal data is regarded as attribute entities and attribute triples are constructed with corresponding entities in the form of links.
[0033] The data layer stores concrete entities, attributes, and their relationships in units of facts, typically represented by triples. This embodiment extracts entities, attributes, and their relationships from structured, semi-structured, and unstructured crop disease and pest data based on the crop disease and pest ontology constructed from the schema layer, thereby constructing the crop disease and pest knowledge graph data layer. For semi-structured web page data such as those from CNKI, Wanfang Data, the Crop Disease, Pest and Weed Image Database, and the China Crop Germplasm Information Network, they are first converted into structured data using regular expressions and other operations and stored in a relational database as the structured data source for constructing the crop disease and pest knowledge graph.
[0034] (1) Structured knowledge extraction Structured data is stored in relational databases and has a fixed data format. Therefore, when extracting knowledge from such data, relevant fields can be directly mapped to entities and attributes by defining mapping rules, and field names become attribute names. This embodiment treats attributes as hasValue relationships between entities and attribute values, and then transforms entities, attributes, and their attribute relationships into (head entity, relation, tail entity) triples, which are then integrated into the knowledge graph under the guidance of the ontology conceptual model constructed at the schema layer. This embodiment uses, for example... Figure 4 Using the crop disease, pest, and weed damage image and text database shown as an example, the knowledge extraction process of structured data is described.
[0035] This embodiment employs mapping rules to map structured data of crop diseases and pests to triples. Then, based on the crop disease and pest ontology conceptual model constructed in the previous section, each record in the database is mapped to a corresponding instance, with the disease / pest name as the head entity, thus directly forming entity nodes and edge relationships in the knowledge graph. The mapping rules used in this embodiment are shown in Table 4. A similar method is used to map structured data from sources such as the China Crop Germplasm Information Network and journal articles to the knowledge graph data layer.
[0036] Table 4 Structured Data Mapping Rules (2) Unstructured knowledge extraction Unlike structured knowledge, unstructured knowledge is often hidden within sentences or paragraphs. For example, in Figure 3 The "Damage Symptoms (plus_dsi_damagesym)" field also hides entity information such as crop part, insect stage / cycle, and variety, as well as corresponding semantic relationships related to the pest and disease, such as the damaged part, the period of damage, and the crop variety. Similarly, the "Introduction (plus_dsi_introduction)" field hides attribute information such as pathogen and scientific name, while the "Control Methods (plus_dsi_cmethod)" field contains entity information such as pesticides and corresponding control pesticide relationships. Because this unstructured knowledge often lacks a specific structure, it cannot be extracted simply by designing rules; different knowledge extraction methods need to be designed based on the characteristics of different texts. The specific implementation in this example is as follows: Depend on Figure 4 It can be seen that the unstructured content shown in the "Symptoms of Harm" field only contains unstructured data of tail entities such as location and insect stage, while head entities such as "crop" and "pests and diseases" have been listed as separate fields, and the attribute relationship between head and tail entities is relatively fixed. In this embodiment, it is only necessary to use the named entity recognition model proposed in Chapters 2 and 3 to identify the tail entities in the unstructured text, and then concatenate them with the main entities such as "crop" and "pests and diseases" to construct the corresponding entity relationship triples. For complex unstructured texts such as Baidu Encyclopedia and journal abstracts, the entity relationship joint extraction model MER-CDP based on multi-round sequence labeling proposed in Chapter 4 is directly used to simultaneously identify the entities in the text and the semantic relationships between entities, thereby constructing entity relationship triples. Finally, Figure 5Taking "wheat scab" as an example, this example visually demonstrates the relationships between entities in the data layer and their inherent attributes. Thus, this embodiment has achieved knowledge extraction from structured data such as crop disease and pest image databases, the China Crop Germplasm Information Network, and journal data by defining mapping rules. It has also achieved knowledge extraction from unstructured data such as Baidu Encyclopedia and journal abstracts through named entity recognition and joint entity relation extraction, forming triples that lay the foundation for subsequent knowledge storage and fusion.
[0037] S1023, the extracted triples are fused at the pattern layer; This embodiment uses the Neo4j database to store the aforementioned triples. Neo4j is a high-performance NoSQL database that employs an unstructured data storage model, offering great flexibility and fast response times. Furthermore, to strengthen the connection between crop pest and disease-related knowledge and journal literature knowledge, this embodiment establishes a "research literature" relationship by determining whether the journal keywords contain entity names related to pests and diseases. This achieves the association between pest and disease instances and journal instances, adding 19,277 association relationships. The Cypher operation statements for knowledge fusion are shown below: MATCH(n:`Disease`),(j:`Title`)-[r:`Keyword`]->(k:`Keyword`)where n.name insplit(k.name,';')CREATE(n)-[r2:Research Literature]->(j); MATCH(n:`Pests`),(j:`Title`)-[r:`Keywords`]->(k:`Keywords`)where n.name insplit(k.name,';')CREATE(n)-[r2:Research Literature]->(j); S1024 uses the Neo4j database to store the fused triples, completes the construction of a multimodal wheat disease and pest knowledge graph, and realizes the query and visualization of the multimodal wheat disease and pest knowledge graph through Cypher statements.
[0038] Ultimately, this embodiment constructed a crop disease and pest knowledge graph database containing 221,026 entities and 252,749 relationships. Figure 6 In the text, a partial visualization of the crop disease and pest knowledge graph is used to more intuitively demonstrate the effect of the constructed crop disease and pest knowledge graph.
[0039] S103, Construct a multimodal question-and-answer model for wheat diseases and pests, and train it using the multimodal wheat disease and pest question-and-answer dataset.
[0040] See Figure 7This is a multimodal question-answering model for wheat diseases and pests, which mainly includes multimodal feature extraction, entity category extraction, topic entity recognition, intent classification, and model training.
[0041] Multimodal feature extraction: The multimodal feature extraction is mainly used to extract visual and linguistic features from the problem, and to extract image modal input. Visual features are extracted using the Vision Transformer model. Text modal input Language features are generated using the pre-trained BERT model. , is defined as: (1) (2) In the formula, and These represent the methods used to extract visual features. and language features Network parameters, These are the parameters for the pre-trained model.
[0042] Entity category extraction: The entity category extraction aims to dynamically identify the category of topic entities based on context, in order to further narrow the search space of the multimodal knowledge graph. It employs a heterogeneous attention mechanism to fully integrate visual features with the contextual information of the question, and implements entity category classification based on an entity category classifier, defined as follows: (3) (4) (5) (6) In the formula, These represent the query, key, and value after transformation by the fully connected layer, respectively. ,express The importance of each character is determined by its value; the higher the value, the greater the correlation between the current character and the visual features. This represents the language features after fusing visual information. Visual information is dynamically integrated through weighted summation. A residual mechanism is introduced to mitigate network degradation, and a layer normalization function (LN) is used to enhance network stability. Furthermore, a gating mechanism is introduced to filter useful features and maintain the model's nonlinearity. In the formula, For trainable weights, For bias, select As an activation function ; This represents a gating factor between 0 and 1. The weight set that constitutes the weight set; This represents the element-wise multiplication of two vectors. The final learned text features are then used; an attention mechanism is then employed to... Dynamic aggregation is used to represent entity categories of fixed size. Finally, a fully connected network is used to classify entity categories: (7) Topic Entity Recognition: This topic entity recognition is primarily used to identify topic entities in an image based on the question context, under entity category constraints and in conjunction with a constructed multimodal wheat pest and disease knowledge graph. The goal of topic entity recognition is to identify the corresponding topic entities from a given multimodal question. The challenge lies in the fact that different question contexts result in different predicted topic entities. Figure 8 This embodiment will focus on two aspects: entity category recognition and topic entity recognition.
[0043] Entity category recognition aims to dynamically identify entity categories based on context, thereby further narrowing the search space of multimodal knowledge graphs. It proposes to leverage a heterogeneous attention mechanism to fully integrate visual features with question context information, and then implement entity category classification based on an entity category classifier.
[0044] Topic entity recognition aims to identify corresponding text entities from a multimodal knowledge graph. Based on narrowing the search scope by entity category, a visual similarity matching method based on noise values is proposed for entity matching.
[0045] To avoid visual omissions, a noise-weighted average method is used to obtain visual representations of entities in the knowledge graph: (8) In the formula, The first entity represents the third The visual representation of each image; This indicates the number of images corresponding to an entity. The noise value is represented using Euclidean distance. (9) The input image is represented using a Euclidean distance metric. The similarity between the image representations in the knowledge graph is used to determine the entity corresponding to the image representation with the highest similarity in the knowledge graph, which is the best matching topic entity.
[0046] Intent classification: The intent context semantics are obtained by sequentially employing an attention mechanism and a fully connected network. and intention probability distribution ,in To represent the total number of intent categories, an intent counting network is then constructed to obtain the number of intents in the question. (Equation 10), finally, from Select before The broad intent category serves as the final intent outcome. .
[0047] (10) Model training: Entity category extraction using cross-entropy loss. Subject entity recognition uses binary cross-entropy loss. Intent counting in intent classification uses binary cross-entropy loss. Intent recognition uses cross-entropy loss. The objective function is defined as follows: The function is trained using the constructed multimodal wheat disease and pest question-and-answer dataset. (11).
[0048] S104. A wheat disease and pest diagnosis and treatment system is constructed by combining a modal wheat disease and pest knowledge graph with a trained multimodal wheat disease and pest question-answering model.
[0049] The wheat disease and pest diagnosis and treatment system in this embodiment performs question parsing on multimodal questions containing questions and images to obtain topic entities and intents, and then performs answer search, using the searched answer as the final result.
[0050] With the support of the constructed multimodal wheat disease and pest knowledge graph, the trained multimodal wheat disease and pest question answering model simultaneously identifies the subject entity and intent in the question, forming a query triple <subject entity, intent, ?>, where ? represents the candidate answer.
[0051] The triples to be searched are converted into Cypher statements, and the corresponding candidate entities are searched in the constructed multimodal wheat disease and pest knowledge graph. After conversion, the final answer is fed back to the user requesting end for reference.
[0052] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0053] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.
Claims
1. A multimodal question-answering method for wheat diseases and pests integrating multimodal knowledge graphs, characterized in that, include: Obtain a multimodal wheat disease and pest question-and-answer dataset; By linking and integrating heterogeneous knowledge from different data sources, a multimodal wheat disease and pest knowledge graph can be constructed. A multimodal question-and-answer model for wheat diseases and pests was constructed and trained using the multimodal wheat disease and pest question-and-answer dataset. A wheat disease and pest diagnosis and treatment system was constructed by combining a modal wheat disease and pest knowledge graph with a trained multimodal wheat disease and pest question-answering model.
2. The wheat disease and pest multimodal question-answering method integrating multimodal knowledge graphs according to claim 1, characterized in that, The acquisition of the multimodal wheat disease and pest question-and-answer dataset includes: The acquired text modality and image modality data are preprocessed by cleaning, encoding, deduplication, and removal of irrelevant samples to obtain the original corpus, which is then stored in a txt file. Categorize according to subject entity type and intent type; The image is labeled with the subject entity and entity category corresponding to the question context, and the corresponding intent category is labeled according to the question. This process is repeated for each original multimodal question-and-answer sample until all samples are labeled. The labeled samples are then formatted and exported in JSON format to complete the construction of the multimodal wheat disease and pest question-and-answer dataset.
3. The multimodal question-answering method for wheat diseases and pests fused with multimodal knowledge graphs according to claim 2, characterized in that, The classification based on subject entity category and intent category includes: The obtained multimodal wheat disease and pest question-and-answer corpus was divided into ten entity categories according to the subject entity category: disease, pest, crop, pesticide, pathogen, insect stage / cycle, part, variety, symptoms, and others. The intention categories are divided into 22 types: harmful crops, aliases, pests, affected parts, distribution areas, crop varieties, periods of damage, control agents, family and genus, insect life stages / cycles, inclusions, control periods, symptoms, control methods, characteristics of damage, morphological features, life habits, brief introduction, disease patterns, diseases, pathogens / transmission vectors, and others.
4. The multimodal question-answering method for wheat diseases and pests fused with multimodal knowledge graphs according to claim 1, characterized in that, The process of associating and integrating heterogeneous knowledge from different data sources to construct a multimodal wheat disease and pest knowledge graph includes: Using various types of structured, semi-structured, and unstructured data as data sources, after cleaning and noise reduction preprocessing, the structured data is stored in a MySQL database, while the semi-structured and unstructured data is stored in txt files. This technique extracts entities and their attribute relationships from unstructured and semi-structured data to form structured triples. Structured data can be directly converted into triples according to mapping rules. Semi-structured data is directly extracted from entities and their types by training the named entity recognition model BiLSTM-CRF and mapped into triples according to predefined rules. Unstructured data is extracted by training the entity relationship joint extraction model CalRel to simultaneously identify entities and their corresponding relationships to form triples. Image modal data is regarded as attribute entities and attribute triples are constructed with corresponding entities in the form of links. The extracted triples are then fused at the pattern layer; The Neo4j database is used to store the fused triples, and a multimodal wheat disease and pest knowledge graph is constructed. The multimodal wheat disease and pest knowledge graph can be queried and visualized using Cypher statements.
5. The multimodal question-answering method for wheat diseases and pests fused with multimodal knowledge graphs according to claim 1, characterized in that, The construction of a multimodal question-answering model for wheat diseases and pests, and its training using the multimodal wheat disease and pest question-answering dataset, includes: Multimodal feature extraction is performed to extract visual and linguistic features from the problem; The topic entity category is dynamically identified based on the context in order to further narrow the search space of the multimodal knowledge graph. A heterogeneous attention mechanism is adopted to fully integrate visual features with the context information of the question, and entity category classification is achieved based on an entity category classifier. Under entity category constraints, and combined with the constructed multimodal wheat disease and pest knowledge graph, thematic entities in the image are identified based on the question context; Intents are classified based on their probability distribution and quantity. The training was conducted using the constructed multimodal wheat disease and pest question-and-answer dataset.
6. The multimodal question-answering method for wheat diseases and pests fused with multimodal knowledge graphs according to claim 5, characterized in that, The multimodal feature extraction process extracts visual and linguistic features from the problem, including: For image modal input Visual features are extracted using the Vision Transformer model. ; Text Modal Input Language features are generated using the pre-trained BERT model. , is defined as: In the formula, and These represent the methods used to extract visual features. and language features Network parameters, These are the parameters for the pre-trained model.
7. The multimodal question-answering method for wheat diseases and pests fused with multimodal knowledge graphs according to claim 6, characterized in that, The process of dynamically identifying topic entity categories based on context, thereby further narrowing the search space of the multimodal knowledge graph, employs a heterogeneous attention mechanism to fully integrate visual features with question context information, and implements entity category classification based on an entity category classifier, including: In the formula, These represent the query, key, and value after transformation by the fully connected layer, respectively. ,express The importance of each character is determined by its value; the higher the value, the greater the correlation between the current character and the visual features. This represents the language features after fusing visual information. Visual information is dynamically integrated through weighted summation. A residual mechanism is introduced to mitigate network degradation. Layer normalization (LN) is introduced to enhance network stability. A gating mechanism is also introduced to filter useful features and maintain the model's nonlinearity. In the formula, For trainable weights, For bias, select As an activation function ; This represents a gating factor between 0 and 1. The weight set that constitutes the weight set; This represents the element-wise multiplication of two vectors. The final learned text features are then used; an attention mechanism is then employed to... Dynamic aggregation is used to represent entity categories of fixed size. Finally, a fully connected network is used to classify entity categories: 。 8. The multimodal question-answering method for wheat diseases and pests fused with a multimodal knowledge graph according to claim 7, characterized in that, Under entity category constraints, and in conjunction with the constructed multimodal wheat disease and pest knowledge graph, the process of identifying topic entities in an image based on the question context includes: The visual representation of entities in the knowledge graph is obtained using the noise-weighted averaging method: In the formula, The first entity represents the third The visual representation of each image; This indicates the number of images corresponding to an entity. The noise value is represented using Euclidean distance. The input image is represented using a Euclidean distance metric. The similarity between the image representations in the knowledge graph is used to determine the entity corresponding to the image representation with the highest similarity in the knowledge graph, which is the best matching topic entity.
9. The multimodal question-answering method for wheat diseases and pests fused with a multimodal knowledge graph according to claim 8, characterized in that, The intent classification based on intent probability distribution and intent quantity includes: The intent context semantics are obtained by sequentially employing an attention mechanism and a fully connected network. and intention probability distribution ,in Total number of intent categories; Then, an intent counting network is built to obtain the number of intents in the question. ; Finally, from Select before The broad intent category serves as the final intent result: 。 10. The multimodal question-answering method for wheat diseases and pests fused with multimodal knowledge graphs according to claim 9, characterized in that, The training process, which combines the constructed multimodal wheat disease and pest question-answering dataset, includes: Cross-entropy loss is used for entity category extraction. Binary cross-entropy loss is used for subject entity recognition. Intent counting uses binary cross-entropy loss. Intent recognition uses cross-entropy loss. The objective function is obtained by training the dataset using the constructed multimodal wheat disease and pest question-and-answer dataset: 。