Grape disease and insect pest automatic question-answering system based on multi-modal knowledge graph

Abstract

The invention belongs to the technical field of knowledge maps, and discloses a grape disease and insect pest automatic question-answering system based on a multi-modal knowledge map. Comprising a data collecting and processing module, a knowledge graph construction module, a main control module, a named entity recognition module, a text classification model module, a multi-modal fusion module, a knowledge graph storage module, an entity matching module and a query evaluation module. Constructing a multi-modal knowledge graph in the field of grape diseases and insect pests; the system analyzes characteristics of grape disease and insect pest data, studies a structured expression and integration method of grape disease and insect pest knowledge, constructs a grape disease and insect pest multi-mode knowledge graph ontology concept framework, and provides support for unified management and efficient utilization of related knowledge; the named entity recognition (NER) is carried out by utilizing a global normalization thought aiming at the condition that a hierarchical or nested relationship exists between entities in a knowledge graph construction process. An entity linking technology is adopted for data of different modes, and a grape pest and disease damage multi-mode knowledge graph is constructed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of knowledge graph, and particularly relates to a grape disease and pest automatic question answering system based on a multi-modal knowledge graph. BACKGROUND

[0002] The planting scale of grapes in China continues to expand, and the types of diseases and pests have also increased. Traditional prevention methods that rely on paper materials and experience are difficult to meet the needs of precision and timeliness. Although the Internet provides a large amount of information, the efficiency and accuracy of farmers in the search and screening process are low. Various prevention knowledge is still scattered in non-structured carriers such as graph courseware and manuals, and there is a lack of unified ontology modeling and visualization platforms, which limits knowledge reuse and intelligent service level.

[0003] Since knowledge graph was systematically proposed in 2012, it has provided a technical basis for structuring and integrating fragmented information in the form of <entity-relation-entity> triples, and has been verified in medical, financial, and transportation fields. Agriculture has a more significant data fragmentation and an urgent need for knowledge graph. Existing research has attempted to explore tea, soil, fungi, and poultry disease sub-directions, and there have been preliminary explorations of entity graphs and intelligent question answering in the field of grape diseases and pests, but there is still a lack of systematic integration of multi-modal data such as images, texts, and videos.

[0004] Multi-modal knowledge graph (MMKG) enables knowledge representation closer to real-world scenarios by incorporating visual, audio, and textual multi-source heterogeneous information. Related work has been carried out on general datasets and industrial and transportation applications, and has promoted the development of multi-modal entity alignment and completion algorithms. However, for the professional field of grape diseases and pests, multi-modal data has not been fully integrated and utilized, limiting the reasoning depth and service experience of knowledge-driven models.

[0005] Knowledge graph-based question answering systems can parse natural language into graph queries to generate accurate answers, but general large models may distort information, consume high resources, and pose data security risks in vertical scenarios. Existing grape disease and pest question answering platforms are mostly limited to text level and lack support from modalities such as vision. To improve professionalism and practicality, it is urgent to build a multi-modal knowledge graph for grape diseases and pests and develop a lightweight, safe, and controllable intelligent question answering system based on it, realizing the leap from "single-modal retrieval" to "multi-modal reasoning". SUMMARY

[0006] In view of the problems in the prior art, the application provides a grape disease and pest automatic question answering system based on a multi-modal knowledge graph.

[0007] The application is implemented as follows: a grape disease and pest automatic question answering system based on a multi-modal knowledge graph comprises:

[0008] the data collection processing module, the knowledge graph construction module, the main control module, the named entity recognition module, the text classification model module, the multi-modal fusion module, the knowledge graph storage module, the entity matching module, and the query evaluation module are connected with the main control module, and are used for controlling normal work of each module.

[0009] The data collection processing module is connected with the main control module, and is used for using HttpClient to request external website articles for data obtained from a webpage, using Hutool's JSONUtil to parse and preprocess the articles, and using jsoup to request related interfaces in real time to obtain pictures.

[0010] The knowledge graph construction module is connected with the main control module, and is used for extracting required information from data by analyzing data of different structures, and storing the information into a knowledge base.

[0011] The main control module is connected with the data collection processing module, the knowledge graph construction module, the named entity recognition module, the text classification model module, the multi-modal fusion module, the knowledge graph storage module, the entity matching module, and the query evaluation module, and is used for controlling normal work of each module.

[0012] The named entity recognition module is connected with the main control module, and is used for detecting and classifying various proper nouns in a text through automatic means.

[0013] The text classification model module is connected with the main control module, and is used for classifying large-scale text data through deep learning.

[0014] The multi-modal fusion module is connected with the main control module, and is used for mixing data of different modalities in a feature space to obtain fused features.

[0015] The knowledge graph storage module is connected with the main control module, and is used for storing recognized entities in a file, using entities as keywords, using the foregoing technology to obtain and preprocess images, and using a label corresponding to each image as an entity name corresponding to the image; storing images that have been filtered, standardized, and preprocessed in a server, and using a storage location of the images in the server as an image entity in a knowledge graph.

[0016] The entity matching module is connected with the main control module, and is used for a multi-modal fusion-based topic entity matching technology; the topic entity matching is divided into three stages of entity recognition, entity sorting, and entity matching, and multi-modal feature matching is used to match a topic entity in the entity matching.

[0017] The query evaluation module is connected with the main control module, and is used for query and evaluation of answers.

[0018] Further, the knowledge graph construction module comprises:

[0019] The construction of the knowledge graph is a process of analyzing different structured data, extracting the required information from the data, and then storing it in the knowledge base.

[0020] The knowledge graph has two construction methods: top-down and bottom-up. The appropriate method can be selected according to the data situation in different fields. If the knowledge system is complete, the top-down method can be used to construct the graph, and the data layer can be filled according to the mode concept. When the data is not systematic, the bottom-up method is needed to abstract the field concept from the data, and the concept mode is supplemented.

[0021] The construction of the field knowledge graph first needs a large amount of field-related knowledge data. These knowledge has multiple sources, including books, Internet websites, and relational databases. According to the structure, the data can be divided into three types: structured data such as relational data in relational databases, semi-structured data including Baidu Encyclopedia data, and unstructured data represented by text data. Extracting knowledge from different structured data is an important task in constructing a knowledge graph. Knowledge extraction is a process of obtaining useful knowledge from data through technical means, including entity, relationship, and event knowledge extraction.

[0022] Another object of the present application is to provide a grape disease and pest automatic question answering method based on a multi-modal knowledge graph, comprising:

[0023] Step 1: The data obtained by the data collection and processing module from the webpage uses HttpClient to request external website articles, uses Hutool's JSONUtil to parse and preprocess the articles, and uses jsoup to request related interfaces to obtain pictures in real time;

[0024] Step 2: The knowledge graph construction module analyzes different structured data, extracts the required information from the data, and then stores it in the knowledge base. The main control module uses automated means to detect and classify various proper nouns in the text through the named entity recognition module;

[0025] Step 3: The text classification model module uses deep learning to classify large-scale text data. The multi-modal fusion module mixes different modal data in the feature space to obtain fused features.

[0026] Step 4: The knowledge graph storage module stores the recognized entities in the file, uses the entities as keywords, and uses the previous technology to obtain and preprocess the images. The label corresponding to each image is the entity name corresponding to the image. The preprocessed images are stored in the server, and the storage location of the images in the server is used as the image entity in the knowledge graph.

[0027] Step 5, based on the multi-modal fusion of the subject entity matching technology through the entity matching module; the subject entity matching is divided into three stages of entity recognition, entity sorting and entity matching, and the subject entity is matched through the multi-modal feature matching of the entity matching; the answer query and evaluation are evaluated through the query evaluation module.

[0028] Another object of the present application is to provide a computer device comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to make the processor execute the steps of the grape disease and pest automatic question and answer method based on the multi-modal knowledge graph.

[0029] Another object of the present application is to provide a computer readable storage medium storing a computer program, the computer program being executed by a processor to make the processor execute the steps of the grape disease and pest automatic question and answer method based on the multi-modal knowledge graph.

[0030] Another object of the present application is to provide an information data processing terminal for realizing the grape disease and pest automatic question and answer system based on the multi-modal knowledge graph.

[0031] In combination with the above technical solutions and the technical problems solved, the technical solution to be protected by the present application has the following advantages and positive effects:

[0032] The present application integrates expert experience and multi-source data to construct a grape disease and pest multi-modal knowledge graph, uses a BERT-GlobalPointer model to improve the performance of nested entity extraction, and fills the blank of multi-modal graph in this field. The question and answer system innovatively integrates text and image modal information to enhance the accuracy of subject entity matching. The ERNIE-Gram model is used to optimize the intent recognition of the question, a multi-modal entity matching method is designed, and the graph information is combined to deal with complex problem scenarios, helping farmers to quickly obtain professional knowledge.

[0033] The multi-modal knowledge graph (MMKG) provides a new solution for the agricultural VQA task, integrates image, text and structured data, and improves the efficiency of agricultural knowledge service. For the pain point of grape disease and pest knowledge acquisition, the present application constructs a professional graph system and proposes a multi-modal question and answer process: through the steps of entity recognition, intent classification, relationship chain construction and multi-modal matching, the answer is accurately generated. The system breaks through the traditional single information mode, integrates visual and text data to solve the term similarity ambiguity, and promotes the innovative development of intelligent agriculture.

[0034] The present application provides an intelligent solution for grape disease and pest control:

[0035] (1) Multi-modal knowledge graph construction: To solve the problem of scattered grape disease and pest data, an ontology framework is established to realize the structured integration of knowledge. Global normalization method is used to process entity level relationship, and entity linking technology is used to associate multi-modal data.

[0036] (2) Multi-modal question answering method: Break through the limitation of traditional text question answering, propose multi-modal fusion entity matching method. Use ERNIE-Gram to enhance intent recognition, BERT-CNN-Siamese to optimize relationship linking, and combine entity constraint to improve answer accuracy.

[0037] (3) Question answering system development: Based on the above method, an actual application system is constructed, which provides precise disease and pest control question answering service for growers through demand analysis, data management, system testing and other links.

[0038] To solve the problem of scattered knowledge and missing multi-modal association, the invention improves the reusability of agricultural knowledge and service efficiency through knowledge graph construction and intelligent question answering. Specifically, it includes three directions: multi-modal knowledge graph construction, question answering technology path exploration, and question answering system development.

[0039] The implementation process includes: crawler obtains multi-modal data, constructs ontology concept layer; use BERT-GlobalPointer to extract information, cosine similarity to associate image and text entities; LOADCSV imports Neo4j to realize storage; through multi-modal data enhancement entity matching, ERNIE-Gram optimizes intent classification, BERT-CNN-Siamese completes relationship chain analysis, and finally the Cypher query result is accurately fed back to the user.

[0040] The invention integrates grape disease and pest data using multi-modal knowledge graph technology, designs a concept mode containing 13 types of entities and 12 types of relationships. Through the BERT-GlobalPointer model (F196.11%), entity recognition and relationship judgment are completed, image entities and text entities are linked through "HasImage" relationship, and finally stored in Neo4j database.

[0041] To solve the problem of inaccurate farmer description, a multi-modal question answering method is proposed: through BERT-GlobalPointer to recognize entity (F196.41%), ERNIE-Gram to classify intent (F198.14%), BERT-CNN-Siamese to link relationship chain (F195.12%), and convert to Cypher statement query. The experimental results show that the question answering accuracy is 88.74%-90.68%, verifying the compensation effect of multi-modal information.

[0042] An interactive platform is built based on a Django framework, and a Neo4j database and a BERT-GlobalPointer entity recognition model are integrated. The question and answer process includes topic identification, intent classification, and relationship chain generation, knowledge retrieval and visualization are realized through Cypher query, and a smart solution is provided for grape disease control.

[0043] In view of the problems of the increasing types of grape industry diseases and insect pests and the scattered knowledge, the first open-source multi-modal knowledge graph and question and answer system are constructed, the problem of feature association mining of unstructured data is broken through, and the efficiency of agricultural knowledge service is improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a structure block diagram of a grape disease and insect pest automatic question and answer system based on a multi-modal knowledge graph provided by the embodiment of the application.

[0045] Figure 2 It is a method flow chart of a grape disease and insect pest automatic question and answer method based on a multi-modal knowledge graph provided by the embodiment of the application.

[0046] Figure 3 It is a hidden Markov model graph provided by the embodiment of the application.

[0047] Figure 4 It is an LSTM model graph provided by the embodiment of the application.

[0048] Figure 5 It is a BERT model structure graph provided by the embodiment of the application.

[0049] Figure 6 It is a FastText model structure graph provided by the embodiment of the application.

[0050] Figure 7 It is a multi-modal knowledge graph construction flow chart of grape diseases and insect pests provided by the embodiment of the application.

[0051] Figure 8 It is an ontology concept hierarchical division graph of pure text data provided by the embodiment of the application.

[0052] Figure 9 It is a multi-modal entity structure graph provided by the embodiment of the application.

[0053] Figure 10 It is a text coding graph provided by the embodiment of the application.

[0054] Figure 11 It is a multi-modal knowledge graph (part) graph provided by the embodiment of the application.

[0055] Figure 12 It is a question and answer method flow chart provided by the embodiment of the application.

[0056] Figure 13 is a subject entity matching method flowchart provided by an embodiment of the present application.

[0057] Figure 14 is a subject entity matching model based on multi-modal fusion provided by an embodiment of the present application.

[0058] Figure 15 is a VMamba model overall architecture diagram provided by an embodiment of the present application.

[0059] Figure 16 is a VSSBlock structure diagram provided by an embodiment of the present application.

[0060] Figure 17 is a data set picture display diagram provided by an embodiment of the present application.

[0061] Figure 18 is a relationship chain linking model diagram provided by an embodiment of the present application.

[0062] Figure 19 is an added entity constraint diagram provided by an embodiment of the present application.

[0063] Figure 20 is a query flowchart provided by an embodiment of the present application.

[0064] Figure 21 is a flowchart of adding multi-modal entity matching provided by an embodiment of the present application.

[0065] Figure 22 is a core module diagram of the grape disease and pest automatic question answering system provided by an embodiment of the present application.

[0066] Figure 23 is a system architecture design diagram provided by an embodiment of the present application.

[0067] Figure 1 In the figure: 1, data collection and processing module; 2, knowledge graph construction module; 3, main control module; 4, named entity recognition module; 5, text classification model module; 6, multi-modal fusion module; 7, knowledge graph storage module; 8, entity matching module; 9, query evaluation module. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0069] As shown in Figure 1 , the grape disease and pest automatic question answering system based on multi-modal knowledge graph provided by an embodiment of the present application comprises:

[0070] Data collection and processing module 1, knowledge graph construction module 2, main control module 3, named entity recognition module 4, text classification model module 5, multimodal fusion module 6, knowledge graph storage module 7, entity matching module 8, query evaluation module 9.

[0071] The data collection and processing module 1 is connected to the main control module 3 and is used to obtain data from the web page by using HttpClient to request articles from external websites, using Hutool's JSONUtil to parse and preprocess articles, and using jsoup to request relevant interfaces in real time to obtain images;

[0072] The knowledge graph construction module 2 is connected to the main control module 3 and is used to extract the required information from the data by analyzing the data with different structures and then store it in the knowledge base;

[0073] The main control module 3 is connected to the data collection and processing module 1, the knowledge graph construction module 2, the named entity recognition module 4, the text classification model module 5, the multimodal fusion module 6, the knowledge graph storage module 7, the entity matching module 8, and the query evaluation module 9 to control the normal operation of each module;

[0074] Named Entity Recognition Module 4, connected to the main control module 3, is used to detect and classify various proper nouns in the text through automated means;

[0075] The text classification model module 5 is connected to the main control module 3 and is used to classify large-scale text data through deep learning;

[0076] The multimodal fusion module 6 is connected to the main control module 3 and is used to mix data of different modalities in the feature space to obtain fusion features;

[0077] The knowledge graph storage module 7 is connected to the main control module 3 and is used to store the identified entities in a file. The entity is used as a keyword and the above-mentioned technology is used to obtain and pre-process the image. The label corresponding to each image is the entity name corresponding to the image; the pre-processed images such as screening and standardization are stored in the server, and the storage location of the image in the server is used as the image entity in the knowledge graph;

[0078] The entity matching module 8 is connected to the main control module 3 and is used for subject entity matching technology based on multimodal fusion; the subject entity matching is divided into three stages: entity recognition, entity sorting and entity matching, and the subject entity is matched by fusing multimodal features in the entity matching;

[0079] The query and evaluation module 9 is connected to the main control module 3 and is used for querying and evaluating answers.

[0080] The knowledge graph construction module provided by the embodiment of the present invention:

[0081] The construction of the knowledge graph is a process of analyzing different structures of data, extracting the required information from the data, and then storing it in the knowledge base;

[0082] The knowledge graph has two construction methods of top-down and bottom-up; the suitable method can be selected according to the data situation of different fields; if the knowledge system is complete, the top-down method can be used to construct the graph, and the data layer can be filled according to the mode concept; when the data is not systematic, the concept of the field needs to be abstracted from the data through the bottom-up method, and the concept mode is supplemented;

[0083] The construction of the field knowledge graph first needs a large amount of field-related knowledge data; these knowledge has multiple sources, including books, Internet websites, and relational databases; according to the structure, the data can be divided into three types: structured data such as relational data in relational databases, semi-structured data including Baidu Encyclopedia data, and unstructured data represented by text data; extracting knowledge from data of different structures is a very important work for constructing a knowledge graph; knowledge extraction is a process of obtaining useful knowledge from data through technical means, including entity, relationship and event knowledge extraction.

[0084] The named entity recognition module provided by the embodiment of the present application comprises:

[0085] (1) a rule and dictionary-based method;

[0086] By using the artificially designed rule template and the pre-constructed field dictionary, the entity is located and classified from the unstructured text;

[0087] (2) machine learning-based entity recognition;

[0088] (3) deep learning-based entity recognition.

[0089] The machine learning-based entity recognition provided by the embodiment of the present application comprises:

[0090] (1) maximum entropy model

[0091] Estimate the probability distribution that best fits the observed data under given constraints;

[0092] For a classification task, we want to estimate a class distribution P(y|x), where x is the input feature, and y is the target class, and the goal is to find a probability distribution P(y|x) that can best describe the data while satisfying the maximum entropy principle under the constraints; as shown in formula (1);

[0093] H(Y|X)=-∑ x,y P(x)P(y|x)logP(y|x) (1)

[0094] (2) Hidden Markov Model

[0095] Modeling stochastic processes with hidden states, particularly suitable for sequential data; the structure of Hidden Markov Model (HMM) can be represented by a state transition graph;

[0096] Hidden state x i is a state that cannot be directly observed, for example, in the part-of-speech tagging task, these may be "noun, verb, adjective"; y i is an observable variable, such as a word, a phoneme, a DNA fragment; P(x i-1 |x i ) is the state transition probability, representing the probability of x i-1 transitioning to x i ; P(y i |x i ) is the observation probability, representing the probability of y i transitioning to x i ;

[0097] The core idea of using Hidden Markov Model (HMM) for entity recognition is to regard the text sequence as an observation sequence, and regard the entity class of each word as a hidden state, and then use state transition probability and observation probability for labeling, where the hidden state set is defined as entity class (such as PER representing personal name, LOC representing place name, ORG representing organization name, and O representing non-entity), the observation sequence is the input text word sequence, the state transition probability matrix (A) of the model is used to represent the transition probability between different entity classes, for example, "after the name, it is more likely to be followed by the surname or the position, rather than the place name", the observation probability matrix (B) is used to represent the probability of a certain word appearing under a specific entity class, for example, "apple" is more likely to correspond to ORG (Apple Inc.) than PER (personal name), and the initial state distribution (π) represents the possibility of each entity class at the beginning of the sentence, although HMM performs well in sequence labeling tasks, the Markov assumption limits the modeling ability of long-distance dependencies;

[0098] (3) Conditional Random Field

[0099] Conditional Random Field is a kind of probabilistic graphical model, which is used to model sequence labeling problems, and this algorithm combines the core ideas of Markov Random Field theoretical framework and maximum entropy modeling method, and solves the limitations of traditional models (such as Hidden Markov Model) in sequence labeling through global normalization;

[0100] CRF calculates the optimal solution of sequence labeling by defining a global probability distribution P(Y|X), and the formula of P(Y|X) is shown in equation (2);

[0101]

[0102] The embodiment of the present application provides the entity recognition based on deep learning:

[0103] (1) LSTM is as follows Figure 4

[0104] In the LSTM architecture, the processing of sequence information is realized by setting three gating mechanisms and a cell state unit, wherein the cell state is mainly responsible for saving long-term dependence features, and the gating system is used for screening key data in the cell state; the forgetting gate, the input gate and the output gate constitute three core components of the LSTM; for an input sequence, assuming that the input sequence of the LSTM network contains n word units, each word unit is represented by a d-dimensional feature vector, and thus the input tensor constructed has a size of nxd; the calculation process of the model can be divided into three main stages: firstly, the forgetting gate evaluates the retention value of information in the cell unit; secondly, the input gate determines new information to be stored in the cell unit; finally, the output gate screens effective data of the final output of the model; the structure design and operation method of each gating unit will be specifically described below;

[0105] ① The forgetting gate decides how much past information the cell state (CellState) of the current time step should forget; the forgetting gate is calculated as follows;

[0106] f t =σ(W f ·[h t-1 ,x t ]+b f ) (3)

[0107] f t is the output of the forgetting gate, and the value range is [0, 1], which represents the forgetting proportion; W f and b f are learnable weights and bias; h t-1 is the hidden state of the last time; x t is the current input; sigma is a sigmoid activation function, which ensures that the output is between [0, 1]; the forgetting gate can make the LSTM remember long-term dependence information;

[0108] ② The input gate controls the influence of the new input x t on the cell state at the current time;

[0109] The specific expression is shown in formula (4);

[0110] i t =σ(W i ·[h t-1 ,x t ]+b i ) (4)

[0111] where σ is a sigmoid activation function; W i is a weight; and b i is a bias term.

[0112] The second part of the formula needs to calculate the candidate memory information The formula is shown in (5)

[0113]

[0114] The input gate can filter out unimportant information and only store key data in the cell state; in time series data, if new input is directly added at each time, the model is easy to forget historical information or learn unstable, and the input gate ensures that only necessary information is added; the input gate and the forget gate cooperate with each other to enable the LSTM to maintain stable memory in a long time span.

[0115] ③ The output gate; the LSTM cannot directly use the cell state as the output, but needs an output gate to filter and determine which information at the current time should be output as the hidden state h t ; In the calculation process of the output gate, the input features are consistent with the forget gate, and are nonlinearly transformed by a sigmoid activation function to finally generate an output result in the range of 0 to 1, which can be mathematically expressed by formula (6);

[0116] o t = σ(W o ·[h t-1 ,x t ]+b o ) (6)

[0117] The cell state of the current layer is activated by a tanh function, and then multiplied by the output value to generate the final output result, as shown in formula (7);

[0118] h t = o t *tanh(C t ) (7)

[0119] The output gate (Output Gate) is responsible for filtering useful information from the cell state and determining the output content of the hidden state h t .

[0120] (2) Gated recurrent unit network

[0121] The structure of the LSTM is simplified while maintaining the ability to learn long-term dependencies; the LSTM controls the flow of information through the forget gate, the input gate, and the output gate.

[0122] (3) Transducer represented by bidirectional encoder

[0123] Bidirectional context modeling has completely changed the field of natural language processing (NLP).

[0124] The entity matching module provided by the embodiment of the application has the following advantages:

[0125] 1) A subject entity matching method based on multi-modal fusion

[0126] First, the question sentence is subjected to named entity recognition, and the recognized entity mentions are sorted to obtain subject entity mentions. Then, the subject entity mentions and the picture corresponding to the question sentence are subjected to feature extraction respectively, and after nonlinear mapping, the vectors after connection are subjected to multi-modal feature fusion. The fused feature vector is matched with the entities in the knowledge graph to obtain the matched subject entities.

[0127] In the first step of the subject entity matching method based on multi-modal fusion, named entity recognition is performed to find all entity mentions in the question sentence. The question entity recognition task of the application adopts the BERT-GlobalPointer model proposed in the third chapter, and the candidate entity set is obtained through recognition.

[0128] 1.1) Entity mention sorting

[0129] After the named entity recognition model is used to recognize the entity mentions of the question sentence, the candidate entity set is obtained, which needs to be prioritized to obtain the subject entity mentions. The application analyzes the grape disease and pest field question and answer expectation, and designs a sorting rule for this field. The specific sorting rule is as follows.

[0130] The triple structure in the knowledge graph is usually defined as (head entity, relationship, tail entity), wherein the head entity is the active connection point in the network, and the tail entity is the entity that passively receives the relationship. From the perspective of query optimization, the starting node is positioned as the initial end of the relationship chain, and the target node is set as the subsequent entity constraint condition, which can more conveniently generate a query statement to enhance the retrieval accuracy of the question and answer system. Based on this logic, in the entity screening stage, the system will preferentially determine the entity mentions of the head entity type as the subject entity mentions that need to be highlighted. If there are multiple head entities, the entity type that points to more other entity types is selected.

[0131] 1.2) Subject entity matching model based on multi-modal fusion

[0132] The model creation includes: a model architecture of a feature extraction module, a fusion module and an entity matching module; a text represented by a topic entity mention and an image about the question are taken as inputs; image features and text features are extracted respectively, then the image features and the text features are nonlinearly mapped and output to the fusion module, in the fusion module, the features of two modalities [I, T] are connected; then the connected features are sent to l self-attention layers to provide outputs for further processing in the entity matching module; then cosine similarity calculation is performed, and the topic entity in the knowledge graph is matched;

[0133] The state space model is composed of two main parts, the state equation: describes how the state of the system is transferred from one time to the next time; the observation equation: describes how to generate observation data from the state of the system;

[0134] x'(t)=Ax(t)+Bu(t)

[0135] y(t)=Cx(t)+Du(t)

[0136] Where x(t) is a state vector representing the system; u(t) represents an input vector, which is an external control signal; y(t) represents an output vector of the system, which is usually a system variable that we can observe; A, B, C and D are all transformation state matrices;

[0137] The self-attention layer of the application uses 5 layers;

[0138] In the MHA of the Transformer model, the input is composed of query (Q), key (K) and value (V) vectors, which are obtained by linear transformation of the input vector X; the Transformer calculation method is as follows:

[0139] Q,K,V=XW q ,XW k ,XW v

[0140]

[0141] Where X is the input feature matrix; W i is a linear transformation matrix used to linearly transform the input X to different subspaces; Q, K and V are query vectors, key vectors and value vectors obtained by multiplying the input by the corresponding weight matrix;

[0142] 2) Multi-modal entity matching

[0143] First, when multi-modal entity matching needs to be performed on an entity, first, the image entity corresponding to the queried entity is retrieved according to the HasImage relationship, and the corresponding picture is found according to the storage path representing the image entity; the image features are extracted using the VMamba model, and by virtue of the SS2D mechanism of the VMamba model, each pixel in the image can effectively integrate the information from all other pixels in different directions by adopting a complementary one-dimensional traversal path, so that a global receptive field can be established in a two-dimensional space, and thus the appropriate image features of the image submitted by the user and the image entity in the multi-modal knowledge graph are extracted;

[0144] Then, the similarity between the image submitted by the user and the image entity in the multi-modal knowledge graph is calculated, that is, the cosine similarity is used to compare the similarity between the image feature vectors; the cosine similarity formula; the entity with the highest similarity is obtained through the similarity calculation, which is the answer to the question.

[0145] As shown in Figure 2 The grape disease and pest automatic question answering method based on the multi-modal knowledge graph provided by the embodiment of the application comprises:

[0146] S101, the data collected by the data collection processing module on the webpage adopts HttpClient to request external website articles, uses Hutool's JSONUtil to parse and preprocess the articles, and uses jsoup to request related interfaces in real time to obtain pictures;

[0147] S102, the knowledge graph construction module analyzes different structures of data, extracts the required information from the data, and then stores it in the knowledge base; the main control module detects and classifies various proper nouns in the text by using the naming entity recognition module as an automatic means;

[0148] S103, the text classification model module classifies large-scale text data by using deep learning; the multi-modal fusion module mixes data of different modalities in a feature space to obtain fused features;

[0149] S104, the knowledge graph storage module stores the recognized entities in a file, uses the entities as keywords, and uses the foregoing technology to obtain and preprocess images; the label corresponding to each picture is the entity name corresponding to the image; the images that have been filtered, standardized and preprocessed are stored in a server, and the storage location of the images in the server is used as the image entity in the knowledge graph;

[0150] S105, through the entity matching module based on multi-modal fusion theme entity matching technology; the theme entity matching is divided into three stages of entity recognition, entity sorting and entity matching, and the theme entity is matched out through the entity matching fusion multi-modal feature matching; the answer query is evaluated through the query evaluation module.

[0151] Another object of the present application is to provide a computer device comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to make the processor execute the steps of the method for automatic question answering of grape diseases and pests based on multi-modal knowledge graph.

[0152] Another object of the present application is to provide a computer readable storage medium storing a computer program, the computer program being executed by a processor to make the processor execute the steps of the method for automatic question answering of grape diseases and pests based on multi-modal knowledge graph.

[0153] Another object of the present application is to provide an information data processing terminal for realizing the automatic question answering system of grape diseases and pests based on multi-modal knowledge graph.

[0154] The present application is specifically implemented:

[0155] 1. The present application constructs an intelligent interactive platform for grape disease control, which realizes automatic question answering function relying on multi-modal knowledge graph technology. In order to solve the difficulties in knowledge acquisition and unprofessional problem expression faced by grape growers in actual operation, the project team uses Django framework for system development, and takes Neo4j graph database as the core carrier of knowledge storage. In terms of technical implementation, the system integrates an entity recognition model based on BERT-GlobalPointer architecture, which is specially used for accurate extraction of grape disease related terms. The question answering process includes: first, the problem theme is identified, then the intent classification is completed, and finally the query parameters are generated through the entity association relationship chain. The system uses Cypher query language to search information in multi-modal knowledge graph, and combines multi-source data matching technology to return the correct answer, and the retrieval result is visualized through Web interface. The platform not only provides strong support for the improvement of farmers' planting benefits, but also promotes the innovative development of intelligent agriculture.

[0156] 2.1 Knowledge graph

[0157] 2.1.1 Overview

[0158] Knowledge graph is a large-scale semantic network that structures knowledge as entities and their relationships. Its core is to build a multi-dimensional knowledge system through "entity-relation-entity" triples and "entity-attribute-value" attribute triples. It integrates scattered heterogeneous data (such as text, images, and videos) into a unified graph structure model, explicitly expresses semantic associations through nodes (entities / concepts) and edges (relationships / attributes), and supports semantic reasoning and complex queries. The construction of knowledge graph usually relies on information extraction, knowledge fusion, and entity alignment technologies. Its characteristics include semantic association, dynamic evolution, and interpretability, and it can extract high-density knowledge networks from massive data. Typical applications include intelligent search (such as precise answer extraction), intelligent question answering, personalized recommendation, natural language understanding (such as semantic disambiguation), and decision support in vertical fields (medical diagnosis, financial risk control). It is an important infrastructure for artificial intelligence to realize cognitive and reasoning capabilities.

[0159] 2.1.2 Ontology Concept Layer

[0160] The organization and management of entities and their relationships in the real world are achieved through the structured form of knowledge graph, and its logical architecture can be divided into schema layer and data layer (Zhang et al. 2022). The construction strategy of knowledge graph mainly includes two types: top-down and bottom-up (Li 2019). The former emphasizes the prior design of ontology and schema layer, while the latter first filters high-confidence information from open datasets to build the schema layer.

[0161] The definition of domain concepts and the explicit expression of their relationships can be achieved through the ontology layer, which enables the systematic and standardized representation of knowledge in a specific domain. In addition, this layer uses computer-readable language, providing an effective way for knowledge sharing and reuse between humans and machines. In the field of grape pest research, the construction of an ontology concept system enables the systematic organization and scientific management of data resources in this field. In this process, the boundaries of knowledge extraction are clearly defined in non-standardized and structured texts through the pre-definition of entity relationships and attribute characteristics, which not only optimizes the standardization of triple extraction, but also significantly improves its execution efficiency (Wu 2021).

[0162] 2.1.3 Knowledge Graph Construction

[0163] Knowledge graph construction is the process of analyzing different structured data, extracting information, and storing it in a knowledge base.

[0164] There are two methods of top-down and bottom-up for constructing knowledge graph. According to the data situation, choose top-down or bottom-up, the former fills the data layer based on the existing knowledge system, and the latter supplements the concept schema through data abstraction.

[0165] Domain knowledge graph construction needs to obtain multi-source data, including books, websites, databases, etc. The data is divided into three categories: structured (such as relational databases), semi-structured (such as encyclopedia data), and unstructured (such as text). Knowledge extraction technology extracts core knowledge such as entities, relationships, and events from these data.

[0166] 2.2 Basic structure of multi-modal graph

[0167] Multi-modal knowledge graph (MMKG) is defined as a knowledge symbol system integrating multiple expression forms, its constituent elements cover text, visual information, audio data and dynamic image and other multi-media, represented as g = {ε, R, A, T, V} where T = {T A ,T R} and T A = ε × A × V, T R = ε × R × ε. The present invention distinguishes two MMKG representation methods: attribute-based multi-modal knowledge graph-A-MMKG and entity-based multi-modal knowledge graph-N-MMKG (X. Z et al. 2024). That is, in the definition of A-MMKG, modal information such as pictures is represented as attribute information of symbolic entities; in the definition of N-MMKG, modal information such as pictures is represented as independent entities, which establish relationships with other entities.

[0168] In detail, attribute-based multi-modal knowledge graph takes attribute as the core, emphasizing the fine-grained description of multi-modal features of entities (such as color, shape, text description or image features) through attribute triples (entity-attribute-value), and the attribute values themselves may be directly associated with multi-modal data (for example, describing the "appearance" attribute of an entity with images or audio), focusing on unified representation across modalities through attribute dimension; while entity-based multi-modal knowledge graph takes entity as the hub, builds a semantic relationship network between entities through triples (entity-relation-entity), and multi-modal data (such as images, videos) usually exist as auxiliary information or embedded features of entities, mainly used to enhance the expression of entities themselves or relationship reasoning (for example, through entity images to assist in judging the rationality of the "cat-predation-bird" relationship).

[0169] 2.3 Named entity recognition

[0170] Named entity recognition (NER) is a key technology in natural language processing, used to extract specific semantic units from unstructured text. It detects and classifies proper nouns in text, covering categories such as names, places, organizations, time, dates, currencies, and professional terms. The core method uses sequence labeling technology (such as BIO / BILOU system) to assign entity labels to words to locate boundaries and types (Huang Hengqi et al. 2019).

[0171] 2.3.1 Rule-based and dictionary-based methods

[0172] Rule-based and dictionary-based named entity recognition is one of the classic methods in the field of natural language processing. Its core idea is to locate and classify entities (such as names, places, and organizations) from unstructured text through artificially designed rule templates and pre-built domain dictionaries. This method dominated early research and is particularly suitable for scenarios with limited annotated data or strong domain specialization.

[0173] Grinberg et al. (1995) first proposed a natural language processing framework called "link grammar" and designed a corresponding parsing algorithm. This algorithm combines rule libraries and dictionary resources and shows significant performance advantages when processing English sentences. Boldyrev et al. (2013) constructed specialized handcrafted dictionaries containing common entity categories such as names, geographic names, and organizations, enabling automatic annotation of named entities in text. Zhang Xiaoheng et al. (1997) established a rule-based organization name recognition model. Yuan Jindou et al. (2022) innovatively used three methods: domain entity dictionary, word formation feature character matching, and part-of-speech combination feature matching, successfully extracting relevant entities in the electricity safety domain from unstructured text.

[0174] Rule-based and dictionary-based named entity recognition methods are still irreplaceable in specific scenarios, especially in conditions with strong domain specialization and limited annotation resources. However, their limitations have driven the integration of statistical learning and deep learning, forming a hybrid paradigm of "rule guidance + data-driven" and laying the foundation for the development of modern NER technology.

[0175] 2.3.2 Machine learning-based entity recognition

[0176] Named entity recognition technology usually relies on the comprehensive application of various machine learning algorithms. Its core lies in automatically extracting text features using various algorithm models to complete entity positioning. Unlike traditional methods that rely on artificial rule construction, this technology path uses data-driven methods for models to learn text rules independently. In terms of specific model selection, statistical learning methods such as maximum entropy models, hidden Markov models, and conditional random fields are widely used in this field.

[0177] (1) Maximum Entropy Model

[0178] Maximum Entropy Model (MEM) is a probabilistic model that estimates the probability distribution that best fits the observed data given the constraints (Berger AL 1996). The core idea of this model is the Maximum Entropy Principle, which chooses the distribution with the maximum entropy to minimize the assumption of unknown information without additional assumptions.

[0179] For a classification task, we want to estimate a class distribution P(y|x), where x is the input feature and y is the target class. The goal is to find a probability distribution P(y|x) that best describes the data while satisfying the maximum entropy principle under the constraints. As shown in equation (1).

[0180] H(Y|X)=-∑ x,y P(x)P(y|x)logP(y|x) (1)

[0181] Modern deep learning methods such as BERT and Transformer have surpassed the Maximum Entropy Model, but their core ideas (such as log-linear modeling and maximum likelihood estimation) are still widely used in the NLP field.

[0182] (2) Hidden Markov Model (HMM) Figure 3

[0183] Hidden Markov Model (HMM) is a probabilistic graphical model that models a random process with hidden states, especially suitable for sequence data. The structure of Hidden Markov Model (HMM) can be represented by a state transition graph, which contains hidden states and observations, and the dependency relationship is represented by arrow symbols, where the horizontal direction arrow represents the transition process between hidden states, and the vertical direction arrow represents the output mapping from hidden states to visible states.

[0184] Hidden state x i is a state that cannot be directly observed, for example, in the part-of-speech tagging task, these may be "noun, verb, adjective". y i is an observable variable, such as words, phonemes, DNA fragments, etc. P(x i-1 |x i ) is the state transition probability, representing the probability of x i-1 transitioning to x i . P(y i |x i ) is the observation probability, representing the probability of y i transitioning to x i .

[0185] The core idea of entity recognition using Hidden Markov Model (HMM) is to treat the text sequence as an observation sequence and the entity class of each word as a hidden state, and then use state transition probability and observation probability for labeling. The hidden state set is defined as the entity class (such as PER for person name, LOC for location name, ORG for organization name, and O for non-entity), the observation sequence is the input text word sequence, the state transition probability matrix (A) of the model is used to represent the transition probability between different entity classes, for example, "after a person name, it is more likely to be followed by a surname or a position, rather than a location name", the observation probability matrix (B) is used to represent the probability of a certain word appearing under a certain entity class, for example, "apple" is more likely to correspond to ORG (Apple Inc.) than PER (person name), and the initial state distribution (π) represents the likelihood of each entity class at the beginning of the sentence. Although HMM performs well in sequence labeling tasks, the Markov assumption limits the modeling ability of long-distance dependencies.

[0186] (3) Conditional Random Field

[0187] Conditional Random Field (CRF) is a probabilistic graphical model used to model sequence labeling problems. This algorithm combines the core concepts of the Markov Random Field theoretical framework and the maximum entropy modeling method, and solves the limitations of traditional models (such as Hidden Markov Model) in sequence labeling through global normalization.

[0188] CRF calculates the optimal solution of sequence labeling by defining a global probability distribution P(Y|X), as shown in formula (2).

[0189]

[0190] 2.3.3 Entity Recognition Based on Deep Learning

[0191] The overall structure of BERT is shown in Figure 5 and can be represented as:

[0192] Input Layer (Input Layer): used to process input text, including TokenEmbedding (word embedding), SegmentEmbedding (sentence differentiation), and PositionEmbedding (position encoding).

[0193] Transformer Encoder Layers: BERT Base consists of 12 layers of Transformers with 768-dimensional hidden layers and 12 attention heads, while BERT Large expands to 24 layers, 1024-dimensional hidden layers, and 16 attention heads, achieving deep understanding of text through Multi-head Self-Attention.

[0194] Output Layer: Adapted according to different tasks, such as text classification, question-answering systems, and sequence labeling.

[0195] BERT's input uses WordPiece Tokenization, and special tokens [CLS] and [SEP] are inserted at the beginning of the text sequence and between sentences to enable the model to better handle single-sentence and sentence-pair tasks.

[0196] BERT has made revolutionary progress in NLP tasks through the innovative design of Transformer architecture and bidirectional language modeling. Its pre-training + fine-tuning paradigm has become the standard method in the NLP field and has inspired the development of numerous subsequent models such as GPT-3, T5, ChatGPT, etc. Although BERT has a large computational overhead, its outstanding performance makes it an important cornerstone in current natural language understanding tasks.

[0197] 2.4 Text Classification Model

[0198] As a key research direction in the field of Natural Language Processing (NLP), text classification technology plays an important role in multiple application scenarios, including sentiment recognition, spam filtering, and news category division. Early classification strategies were mainly based on the Bag of Words (BoW) or TF-IDF feature extraction method, but these techniques have obvious limitations in effectively capturing the deep semantic features of words. The rise of deep learning has made word embedding-based methods the mainstream.

[0199] 2.4.1 FastText Text Classification

[0200] FastText (Joulin A et al. 2017) is a text representation and classification tool designed to efficiently handle large-scale text data. It can simultaneously complete word feature space mapping and text category discrimination analysis through innovative technical means. Its core function is to solve supervised text classification tasks (Dai L and Jiang K 2018). FastText not only has independent word vector training capabilities, but the word vectors trained can also be effectively applied to feature extraction and classification system construction. Figure 6 The structural composition of the model is shown.

[0201] The core idea of FastText is to build an efficient text classification model based on n-gram features and Hierarchical Softmax. Its main structure is as follows:

[0202] Input Layer: The input text is first segmented and converted into word vectors. FastText uses pre-trained word vectors (such as Word2Vec, GloVe) or randomly initialized word vectors to represent the words in the text.

[0203] Hidden Layer: The word vectors of the text are aggregated using Average Pooling to form the overall representation of the text, avoiding complex deep network calculations.

[0204] Output Layer: The class probability is calculated through Softmax or Hierarchical Softmax to achieve efficient classification.

[0205] In generating text representation, FastText not only uses words but also utilizes n-gram features, which can effectively capture phrase information and improve classification effectiveness. For example, "New York" as a bigram can better express the city name. In large-scale classification tasks, FastText uses Huffman tree structure for category prediction, which improves computational efficiency compared to traditional Softmax. FastText can use SGD (Stochastic Gradient Descent) and Negative Sampling for fast optimization, making it faster to train on large-scale datasets. FastText is fast, lightweight, and robust, making it suitable for online classification tasks of large-scale text data. Therefore, in scenarios requiring fast and accurate text classification, FastText is an efficient and easy-to-use solution.

[0206] 2.4.2 CNN text classification

[0207] In recent years, Convolutional Neural Networks (CNN) has made breakthroughs in computer vision field due to its efficient local feature extraction capability, and has been successfully applied to text classification tasks, especially in short text classification, CNN has strong performance.

[0208] The overall structure of CNN can be represented as:

[0209] Input Layer: The sentence is first segmented and then converted into word vectors (Word Embeddings). Pre-trained word vectors (such as Word2Vec, GloVe, FastText) or randomly initialized word vectors can be used. Assuming the input sentence length is n and each word is represented by a d-dimensional vector, the input matrix dimension is n x d.

[0210] Convolutional Layer: Multiple 1D convolution kernels (filters) are used to extract local features at different n-gram levels. The calculation formula is shown in equation (8).

[0211] h i =f(W·x i:i+h-1 +b) (8)

[0212] Where x i:i+h-1 represents the n-gram fragment with window size h in the input text; W is the convolution kernel weight; b is the bias term; f is the activation function (such as ReLU).

[0213] Pooling Layer: Through MaxPooling or AveragePooling, the most important features are extracted from multiple convolution windows, reducing feature dimension and improving computational efficiency. Pooling operation ensures the translational invariance of the model, that is, no matter where the key word appears in the text, the model can capture its key information.

[0214] Fully Connected Layer: The features after the pooling layer are connected into a fixed length vector, and classified through the fully connected layer. The final class probability is calculated using Softmax as shown in equation (9).

[0215]

[0216] Where, W i is the row vector of the weight matrix corresponding to class i; h is the output vector of the hidden layer.

[0217] The CNN has the following advantages over traditional methods (such as TF-IDF) and other neural networks (such as RNN) in text classification tasks: 1) The convolution kernel of the CNN can extract n-gram features in the text, which can automatically learn the local patterns of the sentence, such as sentiment word groups or key phrases; 2) Compared with RNN, which needs to calculate sequence information step by step, CNN can perform parallel computation, greatly improving the training efficiency; 3) CNN is suitable for feature extraction of short text, such as news classification, microblog sentiment analysis, etc.

[0218] 2.5 Multimodal fusion method

[0219] Multimodal fusion is divided into four categories: early fusion (data layer fusion), deep fusion (feature layer fusion), late fusion (decision layer fusion) and hybrid fusion. Early fusion realizes modality association through data alignment, which has high computational cost and is prone to dimension explosion; deep fusion mixes multimodal data in the feature layer, which has the risk of feature expansion; late fusion fuses decisions after independent processing of modalities, which may miss cross-modal information; hybrid fusion integrates multi-layer information, which significantly increases the complexity of the model.

[0220] 2.6 Summary of the present invention

[0221] This chapter systematically explains the ontology design of knowledge graph, multimodal form, named entity recognition, text classification and multimodal fusion method, and establishes a theoretical framework for subsequent research.

[0222] 3 Construction of grape disease and pest multimodal knowledge graph

[0223] In view of the current situation of data scarcity in the field of agriculture, the construction process includes three modules: data collection and preprocessing (web crawler, OCR recognition, field shooting), mode layer architecture design (disease and pest type / symptom / control standard), knowledge extraction and storage (BERT-GlobalPointer entity recognition, multimodal entity linking, graph database storage).

[0224] 3.1 Construction process

[0225] The data sources cover agricultural forums, search engines, professional literature and field collection. After cleaning and standardization, the core elements such as disease and pest species, symptom characteristics and control measures are abstracted to establish the mode layer. The improved entity recognition model is used for knowledge extraction, and the experimental results show that its performance is better than that of traditional methods. The finally constructed multimodal knowledge graph supports the intelligent question answering system.

[0226] 3.2 Construction of grape disease and pest multimodal data

[0227] This study obtained data by crawling web pages and scanning books. HttpClient, Hutool, and jsoup tools were used for data capture and analysis. The pHash algorithm was used to eliminate similar images. OpenCV and Tesseract-OCR were combined to process images and text. After cleaning and screening, standardization preprocessing was completed.

[0228] 3.2.1 Data acquisition and preprocessing

[0229] Web data was captured using HttpClient, parsed using JSONUtil, and images retrieved in real time using jsoup. The pHash algorithm was used to calculate the Hamming distance to screen image quality. Scanned book images were denoised and corrected using OpenCV, and Tesseract-OCR was used to extract text and analyze the layout. After cleaning, the text data was free of irrelevant content.

[0230] 3.2.2 Construction of Ontology Concept Layer

[0231] Using a combination of top-down and bottom-up approaches, the model was divided into broad categories of diseases and pests, with detailed attributes added, supplemented with practical concepts such as "prone climate." Similar concepts were integrated through the "entity-relationship-entity" triple structure, resulting in 13 entity categories.

[0232] Table 3-1 Category Information Table

[0233]

[0234] Continued Table 3-1 Category Information Table

[0235]

[0236] Entity attributes are extracted from semi-structured text. For example, the "breeding" entity includes attributes such as breeding method. The "HasImage" relationship associates multimodal entities, with text and images complementing each other.

[0237] Table 3-2 Relationships in the Grape Disease and Pest Knowledge Map

[0238]

[0239]

[0240] Table 3-2 Relationships in the Grape Disease and Pest Knowledge Map

[0241]

[0242] 3.3 Knowledge Extraction Method

[0243] The BERT-GlobalPointer model is used for entity recognition, forming triples based on predefined relationships. For example, the text "grape brown spot disease is also known as grape spot disease" can extract relationships such as (grape brown spot disease, DiseaseName, grape spot disease).

[0244] 3.3.1 BERT-GlobalPointer Model

[0245] The BERT model is used to fuse word vectors, sentence vectors, and position vectors for text encoding, with sentence structure segmented using [CLS] and [SEP] tags. GlobalPointer performs entity recognition tasks and supports knowledge graph construction.

[0246] When performing named entity recognition, it is found that there are hierarchical or nested relationships between entities, such as the insecticide "cypermethrin" and "cypermethrin". Conventional models find it difficult to accurately identify nested entities in the task, resulting in a decrease in the accuracy of the entity recognition task. In order to identify nested entities in the field of grape pests and diseases, the present invention uses a method called GlobalPointer instead of the traditional CRF method. By adopting a global normalization method, this method achieves unified recognition of nested and non-nested named entities. GlobalPointer naturally supports the recognition of nested entities because it directly classifies text spans and can distinguish different entity start and end positions, thereby handling complex nested relationships. By comprehensively traversing all potential text spans (Span) in the text, the system performs entity recognition and category determination on each span. Suppose the input t of length n is encoded to obtain a vector sequence [h1,h2,…,h n ], by transforming q i,α =W q,α h i +b q,α and k i,α =W k,α h i +b k,α We can get the sequence vector sequence [q 1,α ,q 2,α ,…,q n,α ] and [k 1,α ,k 2,α ,…,k n,α ]They are vector sequences used to identify the αth type of entity. At this point we can define

[0247]

[0248] In formula (10), q i,α is a vector representing the starting position i of the segment; k j,αis a vector representing the end position j of the segment; q i,α is the inner product of k j,α , is the segment t [i:j] scored as an alpha type entity; t i:j] is a Span consisting of the consecutive elements of text t with positions i to j; that is, s α (i,j) is t [i:j] scored as an alpha type entity possibility.

[0249] In order to enhance the GlobalPointer model's ability to perceive the relative position of entity boundaries, a rotating position encoding (RoPE) is added to the model. The role of the rotating position encoding is to integrate the position information into the word vector through a rotation matrix when calculating attention, so that the model can perceive the length and boundaries of the entity span. The vector x m of position m is shown in (11)

[0250] x ′ m = x m ⊙e imθ (11)

[0251] where m is the position; ⊙ represents element-wise multiplication; θ is a learned parameter; e imθ is a rotation matrix, representing the process of rotating the word vector.

[0252] Unlike traditional named entity recognition methods, the GlobalPointer method converts named entity recognition into a multi-label classification problem in a matrix, which can calculate the scores of all position pairs in parallel at one time, greatly improving the training and inference speed; in theory, it can handle entities of any length, and when processing long texts, not only maintains high accuracy, but also significantly optimizes the computational efficiency and decoding complexity. In the data collected in the foregoing, there are not a few long texts, and the GlobalPointer method can help the present application to perform named entity recognition more efficiently. In order to help the GlobalPointer method to perform multi-label classification, the original author proposes a brand new loss function, i.e. (12).

[0253]

[0254] For a given sample, P α is the set of all alpha type entity heads and tails in the sample; Q α is the complement of P α in the entire head and tail set. Note that we only need to consider combinations of i≤j, and (13) is the definition of P α and Q α .

[0255] Ω={(i,j)|1≤i≤j≤n}

[0256] P α ={(i,j)|t i:j] is an entity of type α} (13)

[0257] Q α =Ω-P α

[0258] In the decoding stage, satisfying s α Fragment t where (i,j)>0 i:j] are all considered as entity outputs of type α. The decoding operation is significantly simpler and its efficiency is

[0259] 3.3.2 Experimental results and analysis

[0260] Experimental environment: operating system Ubuntu 20.04.3 LTS; 4 Intel(R) Xeon(R) Gold 5318Y CPUs @ 2.10 GHz, one NVIDIA A40 GPU; 48 GB video memory; compilation environment Tensorflow 1.14, Python 3.6.3.

[0261] Table 3-3 Experimental environment

[0262] Table3-3ExperimentalEnvironment

[0263]

[0264] The evaluation criteria for named entity recognition generally include precision P (Precision), recall R (Recall) and F1 value (F-measure), as shown in (14), (15) and (16).

[0265]

[0266] Where, T P is the number of entities that are actually entities and are correctly identified; F P is the number of entities that are non-entities but identified as entities; F N The number of entities that were not recognized.

[0267] The data used in the experiment is from the relevant texts in the field of grape disease and pest, and after preliminary screening and cleaning, it forms a sample. The dataset consists of 9589 sentences, with a total character amount of more than 510,000. During the annotation process, the BIOES system was used to perform character-level annotation on the text, and finally a dataset of 18631 grape disease and pest entities was completed. The processed text extracted is divided into training set, validation set and test set according to the ratio of 8:1:1. GlobalPointer is compared with the traditional CRF method. The baseline is the combination of BERT+CRF, and the comparison is the combination of BERT+GlobalPointer, and the experimental results are shown in Tables 3-4

[0268] Table 3-4 Experimental Results

[0269] Table 3-4 Experimental Results

[0270]

[0271] From the experimental results, it can be seen that GlobalPointer not only has a 0.7 percentage point higher F1 on the validation set than CRF, but also has a 1.74 percentage point higher F1 on the test set, and the prediction speed is slightly higher than CRF, and in the comparison of training speed, it is much better than CRF.

[0272] Using the same experimental environment and dataset, the BERT+GlobalPointer model is compared with the commonly used named entity recognition method, and the comparison model includes BiLSTM-CRF (Huang Z et al. 2015), NeuralNER (Lample Get al. 2016), SoftLexicon (Ma R et al. 2020), BERT-BiLSTM-CRF (Wang Ziniu et al. 2019), CollaboNet (Yoon W et al. 2019), and the results are shown in Table 3-5.

[0273] Table 3-5 Comparison of Experimental Results

[0274] Table 3-5 Comparison of Experimental Results

[0275]

[0276]

[0277] Compared with these models. The F1 value of the BERT+GlobalPointer model used in the application is the highest, which is 0.53% higher than that of the commonly used BERT-BiLSTM-CRF. CollaboNet is a relatively advanced method, and the performance of BERT+GlobalPointer is also better than that of CollaboNet. Therefore, the excellent recognition ability of BERT+GlobalPointer to nested entities makes it have better performance on the data set of the application.

[0278] 3.4 Multi-modal knowledge graph storage based on Neo4j

[0279] After performing named entity recognition, the recognized entities are stored in a file, the entities are used as keywords, and the images are obtained and preprocessed using the foregoing technology, and the label corresponding to each picture is the entity name corresponding to the image. The application stores the preprocessed images after screening, standardization and the like in a server, and the storage location of the images in the server is used as an image entity in the knowledge graph. The application uses a cosine similarity algorithm to implement an entity linking task. In an n-dimensional vector space, the similarity between two vectors can be measured by the cosine value of the included angle. Specifically, the measurement value is equal to the ratio of the vector dot product and the product of the respective lengths. Based on the cosine value of the vector included angle, the cosine distance can be further derived. By mapping the image label and the entity into vectors A and B respectively, the similarity calculation can be performed according to formula (17), and the image-text entity with the highest similarity is linked.

[0280]

[0281] The application uses a Neo4j database to store a knowledge graph. As a system specially designed for graph data structures, Neo4j organizes information through three core elements of entities (Entities), connections (Connections) and attributes (Attributes), and is particularly suitable for processing highly interwoven and interdependent data sets. The storage method based on graph theory not only improves the data processing efficiency, but also enhances the analysis ability of complex relationships, and therefore has more advantages than traditional relational databases in the management of relationship-intensive knowledge graphs.

[0282] In view of the excellent performance of Neo4j in the processing of entity nodes and relationship edges, and the efficient batch node import function, the application selects it as a storage medium, and stores the grape disease and pest related data in a structured manner, and then realizes the system construction of the knowledge graph. The operation of data storage in Neo4j mainly includes three implementation ways. The CREATE statement in the Cypher language is used as the basic method, and the data storage is realized by creating nodes and relationships one by one. This scheme has good applicability when dealing with a small amount of entities and relationships. In the face of large-scale data, “Neo4j-import” and “LOADCSV” become a better choice. The former performs outstandingly in execution efficiency and resource occupation, but needs to temporarily suspend the database service; the latter supports batch import of CSV format files, and can dynamically adjust the database during running. Based on the actual needs of data management, the application adopts the LOADCSV scheme, and the specific implementation steps are as follows: store the CSV format file in the import directory of the local graph, and then use the LOAD instruction of the Cypher language to complete the batch import of entities, relationships and attributes in the grape disease and pest field. The constructed multi-modal knowledge graph (part) is as shown in Figure 11 The entities of the image modal are stored in the graph database in the form of a storage path in the server, and the relationship between the two different modal entities of the same knowledge is represented by “HasImage”. The multi-modal knowledge graph constructed by the application includes 19542 entities, 48521 triples, and 9894 pictures.

[0283] 3.5 Summary of the application

[0284] The application mainly introduces the construction of the grape disease and pest multi-modal knowledge graph, uses the crawler technology to extract data and pre-processes the text and image data, designs the ontology concept layer of the grape disease and pest multi-modal knowledge graph, and then uses the BERT-GlobalPointer method for named entity recognition according to the characteristics of the data of the application. The F1 value, training speed and prediction speed of BERT-GlobalPointer are better than those of the traditional model. Then the specific relationship category between entities is judged to complete knowledge extraction, and then the entity linking of different modal data is performed, and finally the multi-modal knowledge graph is stored by using the Neo4j graph database.

[0285] 4.1 Question and answer method design process

[0286] The answer ranking method based on deep learning generates an answer sequence through semantic feature extraction and model matching degree evaluation. The core links include question intention recognition, topic entity matching, reasoning chain generation, query graph construction and query sentence conversion (Wang Zhiyue et al. 2020). The implementation steps of the grape disease and pest question answering system are as follows: 1) text question entity recognition; 2) domain sorting rule screening topic entity; 3) multi-modal fusion accurate positioning; 4) ERNIE-Gram intention classification; 5) building candidate relationship chain; 6) screening the optimal relationship chain; 7) adding entity constraints; 8) generating query sentences; 9) multi-modal matching returns unique results, see Figure 12 .

[0287] 4.2 User multi-modal question understanding

[0288] The question understanding of the present application focuses on multi-modal information integration, and the query sentence generation is divided into two core links: 1) topic entity matching through multi-modal feature fusion; 2) deep inference semantic intention. Break through the limitations of traditional pure text processing, and combine visual data to enhance understanding accuracy.

[0289] 4.2.1 Topic entity matching method based on multi-modal fusion

[0290] In view of the current situation that there is a lack of multi-modal knowledge graph in the field of grape diseases and pests, the present application innovatively fuses text and image features: 1) BERT-GlobalPointer model is used to identify candidate entities; 2) head entity priority, grape variety > disease and pest > symptom of domain priority ranking; 3) improve noise tolerance through image-text feature fusion. The matching process ( Figure 13 ) includes three stages of entity recognition sorting, multi-modal feature extraction and fusion, and knowledge graph entity matching.

[0291] Entity mention sorting rules: 1) preferentially select the head entity in the knowledge graph triple as the relationship initiator; 2) select the entity of the same type with more connections; 3) sort according to the domain priority (Table 4-1). This method effectively solves the problem of non-professional description by farmers, and improves the matching robustness through image assistance.

[0292] Table 4-1 Entity type priority

[0293] Table4-1Entitytypepriority

[0294]

[0295] If the unique topic entity mention cannot be selected by sorting according to the designed entity type priority, then the distance between the entity mention and the interrogative word is calculated. By calculating the distance between the candidate entity mention and the interrogative word in the question, the entity with the smallest distance is preferentially selected as the target topic entity.

[0296] 4.2.1.2 Topic Entity Matching Model Based on Multimodal Fusion

[0297] like Figure 14 As shown, this is the topic entity matching model based on multimodal feature fusion in this paper. To strike a balance between high performance and adaptability to user devices, this model adopts a deep fusion architecture, allowing each modality to extract relatively complete and abstract semantic features in its own channel. The self-attention mechanism enables multi-level semantic association modeling between modalities.

[0298] Specifically, this model creates a model architecture consisting of a feature extraction module, a fusion module, and an entity matching module. It takes as input the text representing the subject entity mention and an image related to the question. Image and text features are extracted separately, then nonlinearly mapped to the image and text features before being fed into the fusion module. In this module, the features [I, T] of the two modalities are concatenated. The concatenated features are then fed into a self-attention layer, providing output for further processing in the entity matching module. Cosine similarity is then calculated to match the subject entity in the knowledge graph.

[0299] The present invention uses VMamba (Visual Mamba) (LiuYetal.2024) for image feature extraction. VMamba is a visual feature extraction method specially designed based on Mamba (GuADaoT2023).

[0300] The foundation of the Mamba model lies in the State Space Model (SSM) (GuA et al. 2021). The SSM is a statistical model used to describe time series data. It is widely used in machine learning and statistics to deal with dynamic systems and time-varying processes. The SSM can capture the evolution of system states over time and the relationship between observed data and these states.

[0301] The state-space model consists of two main parts: the state equation (State Equation), which describes how the state of the system changes from one moment to the next, and the observation equation (Observation Equation), which describes how observation data is generated from the state of the system.

[0302] x'(t)=Ax(t)+Bu(t) (18)

[0303] y(t)=Cx(t)+Du(t) (19)

[0304] where x(t) is the state vector of the system; u(t) represents the input vector, which is the external control signal; y(t) represents the output vector of the system, which is usually the system variable that we can observe; A, B, C, and D are all matrices of transformed states.

[0305] Mamba is designed for handling one-dimensional sequential data. When trying to use it for processing visual data, this poses a new challenge, and to solve this problem, VMamba proposes a two-dimensional selective scan (SS2D), which is a four-way scanning mechanism tailored for spatial domain traversal. VMamba divides the input image into multiple non-overlapping patches, and then expands along four different scanning paths, each of which forms a sequence, converting the 2D data of the image into a one-dimensional sequence for processing.

[0306] The overall architecture of VMamba is shown in Figure 15 The input picture is first segmented into small blocks to obtain a two-dimensional feature map. Then, different hierarchical representations are created through multiple network modules, each of which includes a down-sampling layer (except for the first module) followed by a layer of VSS (Visual State Space) blocks.

[0307] The structure of VSSBlock is shown in Figure 16 The core module of VSSBlock is the SS2D (2D-Selective-Scan) module. The data passing through the SS2D module undergoes three small modules: cross-scan, S6 module with selective scan, and cross-merge. Given the input data, SS2D first expands the input patch into a sequence along four different traversal paths (i.e., cross-scan), uses separate S6 (selective SSMs) blocks to enable the patch sequences to be processed in parallel, and its output generates the final cross-merge feature map through reconstruction and merging operations. SS2D innovatively uses mutually orthogonal one-dimensional scanning paths, enabling multidimensional information exchange between image pixels, so that each pixel can fully capture feature information from all directions in space, thus facilitating the establishment of a global receptive field in two-dimensional space.

[0308] In multi-modal tasks, the features of images and texts often have complex spatial relationships, and SS2D can help the model effectively handle these complex relationships. In the fusion process of image and text features, SS2D can help capture local details (such as the color, shape, and structure of objects) and global semantics (such as the overall concept of the scene) in the image region. These information can be effectively matched with the semantic features in the text. Traditional image feature extraction methods, such as CNN, can extract local features, but cannot effectively integrate global context information in different directions. SS2D, through its selective scanning module, can help establish rich contextual relationships between image regions and between image and text features.

[0309] The application uses a pre-trained Bert-base-chinese on a large amount of Chinese text as a text encoder to extract semantic features of the text. The input of the model is in the same format as the input in the previous named entity recognition, consisting of three parts: word vectors, sentence vectors, and position vectors.

[0310] Image and text features have complex interactions. In order to better capture these relationships and reduce the differences between modalities, the model can learn more complex patterns and structures, rather than just simple linear combinations. Inspired by (Akbari H et al. 2018), after extracting the feature vectors of text and image, the feature vectors are processed by nonlinear mapping. For image features, nonlinear mapping is applied 1x1 convolution and LeakyReLU activation function, and for text vectors, the feature vectors are directly input into the LeakyReLU activation function for activation. Compared with ReLU, LeakyReLU activation function can make the output of negative neurons remain a certain output, so that the model can better capture complex features.

[0311] The application uses the self-attention layer of Transformer for multi-modal feature fusion. Traditional multi-modal feature fusion methods (such as feature concatenation and bilinear pooling) can only capture simple inter-modal relationships. Transformer can model long-distance dependencies in input data through parallel computation. This allows it to effectively capture intra-modal and cross-modal relationships in multi-modal data, such as text words, image regions, or visual and textual associations. The image features and text features after nonlinear mapping are connected to get the input vector X of the self-attention layer. The output is given by pooling the last hidden state. The self-attention layer of the application uses 5 layers.

[0312] In the MHA of the Transformer model, the input is composed of query (Q), key (K) and value (V) vectors, which are obtained by linear transformation of the input vector X. The Transformer calculation method is as follows:

[0313] Q, K, V = XW q ,XW k ,XW v (20)

[0314]

[0315] where X is the input feature matrix; W i is a linear transformation matrix used to linearly transform the input X into different subspaces; Q, K, and V are query vectors, key vectors, and value vectors obtained by multiplying the corresponding weight matrix.

[0316] Each entity in the knowledge graph has been pre-trained with Bert-base-chinese to extract feature vectors, which are stored as entity attributes in the graph database. The invention uses cosine similarity to calculate the similarity between the fused feature vector and the text feature vector of the entity in the knowledge graph. The cosine similarity is shown in formula 3.4 section (3-8). The highest value of the cosine similarity calculation result is the subject entity corresponding to the question.

[0317] 4.2.1.3 Multimodal question answering dataset

[0318] In order to verify the method of matching subject entities in the grape disease and pest field based on multi-modal feature fusion proposed by the invention, the invention uses a self-made grape disease and pest field question answering dataset. The text data is mainly from fruit bang farmer's home, Baidu post bar, grape forum, wisdom agricultural platform, and Huinong net forum website. The invention uses crawler technology to crawl the grape disease and pest field text question answering data and the corresponding picture information together, completes data collection and preprocessing. According to the rules, the corresponding related subject entities are manually completed, and the dataset contains 1752 question-image pairs and corresponding subject entities. The dataset is sorted from 1 to 1752, and the picture in each data is named "serial number-I". The dataset is shown in Table 4-2.

[0319] Table 4-2 Sample dataset example

[0320] Table 4-2 Sample dataset example

[0321]

[0322] To enhance the ability of the method of the present application to eliminate ambiguity, so as to improve the performance of the method, the data set constructed above adds data of the picture modality, as shown in Figure 17

[0323] 4.2.1.4 Experimental results and analysis

[0324] The method used in the present application first performs named entity recognition and entity sorting on the text question, and then inputs the cross-modal picture data and text data into a multi-modal feature fusion model for similarity calculation. The training parameters of the model of the present application are shown in Table 4-3. The present application designs an ablation experiment, compares the model using multi-modal feature fusion with the model using only text features, and uses only the text features extracted by the text feature extraction module as the input of the l-layer self-attention layer to capture the context information in the input data, and then performs the topic entity matching task.

[0325] Experimental environment: operating system Ubuntu 20.04.3 LTS; 4 Intel(R) Xeon(R) Gold 5318Y CPUs @ 2.10 GHz, one NVIDIA A40 GPU; 48 GB of video memory; compilation environment Pytorch 2.2.0, Python 3.8.4.

[0326] Table 4-3 Model training parameter

[0327] Table 4-3 Model training parameter

[0328]

[0329] The present application selects the evaluation indicators commonly used in classification models: accuracy, recall rate and F1 score. The accuracy refers to the proportion of truly positive samples in the samples judged as positive by the model. For topic entity matching, it refers to the proportion of correctly matched entity pairs in the entity pairs judged as matched by the model. The recall rate refers to the proportion of correctly judged positive samples by the model in all positive samples. For topic entity matching, it refers to the proportion of successfully matched entity pairs by the model in all entity pairs that should be matched. The F1 value is the harmonic mean of precision and recall, and the higher the F1 value, the more balanced the model performs in terms of precision and recall.

[0330] The ability to extract features will directly affect the understanding ability of the model for data and scenes, so the influence of the model performance is very important. The image feature extraction of the present application uses the VMamba model, and for the difference analysis of the image coding method, the experiment selects multiple typical models to carry out systematic comparison research, and the experimental results are shown in Table 4-4.

[0331] ​Table 4-4Comparison of image feature extraction module results

[0332] Table 4-4Comparison of image feature extraction module results

[0333]

[0334] From the above table, it can be seen that the VMamba used in the application performs well in all indicators, with very high F1 score, accuracy and recall, which indicates that the model is very excellent in balancing precision and recall, can efficiently detect positive class samples and predict accurately. The comprehensive performance of this model is the best. Compared with several models based on Transformer, the comprehensive performance is improved. Compared with several basic models based on convolutional neural network, the performance is obviously improved.

[0335] Table 4-5Ablation results

[0336] Table 4-5Ablation results

[0337]

[0338] The experimental results are shown in Table 4-5. It can be seen that after using image features and text features for fusion, the performance of the method has been significantly improved, with an increase of 3.55 percentage points in F1 to 96.41%. Multi-modal feature fusion greatly enhances the context understanding ability and disambiguation ability of the topic entity matching task by combining the information of different modalities. Text description may have ambiguous, incomplete or implicit expressions. Multi-modal information can make up for these deficiencies.

[0339] Single modality information may be wrong due to data noise such as misspelling and semantic ambiguity. Multi-modal feature fusion enables the model to rely on its image modality data to compensate when the text modality data is limited, thereby improving the robustness of the model. To verify the improvement of the model in robustness, the application designs an experiment to randomly replace text characters in the question with a certain probability, artificially creates text noise, and evaluates the correlation between noise intensity and F1 value decrease. The experimental results show that in the case of 10% probability of character replacement, the F1 value of the model of the application is decreased by 6.2% less than the single text model, in the case of 20% probability of character replacement, the F1 value is decreased by 9.6%, and in the case of 30% probability of character replacement, the F1 value is decreased by 15.1%.

[0340] Table 4-6Effect of image noise on method robustness

[0341] Table4-6Theinfluenceofimagenoiseontherobustnessofmethods

[0342]

[0343] To verify the robustness of the method, the sensor failure scenario is simulated by adding Gaussian noise. As shown in Table 4-6, three sets of noise experiments are set: when σ = 51, the image is slightly blurred, and the F1 value decreases by 3.7%; when σ = 102, the noise is obvious, and the F1 value decreases by 9.7%; when σ = 255, the image is indistinguishable, and the F1 value decreases by 11.9%. Experiments show that noise significantly reduces performance when it affects image distinguishability, but the decrease slows down when it is completely indistinguishable, confirming that the method has noise interference resistance.

[0344] The method realizes entity matching by fusing the multimodal features of the subject entity mention and the image. Comparative experiments (Table 4-7) show that the entity named recognition and sorting steps are superior to the method of directly fusing the question and the image.

[0345] Table4-7Methodcomparisonresult

[0346] Table4-7Methodcomparisonresult

[0347]

[0348] According to the method of the application, the subject entity can be obtained by performing entity named recognition and reordering on the question, which can significantly improve the accuracy of the final result. There may be multiple entities in a question, but the subject entity matching method determines a unique subject entity, and other entities will interfere with the result. The method of the application excludes such interference through named entity recognition and entity sorting. This may be the reason why the method of the application has a significant advantage in performance.

[0349] 4.2.2Question intent classification based on ERNIE-Gram

[0350] 4.2.2.1Question intent classification task

[0351] Intent classification aims to predict the user's intent category according to the user input, which is the core link of the question answering system. By mapping the user's question to the preset category, the system can accurately determine the query purpose and guide the subsequent answer generation. For example, the user's question "Grape leaves have black spots..." is classified as "entity-disease", and the system generates a knowledge graph query sentence accordingly.

[0352] Most grape disease and pest questions are short texts, although the words are simple, the semantics are complex, and contain redundant information, which significantly increases the semantic sparsity and makes classification difficult.

[0353] Compared with traditional keyword matching, deep learning models can capture deep semantic features to improve classification performance. BERT can analyze text semantics through bidirectional encoding, but its MLM method focuses on word prediction, which can easily ignore phrase-level dependencies.

[0354] The present application adopts ERNIE-Gram model to improve semantic understanding through explicit n-gram mask mechanism. The model masks and predicts phrases / entities as a whole, effectively capturing the internal structure and semantic dependencies of n-gram.

[0355] ERNIE-Gram jointly predicts n-gram units and their internal tokens, and combines attention masks to achieve multi-granularity semantic fusion. Meanwhile, it uses generative modeling to generate n-gram alternatives and learn semantic associations through reinforcement.

[0356] The intent classification categories are shown in Tables 4-8.

[0357] Table 4-8 Question intent class

[0358] Table 4-8 Question intent class

[0359]

[0360]

[0361] Table 4-8 Question intent class

[0362]

[0363] Entity-entity class refers to the user's desire for the system to return the entity itself as the answer, i.e., the specific entity node returned to the user. Entity-attribute class is different, as it should return an attribute value of the entity rather than the entity itself. For example, the question "How to propagate Sunlight Rose?" belongs to the "variety-attribute" class, as the user is interested in the propagation method of "Sunlight Rose" variety, not the variety itself.

[0364] 4.2.2.2 Experimental results and analysis

[0365] Experimental environment: operating system Ubuntu 20.04.3 LTS; 4 Intel(R) Xeon(R) Gold 5318Y CPUs @ 2.10 GHz, one NVIDIA A40 GPU; 48 GB of video memory; compilation environment PaddlePaddle 2.0, Python 3.7.3.

[0366] Table 4-9 Experimental environment

[0367] Table4-9Experimental environment

[0368]

[0369] The question intention classification task is inconsistent with the subject entity matching task in the foregoing. Since the expression form of the natural language query has high diversity, and the performance of the classification model is significantly dependent on the quality level of the training data, it is necessary to specially expand and optimize the existing data set. The present application enriches the data samples by using data enhancement methods such as synonym replacement, entity word replacement, and text structure adjustment (such as reversing the front and back), so as to improve the generalization ability of the model on different question expressions.

[0370] In order to ensure the uniform distribution of question samples of various categories, the present application reorders all the data by randomization processing, and then divides them into three subsets according to the allocation principle of 7:1:2: the data used for model training contains 13125 samples, 1875 samples are used in the verification link, and 3750 samples are contained in the test set. Through this division method, the use ratio of the data set in different stages is effectively guaranteed, and the balanced distribution of sample categories is realized.

[0371] Table4-10Comparison of experimental results

[0372]

[0373] Continuation of Table4-10Comparison of experimental results

[0374]

[0375] The data in Table 4-10 shows that the performance of the FastText model is not satisfactory, and its single structure limits the full extraction of the characteristics of the short text of the grape disease and insect pests, thereby affecting the classification effect. In contrast, the ERNIE-Gram question intention classification model significantly outperforms the traditional neural network model in terms of precision, recall, and F1 value and other key indicators. Even compared with the Transformer-based models BERT and RoBERTa, the comprehensive performance is the best. It proves the high performance of the model on this task.

[0376] 4.3 Relationship chain linking

[0377] 4.3.1 Generation of candidate relationship chain set

[0378] The relationship chain is the path of the topic entity and the answer node. For example, in the question "Which parts of the cold honey are damaged by brown spot disease?", the topic entity is "<cold honey>", and the answer node is ": part". There are two paths: "cold honey-disease-part" and "cold honey-pest-part". The candidate relationship chain set contains all the paths from the topic entity to the answer node, and the "cold honey-disease-part" path has the best quality. For "entity-attribute" type queries, the attribute is attached to the end of the relationship chain for similarity calculation.

[0379] 4.3.2 Relationship chain linking

[0380] The BERT-CNN-Siamese model is used to realize the conversion of natural language relationships and graph paths. The model obtains the vector representation of the question and the relationship chain through BERT, extracts features through the CNN-Siamese network, and evaluates semantic matching using cosine similarity. Nodes are divided into entity nodes (with entity values) and variable nodes (without entity values), and the association between variable nodes and entity variables is constrained to improve the accuracy of question answering. When implementing entity constraints, first search for similar variable nodes in the relationship chain, and if they exist, include them in the constraint set.

[0381] 4.3.3 Experimental results analysis

[0382] Experimental environment: Ubuntu 20.04 system, NVIDIA A40 GPU. Use the grape disease and pest question dataset, replace the topic entity with "entity" to construct the de- entity dataset. Positive samples are marked as 1, and 4 negative samples are randomly selected from the same category. If not enough, supplement across categories. The relationship chain contains the associated path of the knowledge graph, and the attribute information is included when querying attributes.

[0383] Table 4-11 Dataset

[0384]

[0385] To prevent the image features from overfitting to the entity information, and to make the model have to combine the text information of the question and the context in the image, rather than relying on a specific entity name. The dataset intentionally replaces the topic entity with "entity". If only positive examples (i.e. correctly matched data) are included in the training set, the model may learn a "convergent" representation, leading to overfitting and failing to effectively distinguish similar but different entities. This helps the model better learn to distinguish similar but not identical features, and improves the model's ability to distinguish difficult-to-distinguish samples. This experiment intentionally adds four groups of negative examples.

[0386] In order to verify the effectiveness of the BERT-CNN-Siamese model used in the present application in the field of grape disease and pest, the present application selects some popular models in the field of natural language processing for comparison experiments. The experimental results are shown in Tables 4-12.

[0387] Table 4-12 Experimental result

[0388] Table 4-12 Experimental result

[0389]

[0390] The F1 of the traditional model such as BERT or BERT-RNN is only 84.48% and 87.50%, which is far lower than the method of the present application. The F1 value of BERT-CNN-Siamese is as high as 95.12%, which is sufficient to prove the effectiveness of the BERT-CNN-Siamese model in the field of grape disease and pest.

[0391] 4.4 Multi-modal entity matching

[0392] The existing method generates Cypher query for knowledge graph query through multi-modal question understanding, relationship chain linking and entity constraint, but ignores the multi-modal data in the graph. The image entity in the grape disease and pest graph contains unique details, such as when the user asks about the stage of grape brown spot disease, the image features such as lesion size and color need to be used to determine the early stage (angular spot spreading to 1 cm, color change) or late stage (dead with mold) symptoms. The present application matches the user image with the image entity in the graph to identify the stage.

[0393] When performing multi-modal entity matching, first retrieve the corresponding image entity path through the "HasImage" relationship, and use the VMamba model to extract features. The SS2D mechanism integrates pixel information through one-dimensional traversal of the path, and constructs a global receptive field to extract features of the user image and the graph image.

[0394] Calculate the cosine similarity of the user image and the graph image features (see section 3.4 for formula), and the highest similarity entity is the answer.

[0395] 4.5 Answer query and evaluation

[0396] The present application extracts a set of candidate entities through the BERT-GlobalPointer model, determines the theme entity through multi-modal feature fusion, and constructs the "Jufeng grape-pest-site" association path. The variable entity constraint mechanism is introduced to narrow the answer range and improve the accuracy of the question and answer. This mechanism serves as an entity attribute optimization and screening process, enhancing the rigor of knowledge reasoning.

[0397] Convert the data into Cypher instructions to retrieve the knowledge graph,Figure 20 The question and answer process without multi-modal data is shown. For the question "Grape two-star leafhopper damage site" and its image, the system completes information feedback.

[0398] When processing multi-modal information, the user picture and answer picture modal entities need to be matched, Figure 21 The multi-modal entity selection process is shown.

[0399] A grape disease and pest dataset containing 600 image-containing questions (divided into 3 groups) is tested, and after the questions are optimized by experts, the answer quality of the system is artificially evaluated. The experimental results are shown in Table 4-13.

[0400] Table 4-13 Evaluation Results

[0401]

[0402] The accuracy of the question and answer system in the grape disease and pest field reaches 90.68%, indicating that it has high professionalism and can effectively handle most related consultations. However, the system still has certain limitations: some questions cannot be accurately answered, mainly due to two aspects: one is that the complete relationship path cannot be constructed in the knowledge graph, and the other is that there is deviation in the identification of user query intent or the matching of theme entities.

[0403] Three groups of grape disease and pest questions are constructed in the question set, and these questions are all questions that need to use image information of the multi-modal knowledge graph to obtain results. Three groups of grape disease and pest questions are randomly selected from the question set, and the opinions of grape planting farmers and experts are listened to. The questions are improved to a certain extent. The automatic question and answer method proposed in the application is used to query the questions, and the accuracy of the query results needs to be evaluated by artificial quality evaluation. The experimental data and results are shown in Table 4-14.

[0404] Table 4-14 Evaluation Results

[0405]

[0406] The accuracy of the knowledge question and answer of the application adding the multi-modal entity matching process is 88.74%, which shows that the grape disease and pest knowledge question and answer system of the application can answer most grape disease and pest knowledge related questions that need to use multi-modal information.

[0407] Traditional grape disease and pest question and answer relies on single text mode, and there are problems of inaccurate theme entity matching and inaccurate information docking. The application proposes an automatic question and answer method based on a multi-modal knowledge graph: the accuracy of entity matching is improved through multi-modal fusion, the ERNIE-Gram model is used for question intention classification, the relationship chain linking model combined with the BERT-CNN-Siamese architecture and the embedding entity constraint mechanism are used to optimize the accuracy of knowledge reasoning. The system returns a unique answer through multi-modal entity matching, realizing efficient question and answer.

[0408] The application constructs a grape disease and pest multi-modal knowledge graph and develops an automatic question and answer system for planters. Multi-modal data resources are integrated to provide accurate prevention and control technical consultation and support convenient acquisition of professional knowledge.

[0409] The system uses a graph visualization to present the answer, and the architecture is established through business, user and function demand analysis. Combined with the automatic question and answer algorithm, the intelligent query and feedback of disease and pest information are realized.

[0410] In view of the problem of limited information of single text knowledge graph, the multi-modal system developed by the application integrates various modal data, innovates the multi-modal information automatic question and answer method, and enhances the noise resistance.

[0411] The system design focuses on the needs of planters, and sets three links of information input (text / picture upload), query submission (button operation) and result presentation (visual graph + text). The multi-modal query solves the problem of insufficient information integration of traditional systems.

[0412] As shown in Figure 22 The platform includes four core modules: the entity recognition module parses user text to extract entities; the graph database query module matches triple elements; the knowledge question and answer module integrates theme entity matching, intention recognition, relationship chain connection and other technologies; and the visualization module displays the associated triple and text answer in the form of a graph.

[0413] The theme entity entity and the intention classification val are input as key parameters to the relationship chain construction link for processing.

[0414] The module includes three functions: Ger_Ner(ques) uses BERT-GlobalPointer for entity recognition, ERNIE_Label(ques) uses ERNIE-Gram for intention classification, and Entity_Match(ques,image) performs multi-modal entity matching.

[0415] (2) User natural language question and answer query

[0416] The system analyzes the user input text and image, extracts the theme entity and intent, constructs the knowledge graph relationship chain, and generates a Cypher statement query with constraints to query the database. Multi-modal query needs to match image features to return a unique answer.

[0417] Input parameters:

[0418] Table 5-3 Input parameters

[0419]

[0420]

[0421] Return parameters:

[0422] Table 5-4 Return parameters

[0423]

[0424] The system generates a relationship chain based on entity and val, matches the optimal chain through similarity calculation, and generates a Cypher statement to retrieve the answer. In the multi-modal scenario, compare image features with Image_entitys to determine the entity.

[0425] The module encapsulates four interfaces: Sim_Fusion(ques, rela), Create_Rel(entity, val), Create_Cyp(rela, entities), and Mul_Match(image, Image_entitys), deployed in the tmodel directory.

[0426] 5.3.2 System function display

[0427] The system contains four modules: entity recognition, entity query, relationship query, and knowledge question and answer.

[0428] (1) Entity recognition: After the user inputs the text, trigger recognition, the system highlights the named entity (such as "Grape Brown Spot Disease"), and displays classification information when hovering.

[0429] (2) Entity query: Input entity name to generate Cypher statement, return associated triple graph.

[0430] (3) Relationship query: Support entity / relationship combination retrieval, generate graph display.

[0431] (4) Knowledge question and answer: Support text or multi-modal query, upload picture to match features and return a unique answer, no result when prompted "no answer". Test results are shown in Table 5-6.

[0432]

[0433] The system is realized based on a multi-modal knowledge graph, is compatible with mainstream browsers after testing, has an average response of ≤1 second, supports 300 concurrent connections, and has a good interface evaluation and no data leakage risk.

[0434] Application scenarios include agricultural pest consultation, AliMeMKG digital anchor, financial risk control graph, etc. User process: input text / image→entity recognition→graph query→multi-modal matching→return answer and evidence graph.

[0435] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any modification, equivalent replacement and improvement made by any person skilled in the art within the technical range disclosed by the present application, within the spirit and principles of the present application, should be covered within the protection scope of the present application.

Claims

1. An automatic question-answering system for grape pests and diseases based on a multimodal knowledge graph, characterized in that: include: Data collection and processing module, knowledge graph construction module, main control module, named entity recognition module, text classification model module, multimodal fusion module, knowledge graph storage module, entity matching module and query evaluation module; among them, a) The data collection and processing module is configured to request an external web page through a hypertext transfer protocol client, parse and pre-process the article data, and obtain images corresponding to the article in real time; b) The knowledge graph construction module is configured to extract entity, relationship and event information from structured, semi-structured and unstructured data and load it into the knowledge base; c) The main control module is configured to call and coordinate the operation of each module; d) the named entity recognition module is configured to automatically detect and classify domain-specific nouns in the text; e) The text classification model module is configured to use a deep learning model to determine the category of the text; f) the multimodal fusion module is configured to fuse text features and image features in a unified feature space; g) the knowledge graph storage module is configured to store the standardized entities and corresponding images in the form of triples; h) The entity matching module is configured to match the fused features of the question and the image in three stages: entity recognition, entity ranking, and entity matching, to obtain the subject entity; i) The query evaluation module is configured to generate queries based on the subject entities and evaluate the accuracy of the answer results.

2. The system according to claim 1, wherein: The data collection and processing module includes: an HTTP client request unit, a JSON parsing unit and an HTML parsing unit; the HTTP client request unit is used to request a target web page; the JSON parsing unit is used to parse text data in the web page; and the HTML parsing unit is used to parse image links in the web page and obtain corresponding images.

3. The system according to claim 1, wherein: The knowledge graph construction module adopts a top-down and bottom-up approach to construct a domain knowledge graph, wherein the top-down process fills data based on predefined ontologies, and the bottom-up process supplements the concept layer based on extracted entities and relationships.

4. The system according to claim 1, wherein: The entity matching module adopts a multimodal self-attention network and calculates the matching score through cosine similarity, and entities above a preset threshold are determined as subject entities.

5. An automatic question-answering method for grape pests and diseases based on multimodal knowledge graph, characterized in that: The steps include: S1. Capture webpage text and images and preprocess them through the data collection and processing module; S2. Extract entities, relationships, and events through the knowledge graph construction module and generate a domain knowledge graph; S3. Identify entity mentions in the question using the named entity recognition module and determine the question category using the text classification model module; S4. Fusing question text features with corresponding image features through a multimodal fusion module; S5. Input the fused features into the entity matching module to obtain matching results with entities in the knowledge graph; S6. The query evaluation module generates a query statement based on the matching results and outputs an evaluated answer.

6. The method according to claim 5, characterized in that The preprocessing in step S1 includes removing redundant tags, unifying character encoding, and compressing image size.

7. The method according to claim 5, characterized in that The fusion in step S4 adopts a multi-layer self-attention mechanism, and each layer contains parallel multi-head attention calculation units.

8. A computer-readable storage medium having instructions stored thereon, wherein when the instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 5 to 7.

9. A computer device comprising a memory and a processor, wherein the memory stores the computer-readable storage medium according to claim 8, and the processor is configured to implement the method according to any one of claims 5 to 7 when executing the instructions.

10. An information data processing terminal, comprising a user interaction interface, a communication module and a processing chip, wherein the processing chip integrates the system according to claim 1 or is capable of executing the method according to any one of claims 5 to 7.

Images