Entity relation joint extraction method based on multi-modal information fusion
By constructing the MER-CDP model, the problems of data scarcity and difficulty in identifying overlapping triples in the field of crop diseases and pests were solved, achieving efficient entity relation extraction and improving recognition accuracy and applicability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-30
- Publication Date
- 2026-04-07
AI Technical Summary
Existing technologies for entity relation extraction in the field of crop diseases and pests suffer from problems such as scarce datasets, complex entity relations, and difficulty in identifying overlapping triples. Traditional methods also have issues with error propagation, sparsity, and limitations of sequence labeling strategies.
A joint extraction model for crop pest and disease entity relationships based on multi-round sequence labeling, MER-CDP, is constructed. By collecting and labeling corpus, introducing the CasRel model, adopting a multi-round sequence labeling strategy, and combining BERT and MFE-CNN feature extraction, the relationship between head and tail entities is identified.
It improves the recognition performance of overlapping triples in the field of crop diseases and pests, enhances the overall recognition accuracy and applicability of the model, and performs particularly well in the recognition of complex overlapping triples.
Smart Images
Figure CN121808064A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of text extraction technology, specifically to a method for joint extraction of entity relationships based on multimodal information fusion. Background Technology
[0002] Triples are the smallest building blocks of crop disease and pest knowledge graphs, and represent another important research area in knowledge graph construction. They are generally composed of head entities. Tail entity and their relationship Composition, that is Here, the head and tail entities are the named entities extracted from the first two chapters, while the relations represent the semantic relationships that may exist between the head and tail entities. For example, in the sentence "Wheat scab easily harms crops such as wheat", the disease "wheat scab" and the crop "wheat" can form a triple (wheat scab, harms crops, wheat) through the relation "harms crops".
[0003] Currently, entity relation extraction in the field of crop diseases and pests is extremely challenging, mainly due to the following: (1) a lack of entity relation annotation datasets in the field of crop diseases and pests. (2) complex entity relations. There are often multiple overlapping triples in crop disease and pest samples, and there may be positional overlap between different entity pairs. Therefore, how to effectively identify complex relation triples in sentences, especially overlapping triples, will be the research difficulty and focus of this embodiment.
[0004] Currently, commonly used triplet extraction methods include pipelined methods and joint extraction methods. The former independently models named entity recognition and relation extraction tasks, offering some flexibility but suffering from problems such as error propagation, ignoring the correlation between the two tasks, and failing to fully extract and utilize latent semantic features in the text. Therefore, end-to-end joint extraction models based on deep learning have begun to attract widespread attention. These models jointly model the two tasks, simultaneously extracting entities and their semantic relationships, and can be further subdivided into parameter-sharing-based, table-filling-based, Seq2Seq-based, and sequence-labeling-based methods. Parameter-sharing-based methods perform joint training by sharing parameters, but the two tasks remain essentially independent, easily leading to entity redundancy. Table-filling-based methods theoretically can learn the correlation between pairs of characters in a sample, but in practice, they are easily limited by text sparsity and have high spatial complexity. Seq2Seq pattern generation methods are somewhat dependent on the input order of triples, limiting their ability to identify entity boundaries. Sequence-labeling-based methods transform the joint entity-relation extraction problem into a sequence-labeling task, showing some effectiveness, but simple sequence-labeling strategies are not applicable to overlapping triplet extraction. Summary of the Invention
[0005] In order to solve the above-mentioned technical problems, this application proposes the following technical solution: In a first aspect, embodiments of this application provide a method for joint extraction of entity relationships based on multimodal information fusion, including: A dataset of entity relationships for crop diseases and pests was determined through corpus collection and annotation; A joint entity relation extraction model for crop diseases and pests based on multi-round sequence labeling, MER-CDP, was constructed to address the characteristics of corpora in the field of crop diseases and pests. Experimental results and analysis on entity relation datasets validate the effectiveness of the model in identifying overlapping triples of crop diseases and pests.
[0006] In one possible implementation, the step of determining the entity relation dataset for crop diseases and pests through corpus collection and annotation includes: Determine the category information of the entity relationship dataset; The collected data is labeled using manual annotation, rule-based semi-automatic annotation, and heuristic annotation. Corpus feature analysis is achieved by statistically analyzing the proportion of various relations and overlapping triples, as well as the number of head entities, entity pair distances, and the distribution of relation numbers.
[0007] In one possible implementation, the annotation of the collected data through manual annotation, rule-based semi-automatic annotation, and heuristic annotation includes: Initially, entities, their locations, and their relationships were labeled manually. The labeling of entity locations was done to avoid incorrect alignment of entities. To ensure the accuracy and consistency of the labeled relationships and speed up the labeling process, the original labeling tool was upgraded to have a dynamic storage function for triples, thereby enabling semi-automatic labeling of triples and thus speeding up the labeling process. The CasRel model is introduced to assist in labeling relation triples. First, some samples are labeled, then the data is input into the CasRel model for training, and finally the model is used to predict the remaining samples.
[0008] In one possible implementation, the construction of MER-CDP based on the characteristics of crop disease and pest domain corpora includes: First, the head entity recognition module is used to identify k head entities in a given sentence S; For specific head entities The "BIO + Relationship" annotation pattern is used to annotate tail entities that have relationships. The annotation is performed, where "BIO" indicates the position of the current character within the entity, and "relationship" indicates the possible relationship between the head entity and the tail entity. ; For each header entity, repeat the previous step until all header entities have been traversed; The corresponding triples are parsed using the label mapping. .
[0009] In one possible implementation, the MER-CDP includes: a text embedding module, a context feature extraction module, a head entity recognition module, and a multi-round sequence labeling module.
[0010] In one possible implementation, the text representation output by the text embedding module mainly consists of two parts: character-level embedding representation and intrinsic glyph features, wherein the character-level embedding representation and the glyph features are cascaded.
[0011] In one possible implementation, the context feature extraction module uses a bidirectional long short-term memory (BiLSTM) network to learn global context features of the input sequence in both forward and backward directions, and finally combines the forward vector... With backward vector Cascade, i.e. , as the distributed representation of the i-th character The final output of this layer is .
[0012] In one possible implementation, the head entity recognition module decomposes the triple extraction process into a head entity recognition module and a heuristic multi-round sequence labeling module. The head entity recognition module is the first step in implementing multi-round sequence labeling and is mainly used to extract all potential head entities in the sentence. It uses two binary classifiers to extract the start and end positions of the head entities, respectively. The formal definitions of these two binary classifiers are as follows: formula; formula; In the formula, Let be vectors representing the beginning and end characters of the header entity, respectively. Let i be the probability that the i-th character is the first character of the entity. The activation function maps the final output to a range of 0 to 1. and These are trainable weights; If the output of the context feature extraction module is... and The corresponding loss functions are as follows: ; ; In the formula, For the identifier corresponding to the real label, i.e., 0 or 1, the total loss of the head entity extraction module is: .
[0013] In one possible implementation, the multi-round sequence labeling module assigns different labels to each character given a head entity, thereby identifying the corresponding triples. In addition to relying on the text representation obtained by the context feature extraction module, the multi-round sequence labeling module also needs to fuse the feature information of the candidate head entities. The specific operational details of this module are as follows: ; In the formula, For the transformed contextual semantic features, and For trainable parameters, Indicates a cascading operation. The embedding representation of the k-th head entity is calculated using the following formula: In the formula, This indicates the average pooling operation. , For trainable parameters, The embedding representation of the k-th head entity, after unsqueeze and expand operations, yields the result... Candidate head entity vectors of the same dimension This allows for cascading operations along the feature dimension; Further, the following formula is used to... Feature extraction is performed to obtain the final embedded representation information. Where r is the number of tags, , The number of relations; Conditional Random Fields (CRF) are chosen as the decoder to predict pre-designed relation labels, and the loss is calculated by minimizing the negative log-likelihood. ; Finally, the Viterbi algorithm is used to decode the highest-scoring label sequence to obtain the final predicted label: ; For each head entity, the same iterative process is repeated until all relation triples are identified. The overall loss of the proposed model is then: .
[0014] In one possible implementation, the experimental results and analysis on the entity relation dataset verify the effectiveness of the model in identifying overlapping triples of crop diseases and pests, including: Determine the dataset, hyperparameters, comparison model, and evaluation metrics for the experiment; Comparative experiments were conducted on an entity relation dataset to evaluate the performance of MER-CDP. The effectiveness of the improvements to each module in the MER-CDP was evaluated through ablation experiments; Comparative experiments were conducted using precision, recall, and F1 score as evaluation metrics to verify the applicability of MER-CDP on other datasets. The test set samples were divided into three categories based on the type of triplet entity overlap: no overlap, single entity overlap, and double entity overlap to verify the effectiveness of the model in overlapping triplet identification. Finally, the performance evaluation and convergence analysis of the font features of MER-CDP were performed.
[0015] In this embodiment, for the task of entity relation triple extraction in the field of crop diseases and pests, an entity relation dataset for crop diseases and pests is first constructed. Secondly, a joint entity relation extraction model for crop diseases and pests based on multi-round sequence labeling, MER-CDP, is proposed. A multi-round sequence labeling strategy is proposed based on the characteristics of the domain corpus to achieve overlapping triple extraction in the field of crop diseases and pests. Furthermore, the contextual semantics are enriched by combining a language model fine-tuned on the crop disease and pest named entity recognition corpus with glyph features, thereby improving the overall recognition performance of the model. Attached Figure Description
[0016] Figure 1 A flowchart illustrating a joint entity relationship extraction method based on multimodal information fusion provided in this application embodiment; Figure 2 This is a schematic diagram of the relational corpus annotation process provided in the embodiments of this application; Figure 3 This is a statistical diagram of JRE-CDP dataset information provided in an embodiment of this application; Figure 4 This is a schematic diagram of the overall architecture of the MER-CDP model provided in the embodiments of this application; Figure 5 A schematic diagram illustrating the recognition results of each model provided in the embodiments of this application on the JRE-CDP dataset; Figure 6 The F1 scores of the model provided in this application embodiment for various relations in the JRE-CDP dataset; Figure 7 This is a schematic diagram illustrating the experimental results of all models provided in the embodiments of this application on the Agriculture dataset; Figure 8 The diagram illustrates the experimental results of all models provided in the embodiments of this application for different types of relationships; Figure 9 The following graphs illustrate the variation trend of F1 values of various models with the number of iterations in the embodiments of this application. Figure 10 A schematic diagram of experimental results of the MER-CDP model under different hyperparameter conditions provided in the embodiments of this application. Detailed Implementation
[0017] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.
[0018] See Figure 1 The entity relationship joint extraction method based on multimodal information fusion provided in this application includes: S101, through corpus collection and annotation, determines the entity relation dataset for crop diseases and pests.
[0019] Currently, there is no unified standard for classifying entity relationships in specific fields. Based on Chapter 2 and existing research, this embodiment introduces common relationships such as harmful crops, pathogens, and control agents, and adds new relationship categories such as aliases, scientific names, and crop varieties. Furthermore, based on the corpus content, biological taxonomic relationships are further subdivided into common relationship types such as phylum, order, family, and genus, and relationships related to plant parts are subdivided into plant and animal parts and harmful parts, ultimately forming 19 relationships, detailed in Table 1.
[0020] Table 1. Details of the relational categories in the dataset. The data used in this application mainly comes from Baidu Encyclopedia (https: / / baike.baidu.com / ), CNKI (https: / / www.cnki.net / ), and Wanfang Data (http: / / www.wanfangdata.com.cn / ). After manual review, 4222 original corpora were obtained. Figure 2 The detailed annotation process is presented. As shown in the figure, the annotation process mainly consists of three steps: manual annotation, rule-based semi-automatic annotation, and heuristic annotation.
[0021] (1) Manual labeling: In the initial stage of labeling, entities, their positions and relationships are mainly labeled manually. The reason for labeling the position of entities is to avoid the problem of incorrect alignment of entities. For example, in the sample “wheat susceptible to Fusarium head blight”, the correct position of the tail entity “wheat” should be “7” to “8”, but if the position is ignored, it is easy to correspond to the incorrect position “0” to “1”.
[0022] (2) Rule-based semi-automatic annotation. In order to ensure the accuracy and consistency of the annotated relationships and speed up the annotation process, this embodiment upgrades the original annotation tool based on Chapter 2 to enable it to have a dynamic storage function for triples, thereby realizing semi-automatic annotation of triples and thus speeding up the annotation process.
[0023] (3) Relation triple extraction based on the annotation framework. The above methods cannot identify unknown triples, so this embodiment introduces the CasRel model to assist in the annotation of relation triples. Specifically, some samples are first annotated, then input into the CasRel model for training, and finally the model is used to predict the remaining samples. In addition, inspired by the literature, this embodiment uses the five-fold cross-validation method to further solve the labeling inconsistency problem. The specific method is as follows: the labeled data is randomly divided into 5 parts, and 4 parts are used as training data in turn to train the CasRel model. Then, triple prediction is performed on the last part of the sample, and the prediction results are compared with the labeled labels to find triples with inconsistent labels.
[0024] After all the above steps, the final crop disease and pest entity relation dataset JRE-CDP, containing 31,180 triples and 4,222 training samples, is obtained. Some samples are shown in Table 2.
[0025] Table 2 shows some examples of entity relation triples in the JRE-CDP dataset. Statistical analysis was conducted on various aspects, including the proportion of different types of relationships and overlapping triples, the number of head entities, entity pair distances, and the distribution of relationship numbers. Through comparison... Figure 3 The following conclusions can be drawn: (1) There are a large number of overlapping triples. (From...) Figure 3 (a) It can be seen that JRE-CDP has the most triples with single entity overlap, followed by triples with two entities overlapping, and the fewest triples with no entity overlap, which shows complex entity relationships and increases the difficulty of model recognition.
[0026] (2) There exist multiple relation triples with different head entities. Figure 3 (b) shows that nearly half of the samples have more than two head entities, meaning that the traditional "one-to-many" sequence labeling mode (i.e., containing only one unique head entity) is no longer applicable. Furthermore, from Figure 3(d) shows that most samples contain more than two relation triples, with the largest number of samples containing 2-12 relation triples, and some samples containing more than 12 triples, exhibiting a high relation density. A unique feature is that there is at most one semantic relation between individual entity pairs, which lays the foundation for proposing multi-round sequence labeling strategies adapted to domain corpora.
[0027] (3) The distance between entity pairs is relatively large. (From...) Figure 3 (c) It can be seen that some entity pairs are far apart, even exceeding 50 characters, which poses a challenge to the model's ability to capture long-distance dependencies.
[0028] (4) The proportions of various relationships are unbalanced. Figure 3 (e) It can be seen that in the JRE-CDP dataset, the proportion of relationship types such as “harmful crops” and “distribution area” is significantly higher than that of “pathogen name” and “genus and phylum”, and it is challenging to identify such a small proportion of relationships.
[0029] (5) In addition, by Figure 3 (f) shows that the JRE-CDP dataset has more entity relation types, more triples, and a larger data scale than similar data corpora. It can provide data support for subsequent relation extraction research and also provide more knowledge about pests and diseases, laying the foundation for building a rich crop pest and disease knowledge graph.
[0030] S102, a joint extraction model for entity relations of crop diseases and pests based on multi-round sequence labeling, MER-CDP, is constructed to address the characteristics of corpora in the field of crop diseases and pests.
[0031] Depend on Figure 4 As can be seen, MER-CDP mainly consists of four modules: a text embedding module, a context feature extraction module, a head entity recognition module, and a multi-round sequence labeling module. This application first introduces the multi-round sequence labeling strategy, and then describes the implementation details of the model.
[0032] To address the problem of extracting overlapping triples in this field, this study proposes a novel multi-round sequence annotation strategy based on the characteristics of crop disease and pest corpora. The specific annotation process is as follows: (1) First, the head entity recognition module is used to identify k head entities in a given sentence S; (2) For specific head entities... The "BIO + Relationship" annotation pattern is used to annotate tail entities that have relationships. Add annotations. "BIO (Begin-Inside-Other)" indicates the current character's position within the entity, while "relationship" indicates the possible relationship between the head and tail entities. (3) Repeat step 2 for each head entity until all head entities have been traversed; (4) Parse the corresponding triples through the label mapping. .
[0033] by Figure 4 Taking the sample "Susceptibility of rice leaves, panicles, and nodes to rice blast" as an example, first, all head entities "rice blast" and "rice" are identified. Then, for the specific head entity "rice blast," since "rice" has a "harmful crop (HaCrop)" relationship with it, the corresponding position of "rice" is labeled "B-HaCrop / I-HaCrop". Decoding the above labels under the given head entity "rice blast" yields the triples (rice blast, HaCrop, rice). Similarly, decoding the labels "leaf / B-Haloc", "panicle / B-Haloc", and "node / B-Haloc" containing "harmful part (Haloc)" yields the triples (rice blast, Haloc, leaf), (rice blast, Haloc, panicle), and (rice blast, Haloc, node), respectively. Then, given the head entity "rice", the above process is repeated to identify all triples in the sentence with "rice" as the head entity: (rice, Part, leaf), (rice, Part, ear), and (rice, Part, node).
[0034] The multi-round annotation pattern based on "BIO + Relationship" can not only effectively alleviate the error propagation problem caused by the pipeline method of first extracting and then classifying, but also reduce the impact of redundant entities on experimental results. In addition, the biggest difference between this annotation strategy and the literature is that it can extract not only "one-to-many" overlapping triples (i.e., overlapping triples containing only one unique head entity in a given sentence), but also "many-to-many" overlapping triples (i.e., multiple head entities in a given sentence).
[0035] The text representation output by the text embedding module mainly consists of two parts: character-level embedding representation and intrinsic glyph features. Considering that entity types and relationships often have a certain correlation—for example, there is often a control relationship between the disease "wheat scab" and the pesticide "carbendazim," while there is often a harmful relationship between the disease and the crop—and considering that relation triple extraction relies more heavily on contextual semantics, this embodiment uses the pre-trained BERT model fine-tuned in Chapter 3 to generate character-level embedding representations with rich domain knowledge and deep features to enhance the contextual semantic representation capability. Furthermore, this embodiment constructs a Chinese character morphological feature extraction model based on convolutional neural networks (MFE-CNN) to fully utilize the intrinsic glyph features of Chinese characters. The implementation details of its key hidden layers are shown in Table 3. The final output of the text embedding module is a concatenation of character-level embedding representation and glyph features, i.e. .
[0036] Table 3 Implementation details of key hidden layers in the MFE-CNN model The context feature extraction module is crucial for downstream entity recognition and relation extraction due to domain specificity, as contextual semantic information at different locations is essential. This is especially true in crop disease and pest texts, where entity word formation is complex and diverse, and the construction of triples often relies on both forward and backward contextual semantic information. Figure 4 Taking the example shown, backward contextual semantic information such as "leaf," "ear," and "rice blast" is crucial for determining the triple with "rice" as the head entity, while the head entity "rice blast" relies more on forward global contextual semantic information. Therefore, in order to comprehensively learn the semantic information in the text sequence, this embodiment uses a bidirectional long short-term memory network (BiLSTM) to learn the forward and backward global contextual features of the input sequence, and finally combines the forward vectors... With backward vector Cascade, i.e. , as the distributed representation of the i-th character The final output of this layer is .
[0037] Based on the proposed annotation strategy, this embodiment decomposes the triple extraction process into a head entity recognition module and a heuristic multi-round sequence annotation module. The head entity recognition module is the first step in implementing multi-round sequence annotation, primarily used to extract all potential head entities in the sentence. This embodiment employs two binary classifiers to extract the start and end positions of the head entities, respectively. Figure 4Taking the middle sample as an example, if the element at the first position in the starting vector is 1 and the element at the third position in the tail vector is 1, then the head entity "rice blast" is identified. This process is repeated until all head entities are obtained. The formal definitions of these two binary classifications are as follows: (1) (2) In the formula, Let be vectors representing the beginning and end characters of the header entity, respectively. Let be the probability that the i-th character is the first entity. The activation function maps the final output to a range of 0 to 1. and These are trainable weights. This is the output of the context feature extraction module. and The corresponding loss functions are as follows: (3) (4) In the formula, This is the identifier corresponding to the actual label, i.e., 0 or 1. The total loss of the head entity extraction module is then... .
[0038] The multi-round sequence labeling module assigns different labels to each character given a head entity, thereby identifying the corresponding triples. Unlike the head entity extraction module, this module, in addition to relying on the text representation obtained by the context feature extraction module, also needs to fuse the feature information of candidate head entities. The specific operational details of this module are as follows: (5) In the formula, For the transformed contextual semantic features, and For trainable parameters, Indicates a cascading operation. The embedding representation of the k-th head entity is calculated using the following formula: (6) In the formula, This indicates the average pooling operation. , For trainable parameters, The embedding representation of the k-th head entity, after unsqueeze and expand operations, yields the result... Candidate head entity vectors of the same dimension This allows for cascading operations along the feature dimension.
[0039] Further use formula (7) to Feature extraction is performed to obtain the final embedded representation information. Where r is the number of tags, , The number of relations.
[0040] (7) To fully utilize the dependencies between adjacent characters and thus improve the model's recognition accuracy, this embodiment also selects a Conditional Random Field (CRF) as the decoder to predict pre-designed relational labels, and calculates the loss by minimizing the negative log-likelihood. Finally, this embodiment uses the Viterbi algorithm to decode the highest-scoring label sequence to obtain the final predicted label.
[0041] (8) (9) (10) For each head entity, the same iterative process described above is repeated until all relation triples are identified. The overall loss of the model proposed in this embodiment is then: (11).
[0042] S103, Experimental results and analysis on entity relation datasets verify the effectiveness of the model in identifying overlapping triples of crop diseases and pests.
[0043] The models in this application were implemented using the PyTorch 1.7.0 framework, and all models were trained on an NVIDIA GeForce GTX1080Ti.
[0044] Experimental setup: (1) Datasets: This embodiment conducts experiments on the JRE-CDP and Agriculture datasets. The JRE-CDP dataset is used to verify the performance of the MER-CDP model in jointly extracting crop disease and pest entity relationship triples. The Agriculture dataset, also from the agricultural field, contains 1091 samples and 3102 triples, and is used to verify the applicability of the MER-CDP model on different datasets. All datasets are divided in an 8:2 ratio, and the average of the five results is used as the final result.
[0045] (2) Hyperparameter settings: In this embodiment, Dropout is set to 0.5, and the maximum span of early stopping is set to 10 to alleviate the overfitting problem. The learning rate is set to 0.1, the character-level embedding dimension is 768, the number of hidden units of BiLSTM is 300, SGD is selected as the optimizer, and according to the experimental results of the model, the batch size is set to 1 and the maximum number of iterations is 50.
[0046] (3) Comparison Models: To evaluate the performance of the MER-CDP model in extracting crop pest and disease entity relationship triples, this embodiment selects MultiHead, SMHSA, JointER, Tagging, CasRel, and the BERT-BiLSTM-CRF model (BBC) as comparison models. Among them, MultiHead is a joint extraction model based on multi-head selection, SMHSA is a model based on multi-head self-attention mechanism, JointER and CasRel are models based on head and tail pointers, while Tagging, BBC, and the model proposed in this embodiment belong to sequence labeling models.
[0047] (4) Evaluation metrics: In this embodiment, precision (P), recall (R), and F1 score are used as evaluation metrics. A triple is considered to be completely correct if and only if the entity boundaries and relation types of a given relation triple are correctly identified.
[0048] To evaluate the performance of the MER-CDP model, this embodiment conducted comparative experiments on the JRE-CDP dataset. Figure 5(c) It can be seen that the model proposed in this embodiment achieves the best F1 score of 89.99% on the JRE-CDP dataset, which is 3.94% and 6.47% higher than the corresponding values of CalRel and JointER models, respectively. Compared with BBC, MultiHead, Tagging and SMHSA, the improvement is even greater, at 29.18%, 34%, 42.49% and 55.43%, respectively. The relation triplet recognition performance is greatly improved, which is due to: (1) The proposed multi-round sequence labeling strategy can effectively identify overlapping triples with multiple head entities, thus improving the relation triplet recognition performance of the model as a whole. (2) The pre-trained model fine-tuned on the crop disease and pest corpus has domain awareness and can generate high-quality contextual semantics. (3) Experimental results show that font features have a certain auxiliary effect on improving model performance. The SMHSA model achieved the lowest F1 score of 32.56% because its multi-head attention mechanism maps each relation to a different relation subspace. While theoretically capable of modeling the relevance between each relation type and character, its actual recognition performance is less than ideal due to the sparsity of entity relations. Furthermore, the Tagging model only achieved an F1 score of 47.50% because its sequence labeling strategy has limitations in recognizing overlapping triples. In contrast, the BBC model shows further improvement because it proposes a sequence labeling method adapted to the domain corpus, effectively recognizing overlapping triples under the condition of a unique head entity. The MER-CDP model proposed in this embodiment can further recognize complex overlapping triples with multiple head entities, thus further improving its precision, recall, and F1 score. Therefore, the above experimental results demonstrate the effectiveness of the proposed model in recognizing crop disease and pest relation triples, especially complex overlapping triples.
[0049] Figure 6The F1 score was used as the evaluation metric to visualize the detailed recognition of each type of relationship in the JRE-CDP dataset by all models. All models showed high recognition accuracy in relationships such as crop damage, alternative names, pathogen / vector, family, pesticide, and affected parts. However, most models generally performed poorly in recognizing relationships such as damage period, pathogen scientific name, and control period. For example, except for MultiHead, most models achieved a maximum F1 score of only 39.77% for "damage period," because this type of relationship is similar to relationships such as "control period" or "insect stage / cycle," easily leading to semantic confusion. The model proposed in this embodiment not only achieved the best F1 score for easily identifiable relationships but also performed well in most difficult-to-identify relationships. For example, the F1 score for "control period" was 58.23%, significantly higher than other models. Therefore, the above experimental results further validate the effectiveness of the model proposed in this embodiment in extracting triplet relationships of crop pest and disease entities, laying the foundation for the next step of constructing a knowledge graph for crop pests and diseases.
[0050] To evaluate the effectiveness of each improved module, Table 4 lists five control groups, including MER-CDP. crf This refers to the MER-CDP model using CRF as the decoder; MER-CDP softmax This indicates that softmax is used as the decoder, "MER-CDP" 原始BERT "" indicates the use of the original BERT, "MER-CDP" word2vec "" indicates that traditional word2vec is used to generate character-level embeddings, and "-glyph features" indicates that glyph features are not introduced. Comparing Experiments 1 and 2, it can be seen that when the MER-CDP model uses CRF, its F1 score is 0.56% higher than that using softmax, demonstrating the effectiveness of using Conditional Random Fields for decoding. Comparing Experiments 1 and 3, it can be seen that MER-CDP... crf The F1 value is higher than that of MER-CDP 原始BERT The performance improved by 0.42%, indicating that fine-tuning BERT helps improve the overall recognition performance of the model. Additionally, Experiment 4 introduced the word2ve model for comparison. In this experiment, the character-level embedding dimension was set to 300, while other settings remained unchanged. Comparing Experiments 1, 3, and 4, it can be seen that MER-CDP... word2vec On the JRE-CDP dataset, the F1 score was only 81.76%, far lower than the corresponding values in Experiments 1 and 3. Therefore, the above experimental results indicate that fine-tuning the BERT model on the crop disease and pest corpus helps improve the overall recognition performance of the model. In addition, the results of Experiments 1 and 5 show that character shape features have a certain promoting effect on improving the overall recognition performance of the model.
[0051] Table 4 Ablation Experiment Results To verify the applicability of the MER-CDP model on other datasets, this embodiment uses precision, recall, and F1 score as evaluation metrics and conducts comparative experiments on the Agriculture dataset. Figure 7 As can be seen, although the recognition accuracy of MultiHead, SMHSA, Tagging, and BBC models improved slightly on this dataset, their accuracy remained low due to issues such as error propagation, sparsity, and limitations of sequence labeling strategies, with the highest F1 score being only 86%. In contrast, the MER-CDP model proposed in this embodiment still achieved the highest F1 score of 87.75% on this dataset, 1.75% higher than the CasRel model. Therefore, the experimental results demonstrate that the MER-CDP model has certain applicability on other datasets.
[0052] To verify the effectiveness of the MER-CDP model in recognizing overlapping triples, this embodiment first divides the samples in the test set into three categories according to the type of triple entity overlap: no overlap, single entity overlap, and double entity overlap. The statistical analysis is then performed on the recognition results of MultiHead, SMHSA, JointER, CasRel, BBC, and MER-CDP. Figure 8 As can be seen, the MER-CDP model outperforms other comparative models in F1 scores for different types of overlapping triples. Compared to the BBC model, the model proposed in this embodiment not only achieves better F1 scores on "non-overlapping" and "single-entity overlapping" triples, but also improves the recognition accuracy of "double-entity overlapping" triples. This is because the multi-round sequence labeling strategy proposed by the MER-CDP model overcomes the limitation of the BBC model in effectively recognizing overlapping triples containing multiple head entities. As the degree of overlap increases, the recognition ability of the MultiHead and SMHSA models for overlapping triples gradually decreases. In addition, because the JointER and CasRel models have a certain ability to recognize overlapping triples, they achieve better results on all three datasets, but are still slightly inferior to the MER-CDP model. Therefore, the above experimental results intuitively demonstrate the effectiveness of the MER-CDP model in recognizing overlapping triples.
[0053] Furthermore, this embodiment is based on the number of triples in a sample. NThe test set was divided into five classes, containing 1, 2, 3, 4, and 5 or more relation triples, respectively, to analyze the ability of the MER-CDP model to extract multiple triples from a sentence. The statistical results are shown in Table 5. As the table shows, the recognition accuracy of MultiHead and SMHSA decreases with increasing triple number, possibly because their ability to extract overlapping triples weakens as triple complexity increases. The BBC model, however, shows an upward trend. Statistical analysis of the final prediction results reveals that the proportion of relation triples with unique head entities gradually increases, and BBC excels at recognizing these triples. However, its overall recognition performance is inferior to the JointER and CasRel models. Furthermore, the JointER, CasRel, and MER-CDP models show relatively stable performance, with the MER-CDP model achieving the highest F1 score across all datasets, improving upon the CasRel model by 10.44%, 0.33%, 0.14%, 1.78%, and 2.33%, respectively, demonstrating the best performance. Therefore, the above experimental results show that MER-CDP has advantages in extracting complex overlapping triples.
[0054] Table 5. Experimental results of all models on different types of samples. This embodiment compares MFE-CNN with a 3D convolutional neural network-based morphological feature extraction model (3D CNN) and a radical embedding model based on CNN and BiLSTM to further illustrate the rationality of the character feature extractor proposed in this embodiment. Table 6 shows that regardless of the pre-trained model used, the MER-CDP model based on MFE-CNN achieves the highest F1 score, at 89.57% and 89.99%, respectively. The F1 score of the MER-CDP model using 3D CNN is slightly lower than that in Experiment 5, possibly because 3D CNN tends to increase model complexity, which can actually reduce model performance when the amount of data is small. Furthermore, the radical embedding method (Experiments 3 and 4) did not achieve optimal performance in the relation extraction task of this embodiment because splitting Chinese characters into radical sequences easily destroys their inherent character features, which are crucial for enriching contextual semantics and thus improving recognition ability. Therefore, the above experimental results show that the character feature extraction method based on 2D CNN is more advantageous in the joint entity relation extraction task of this embodiment.
[0055] Table 6. MER-CDP recognition results based on different morphological feature extraction methods The trends of F1 scores for each model on the JRE-CDP dataset with training epochs are as follows: Figure 9As shown in the figure. The red solid line represents the MER-CDP model proposed in this embodiment. As can be seen from the figure, the MER-CDP model has the best performance, mainly reflected in: (1) The initial F1 value of this model is 66.62%, which is much higher than other models, while MultiHead is close to 0%, because MultiHead needs to identify all possible candidate entities first and then extract relations, resulting in a slower convergence speed. (2) The F1 value of MER-CDP always remains ahead throughout the training process. Similar to the JointER model, although the CasRel model has a faster convergence speed, its F1 value is still lower than that of the MER-CDP model when it is close to convergence. Therefore, the above results show that the model proposed in this application has better convergence.
[0056] Table 7 lists some typical recognition cases of the BBC and MER-CDP models on the JRE-CDP dataset. As shown in the table, both the BBC and MER-CDP models can correctly identify triples containing only one triple (Case 1) or multiple triples with unique head entities (Case 2). When a sample contains multiple triples with different head entities, the BBC can only identify one triple with the same head entity. For example, Case 3 contains two head entities, "spinach downy mildew" and "spinach," and the BBC's recognition result is missing "(spinach, part, leaf)". The MER-CDP model, however, can correctly identify all triples. This is due to its multi-round sequence labeling strategy, which decomposes the problem of identifying triples with multiple head entities into a multi-round unique head entity triple sequence labeling problem, thus identifying all triples.
[0057] Table 7 Typical Cases of the MER-CDP Model on the JRE-CDP Dataset This embodiment focuses on comparing two hyperparameters that significantly impact model performance: batch size and Dropout, to illustrate the rationality of hyperparameter settings. First, Figure 10 (a) Comparative experiments were conducted with batch sizes of 1, 2, 4, 8, 10, 12, 14, and 16. As shown in the figure, when the batch size is set to 1, the model achieves the highest F1 score of 89.99% on the JRE-CDP dataset, demonstrating the best overall performance. Furthermore, combined with... Figure 10 (b) It can be seen that the model converges fastest under this condition. Therefore, this embodiment sets the batch size to 1. Furthermore, to illustrate the rationality of the Dropout setting, this embodiment conducted comparative experiments using values between 0.1 and 0.9 with an interval of 0.1. Figure 10(c) It can be seen that the model achieves the maximum F1 score when the Dropout value is 0.5. Therefore, in this embodiment, the Dropout value of the model is set to 0.5.
[0058] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0059] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.
Claims
1. A method for joint extraction of entity relations based on multimodal information fusion, characterized in that, include: A dataset of entity relationships for crop diseases and pests was determined through corpus collection and annotation; A joint entity relation extraction model for crop diseases and pests based on multi-round sequence labeling, MER-CDP, was constructed to address the characteristics of corpora in the field of crop diseases and pests. Experimental results and analysis on entity relation datasets validate the effectiveness of the model in identifying overlapping triples of crop diseases and pests.
2. The entity relation joint extraction method based on multimodal information fusion according to claim 1, characterized in that, The entity relation dataset for crop diseases and pests, determined through corpus collection and annotation, includes: Determine the category information of the entity relationship dataset; The collected data is labeled using manual annotation, rule-based semi-automatic annotation, and heuristic annotation. Corpus feature analysis is achieved by statistically analyzing the proportion of various relations and overlapping triples, as well as the number of head entities, entity pair distances, and the distribution of relation numbers.
3. The entity relation joint extraction method based on multimodal information fusion according to claim 2, characterized in that, The annotation of collected data through manual annotation, rule-based semi-automatic annotation, and heuristic annotation includes: Initially, entities, their locations, and their relationships were labeled manually. The labeling of entity locations was done to avoid incorrect alignment of entities. To ensure the accuracy and consistency of the labeled relationships and speed up the labeling process, the original labeling tool was upgraded to have a dynamic storage function for triples, thereby enabling semi-automatic labeling of triples and thus speeding up the labeling process. The CasRel model is introduced to assist in labeling relation triples. First, some samples are labeled, then the data is input into the CasRel model for training, and finally the model is used to predict the remaining samples.
4. The entity relation joint extraction method based on multimodal information fusion according to claim 1, characterized in that, The construction of MER-CDP based on the characteristics of crop disease and pest corpus includes: First, the head entity recognition module is used to identify k head entities in a given sentence S; For specific head entities The "BIO + Relationship" annotation pattern is used to annotate tail entities that have relationships. The annotation is performed, where "BIO" indicates the position of the current character within the entity, and "relationship" indicates the possible relationship between the head entity and the tail entity. ; For each header entity, repeat the previous step until all header entities have been traversed; The corresponding triples are parsed using the label mapping. .
5. The entity relation joint extraction method based on multimodal information fusion according to claim 4, characterized in that, The MER-CDP includes: a text embedding module, a context feature extraction module, a head entity recognition module, and a multi-round sequence labeling module.
6. The entity relation joint extraction method based on multimodal information fusion according to claim 5, characterized in that, The text representation output by the text embedding module mainly consists of two parts: character-level embedding representation and intrinsic glyph features. The character-level embedding representation and glyph features are cascaded.
7. The entity relation joint extraction method based on multimodal information fusion according to claim 6, characterized in that, The context feature extraction module uses a bidirectional long short-term memory (BiLSTM) network to learn the global context features of the input sequence in both forward and backward directions. Finally, the forward vector is processed... With backward vector Cascade, i.e. , as the distributed representation of the i-th character The final output of this layer is .
8. The entity relation joint extraction method based on multimodal information fusion according to claim 7, characterized in that, The head entity recognition module decomposes the triple extraction process into a head entity recognition module and a heuristic multi-round sequence labeling module. The head entity recognition module is the first step in implementing multi-round sequence labeling and is mainly used to extract all potential head entities in the sentence. It uses two binary classifiers to extract the start and end positions of the head entities, respectively. The formal definitions of these two binary classifiers are as follows: official; official; In the formula, Let be vectors representing the beginning and end characters of the header entity, respectively. Let i be the probability that the i-th character is the first character of the entity. The activation function maps the final output to a range of 0 to 1. and These are trainable weights; If the output of the context feature extraction module is... and The corresponding loss functions are as follows: ； ； In the formula, For the identifier corresponding to the real label, i.e., 0 or 1, the total loss of the head entity extraction module is: .
9. The entity relation joint extraction method based on multimodal information fusion according to claim 8, characterized in that, The multi-round sequence labeling module assigns different labels to each character given a head entity, thereby identifying the corresponding triples. In addition to relying on the text representation obtained by the context feature extraction module, the multi-round sequence labeling module also needs to fuse the feature information of the candidate head entities. The specific operation details of this module are as follows: ； In the formula, For the transformed contextual semantic features, and For trainable parameters, Indicates a cascading operation. The embedding representation of the k-th head entity is calculated using the following formula: In the formula, This indicates the average pooling operation. , For trainable parameters, The embedding representation of the k-th head entity, after unsqueeze and expand operations, yields the result... Candidate head entity vectors of the same dimension This allows for cascading operations along the feature dimension; Further, the following formula is used to... Feature extraction is performed to obtain the final embedded representation information. Where r is the number of tags, , The number of relations; Conditional Random Fields (CRF) are chosen as the decoder to predict pre-designed relation labels, and the loss is calculated by minimizing the negative log-likelihood. ; Finally, the Viterbi algorithm is used to decode the highest-scoring label sequence to obtain the final predicted label: ； For each head entity, the same iterative process is repeated until all relation triples are identified. The overall loss of the proposed model is then: 。 10. The entity relation joint extraction method based on multimodal information fusion according to any one of claims 1-9, characterized in that, The experimental results and analysis on the entity relation dataset verify the effectiveness of the model in identifying overlapping triples of crop diseases and pests, including: Determine the dataset, hyperparameters, comparison model, and evaluation metrics for the experiment; Comparative experiments were conducted on an entity relation dataset to evaluate the performance of MER-CDP. The effectiveness of the improvements to each module in the MER-CDP was evaluated through ablation experiments; Comparative experiments were conducted using precision, recall, and F1 score as evaluation metrics to verify the applicability of MER-CDP on other datasets. The test set samples were divided into three categories based on the type of triplet entity overlap: no overlap, single entity overlap, and double entity overlap to verify the effectiveness of the model in overlapping triplet identification. Finally, the performance evaluation and convergence analysis of the font features of MER-CDP were performed.