Semi-supervised entity extraction method for citrus mapping knowledge domain
Through semi-supervised learning and mutual reinforcement learning between large language models and small models, combined with self-attention mechanism and CRF model, the problem of insufficient labeled data in entity extraction in citrus knowledge graph is solved, and the accuracy and efficiency of naming entity recognition is improved. , reducing the cost of data labeling.
Patent Information
- Application Number
- CN202411160600.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-12-13
AI Technical Summary
The entity extraction method in the citrus knowledge graph has the problem of less labeling data, which leads to insufficient recognition capabilities of named entity models and high data labeling costs.
Using semi-supervised learning method, pseudo-notation data is pseudo-notated through large language models, and pseudo-notation data is input into small models for training. The self-attention mechanism and CRF model are used for feature extraction and label classification, combining large models and small models mutual reinforcement learning to improve recognition accuracy.
With less labeled data, the accuracy of naming entity recognition is improved, the cost of data labeling is reduced, and the accuracy and efficiency of entity extraction in the citrus knowledge graph are achieved.
Smart Images
Figure CN119128171A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of named entity recognition in the field of information extraction, and specifically to a semi-supervised entity extraction method for citrus knowledge graphs. Background Art
[0002] Citrus is one of the ancient fruit varieties in my country, with a long history of cultivation. After years of industrial development, the modern citrus industry is a technology-intensive industry. How to use digital technology to promote the standardization, scale, branding and harmlessness of citrus production has become the development direction of the citrus industry.
[0003] At present, the construction of the agricultural industry brain based on artificial intelligence and big data technology has become the main means of agricultural industry digitalization, which includes designing data, knowledge, rules, algorithms, models, components and other resources and capability elements. The digitalization of the citrus industry needs to be further improved, and the big data of the entire chain of the citrus industry needs to be established to accelerate the digitalization of production, processing, market, circulation supervision, marketing, brand operation, resource circulation, cold chain logistics, and industrial chain services.
[0004] Knowledge graph has powerful semantic processing capabilities. It can extract structured knowledge from massive texts and images, and integrate data from different sources to form a knowledge network rich in semantic relationships, that is, a structured knowledge system. As an emerging technology in the field of knowledge management, knowledge graph has been widely used due to its strong scalability and support for intelligent applications. It is an important technical means to promote the digitalization of the agricultural industry and build the brain of the agricultural industry. Knowledge graph is a large-scale knowledge base system built on the framework of "semantic network". The semantic network can be regarded as a huge graph, in which nodes represent entities and edges represent semantic relationships between entities. The knowledge graph is filled with a large amount of knowledge content in the semantic network framework. This knowledge content comes from various digital resources such as databases, document libraries, and data files. The knowledge graph collects and organizes scattered knowledge, which can help to achieve the association and integration of knowledge resources, and provides an ideal technical means to solve the problem of "knowledge islands".
[0005] Knowledge graphs have broad application prospects in agricultural technology services in the citrus industry. Focusing on the citrus industry, we build a citrus knowledge graph covering the primary, secondary and tertiary industries, involving knowledge such as citrus varieties, business value, pest and disease treatment, pesticides and fertilizers, production and processing, logistics and sales, etc., converting human-oriented unstructured knowledge into machine-oriented structured knowledge, combining intelligent retrieval, personalized recommendation and other technologies to provide agricultural technology services for citrus farmers and solve the current problem of lack of talents in agricultural technology promotion and agricultural knowledge promotion.
[0006] Entity vocabulary is an important foundation for the knowledge graph. In order to build an accurate and complete citrus knowledge graph, it is necessary to obtain accurate and complete entity vocabulary. Therefore, an accurate named entity extraction method is particularly important. It is necessary to identify and extract entity vocabulary from citrus planting related information. Entity extraction is the most basic and critical part of information extraction, which is divided into rule-based and dictionary-based, statistical machine learning-based, and deep learning-based extraction methods.
[0007] The rule-based extraction method is to customize rules according to the characteristics of the text for entity recognition and extraction. It has good flexibility and is easy to operate. The dictionary-based method was first used in conjunction with the rule-based extraction method. The dictionary-based entity recognition method is to search and match the corpus with the dictionary, extract the words in the dictionary, and thus obtain prior information. Although the dictionary-based and rule-based entity recognition method is much higher than manual extraction in terms of evaluation indicators such as extraction efficiency and accuracy, this method requires artificial rules to be formulated according to entities in different fields, and its portability is poor.
[0008] The method based on statistical machine learning is to manually construct features based on a given, labeled training set, and label each word in the text according to a specific model to achieve named entity recognition. After using the machine learning method, there is no need to manually build templates or rules, which reduces a certain amount of labor costs.
[0009] With the continuous development of computer hardware, computer computing power has been gradually improved, and the named entity method based on deep learning has gradually emerged and has become the mainstream method of named entity recognition. This method replaces the original complex and cumbersome feature engineering by introducing deep neural networks as semantic encoders for feature extraction, and can automatically learn grammatical features and other information features provided from the text, thereby greatly improving recognition.
[0010] Although the deep learning-based method can greatly improve the accuracy and efficiency of recognition, a large amount of labeled data is often required to train a named entity model with high accuracy. The citrus knowledge graph involves a wide variety of entity types, including citrus varieties, business value, pest and disease treatment, pesticides and fertilizers, production and processing, logistics and sales, etc. The required labeled data is more complicated and there is less labeled data. In order to improve the recognition ability of the named entity model when there is less labeled data and reduce the related data labeling costs, this embodiment adopts a semi-supervised method for training, and improves the recognition accuracy by enhancing the large language model and the small model. Summary of the invention
[0011] The purpose of this invention is to address the existing deficiencies in entity extraction methods in citrus knowledge graphs and propose a semi-supervised entity extraction method for citrus knowledge graphs.
[0012] In order to solve the above technical problems, the present invention provides the following technical solution: a semi-supervised entity extraction method for citrus knowledge graph, comprising the following steps:
[0013] Step (1) prompting the large model with labeled data;
[0014] Step (2) using the large model obtained in step (1) to pseudo-annotate the unannotated training set to obtain a pseudo-annotated training set;
[0015] Step (3) inputting the pseudo-annotated data obtained in step (2) into the small model for training;
[0016] Step (4) After a round of training, the prediction ability of the small model is verified using a labeled validation set to obtain the prediction results of the labeled validation set;
[0017] Step (5) processes the verification result obtained in step (4), and for the sentences with incorrect prediction, inputs them into the large model that has passed step (1) through the verification prompt sample template for re-prompting;
[0018] Step (6) repeats the above process until a preset number of training rounds is reached or the verification result accuracy reaches the highest value.
[0019] Preferably, the specific implementation process of step (1) is as follows:
[0020] Use a small amount of labeled data to prompt the large model. Use GPT3.5 to input the labeled data sentence S into the large language model GPT3.5 through the prompt template for prompting;
[0021] Preferably, the specific implementation process of step (2) is as follows:
[0022] Step (2) Split the unlabeled corpus data to obtain unlabeled training set data train Then use the large model from step (1) to train the unlabeled training data train Perform pseudo-annotation to obtain pseudo-annotation training.
[0023] Preferably, the specific implementation process of step (3) is as follows:
[0024] Step (3) inputs the pseudo-annotated data obtained in step (2) into the small model for training. The small model is based on the BERT-BiLSTM-CRF model, and a self-attention layer is added after the BiLSTM, using the attention mechanism to give different degrees of attention to different contextual information.
[0025] Preferably, the feature vector processing method in step (3) is:
[0026] For the pseudo-labeled data data obtained in step (2) T First, process the data and edit it into a suitable data format to get data T′ , and then use BERT to encode the text vector. T′ The sentence DT in DT, the word sequence token in DT=(D T1 ...D TL ) We use BERT to encode the character sequence vector to obtain the character sequence encoding vector H bert =(h1...h L ), L is the length of the character label sequence, and d is the vector dimension. bert The final sentence vector representation with contextual semantics is obtained through BiLSTM N is the number of characters in the sentence, d h The number of feature dimensions for each character;
[0027] The obtained sentence vector representation H with contextual semantics is input into the self-attention module for feature extraction to obtain H end , the formula is:
[0028] H end =Self-Attention(H)
[0029]
[0030] where d k is the number of K vector dimensions;
[0031] The final eigenvector H emd , in this embodiment, it is input into CRF for label classification to obtain the final character label y;
[0032] y=CRF(H end ).
[0033] In the small model, it is necessary to calculate the score Sc(DT,y) of the output tag sequence y corresponding to the sentence DT;
[0034]
[0035] Where A is the tag transition probability matrix, where P i,yi Represents a single character mapped to a tag yi The unnormalized probability of n is the length of the character sequence. Using the softmax function, a probability value is defined for each correct tag sequence y, Y xRepresents all tag sequences;
[0036]
[0037] In training, we need to maximize the likelihood probability, so we use the log-likelihood;
[0038]
[0039] Finally, the loss function in CRF is used to calculate the loss LOSS, which is defined as:
[0040] LOSS = -log(p(y|X)).
[0041] Preferably, the specific implementation process of step (4) is as follows:
[0042] After a round of training, the labeled validation set data dev The prediction ability of the small BERT-BiLSTM-CRF model with self-attention is verified, and the prediction result of the labeled validation set is obtained. dev .
[0043] Preferably, the specific implementation process of step (5) is as follows:
[0044] Results for the validation set data with Ground Truth labels dev After processing, if it is found that there is an entity tag prediction error, the sentence with the wrong prediction is input into the large model that has been prompted in step (1) to prompt it again, so as to update, enrich and stimulate the knowledge in the large model. For the wrong prediction, "entity word 1" does not belong to label 1.
[0045] Preferably, the specific implementation process of step (6) is as follows:
[0046] Repeat the process from step (2) to step (5) until the preset training rounds are reached or the verification result accuracy reaches the highest value.
[0047] Compared with the prior art, the present invention has the following beneficial effects:
[0048] The model disclosed in the present invention is composed of a large model and a small model, and a semi-supervised training method is used to train the model. In a round of training, the method uses a prompted large language model to pseudo-annotate an unlabeled training set to obtain pseudo-annotated data, and then inputs it into the small model for training. After a round of training, the small model is verified using a labeled validation set. For sentences with prediction errors in the prediction results, this embodiment combines the sentence with a verification prompt sample template and inputs it into the large model for re-prompting, thereby improving the data annotation ability of the large model and digging deep into the relevant knowledge contained in the large language model. Through the mutual reinforcement learning and prompting of the two models, the accuracy of the overall model can be improved when there is less labeled data. Specifically, in one round of training, firstly, a small amount of labeled data is used to prompt the large model; secondly, the prompted large model is used to predict the unlabeled data training set to obtain entity vocabulary and its pseudo-label, and then the obtained pseudo-labeled training set is input into the small model for training. After one round of training, the small model is verified using the verification set composed of labeled data; finally, the prediction results of the verification set are processed. For the sentences with incorrect predictions, this embodiment uses the verification prompt sample template to input it into the large model to prompt the large model again to update the knowledge of the large model. Then the above training steps are repeated again until the accuracy of the verification result reaches the expected level or the pre-set training round is reached. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is a work operation flow chart of the present invention.
[0050] Figure 2 It is a schematic diagram of the model structure of the present invention.
[0051] Figure 3 It is a prompt sample template of the present invention.
[0052] Figure 4 It is the question template of the present invention.
[0053] Figure 5 It is a sample template of the verification prompt of the present invention. Specific implementation methods
[0054] The working process of the present invention is further described below in conjunction with the accompanying drawings.
[0055] Reference Figure 1 The overall operation flow chart of the present invention is a semi-supervised entity extraction method for citrus knowledge graph, which includes the following steps:
[0056] Step (1) combines the labeled data into prompts through prompt sample templates and inputs them into the large model to prompt the large model.
[0057] Step (2) uses the large model from step (1) to transform the unlabeled training set data train The sentences in the question template are assembled and pseudo-annotated to obtain the pseudo-annotated training set data T .
[0058] Step (3) inputs the pseudo-annotated data obtained in step (2) into the BERT-BiLSTM-CRF small model with a self-attention mechanism added for training.
[0059] Step (4) After a round of training, use the labeled validation set data dev Verify the prediction ability of the small model and obtain the prediction result of the labeled verification set dev .
[0060] Step (5) processes the verification result obtained in step (4). For the sentences with incorrect prediction, the verification prompt sample template is used to input them into the large model that has passed step (1) for re-prompting.
[0061] Step (6) repeats the above process until a preset number of training rounds is reached or the verification result accuracy reaches the highest value.
[0062] Furthermore, the specific implementation process of step (1) is as follows:
[0063] Step (1) Use a small amount of annotated data to prompt the large model. This embodiment uses GPT3.5 as an example, and inputs the annotated data sentence S into the large language model GPT3.5 through a prompt template for prompting. The prompt template can be as follows: Figure 3 shown.
[0064] For example, for sentence S: "The ugly one, also known as the unknown fire, is widely planted in Sichuan Province.", "ugly" and "unknown fire" are citrus names, and Sichuan Province is the place of origin. For sentence S, according to the template prompt sentence, "The given sentence is: @"ugly" is also known as "unknown fire" and is widely planted in Sichuan Province.@; The given label set is: ["place of origin", "citrus name", ...]; In the given sentence, "ugly" belongs to the citrus name label, "unknown fire" belongs to the citrus name label, and Sichuan Province belongs to the place of origin label." The prompt can mobilize the relevant prior knowledge of the large model.
[0065] Furthermore, the specific implementation process of step (2) is as follows:
[0066] Step (2) Split the unlabeled corpus data to obtain unlabeled training set data train Then use the large model from step (1) to train the unlabeled training data trainPseudo-labeling is performed to obtain a pseudo-labeled training set. Specifically, the pseudo-labeled training set is combined with the question template and input into the GPT3.5 large model for prompting. The question template is as follows: Figure 4 shown.
[0067] For example, for sentence T: "The main planting area of sugar oranges is Guangdong.", use the template to let the big model annotate it, which can be expressed as: "The given sentence is: @"The main planting area of sugar oranges is Guangdong." @, the given label set is: ["place of origin", "name of citrus", ...]; please judge which entity label category the entity noun in the given sentence belongs to in the given label set, and return it in JSON data format, the format is as follows {"sentence": "given sentence", "word": ["entity 1", "entity 2"], "type": ["entity label 1", "entity label 2"]}". After being annotated by the big model, the pseudo-annotated data data is obtained. T .
[0068] Furthermore, the specific implementation process of step (3) is as follows:
[0069] Step (3) inputs the pseudo-annotated data obtained in step (2) into the small model for training. The small model of this embodiment is based on the BERT-BiLSTM-CRF model, and adds a self-attention layer after the BiLSTM, that is, the attention mechanism is used to give different degrees of attention to different context information, so that the BERT-BiLSTM-CRF entity recognition model that introduces the attention mechanism strengthens the weight of semantic features related to the citrus field when extracting semantic features, and also weakens the weight of semantic features unrelated to the citrus field.
[0070] 3-1 For the pseudo-labeled data obtained in step (2) T In this embodiment, data is first processed and edited into a suitable data format to obtain data T′ , and then use BERT to encode the text vector. T′ The sentence DT in DT, the word sequence token in DT=(D T1 ...D TL ) We use BERT to encode the character sequence vector to obtain the character sequence encoding vector H bert =(h1...h L ), L is the length of the character label sequence, and d is the vector dimension. bert The final sentence vector representation with contextual semantics is obtained through BiLSTM N is the number of characters in the sentence, d h The number of feature dimensions for each character.
[0071] 3-2 The context semantic sentence vector representation H obtained in 3-1 is input into the self-attention module for feature extraction to obtain H end , the formula is:
[0072] H end =Self-Attention(H)
[0073]
[0074] where d k is the number of vector dimensions K.
[0075] The final eigenvector H obtained by 3-3 against 3-2 end , this embodiment inputs it into CRF for label classification to obtain the final character label y.
[0076] y=CRF(H end ).
[0077] In the small model, it is necessary to calculate the score Sc(DT,y) of the output tag sequence y corresponding to the sentence DT.
[0078]
[0079] Where A is the tag transition probability matrix, where P i,yi Represents a single character mapped to a tag yi The unnormalized probability of n is the length of the character sequence. Using the softmax function, a probability value is defined for each correct tag sequence y, Y x Represents all tag sequences.
[0080]
[0081] In training, we need to maximize the likelihood probability, so we use the log-likelihood.
[0082]
[0083] Finally, the loss function in CRF is used to calculate the loss LOSS, which is defined as:
[0084] LOSS = -log(p(y|X))
[0085] Furthermore, the specific implementation process of step (4) is as follows:
[0086] Step (4) After a round of training, the labeled validation set data devInput into the small model to verify its predictive ability and obtain the prediction result of the labeled validation set. dev .
[0087] Furthermore, the specific implementation process of step (5) is as follows:
[0088] Step (5) Result of the validation set data with Ground Truth label dev If an entity tag prediction error is found after processing, the incorrectly predicted sentence will be Figure 5 The verification prompt sample template combination shown is input into the large model that has been prompted in step (1) to prompt it again to update, enrich and stimulate the knowledge in the large model. For the wrong prediction, it is "entity word 1" that does not belong to label 1.
[0089] Furthermore, the specific implementation process of step (6) is as follows:
[0090] Repeat steps 2-5 until the preset training rounds are reached or the verification result accuracy reaches the highest value.
Claims
1. A semi-supervised entity extraction method for citrus knowledge graph, characterized in that: The following steps are involved: Step (1) prompting the large model with labeled data; Step (2) using the large model obtained in step (1) to pseudo-annotate the unannotated training set to obtain a pseudo-annotated training set; Step (3) inputting the pseudo-annotated data obtained in step (2) into the small model for training; Step (4) After a round of training, the prediction ability of the small model is verified using a labeled validation set to obtain the prediction results of the labeled validation set; Step (5) processes the verification result obtained in step (4), and for the sentences with incorrect prediction, inputs them into the large model that has passed step (1) through the verification prompt sample template for re-prompting; Step (6) repeats the above process until a preset number of training rounds is reached or the verification result accuracy reaches the highest value.
2. A semi-supervised entity extraction method for citrus knowledge graph according to claim 1, characterized in that: The specific implementation process of step (1) is as follows: A small amount of labeled data is used to prompt the large model. GPT3.5 is used to input the labeled data sentence S into the large language model GPT3.5 through the prompt template for prompting.
3. A semi-supervised entity extraction method for citrus knowledge graph according to claim 2, characterized in that: The specific implementation process of step (2) is as follows: Step (2) Split the unlabeled corpus data into unlabeled training set data train Then use the large model from step (1) to train the unlabeled training data train Perform pseudo-annotation to obtain pseudo-annotation training.
4. A semi-supervised entity extraction method for citrus knowledge graph according to claim 3, characterized in that: The specific implementation process of step (3) is as follows: Step (3) inputs the pseudo-annotated data obtained in step (2) into the small model for training. The small model is based on the BERT-BiLSTM-CRF model, and a self-attention layer is added after the BiLSTM, using the attention mechanism to give different degrees of attention to different contextual information.
5. A semi-supervised entity extraction method for citrus knowledge graph according to claim 4, characterized in that: The feature vector processing method in step (3) is: For the pseudo-labeled data obtained in step (2), T First, process the data and edit it into a suitable data format to get data T′ , and then use BERT to encode the text vector. T′ The sentence DT in DT, the word sequence token in DT=(D T1 ...D TL ) We use BERT to encode the character sequence vector to obtain the character sequence encoding vector H bert =(h1...h L ), L is the length of the character label sequence, and d is the vector dimension. bert The final sentence vector representation with contextual semantics is obtained through BiLSTM N is the number of characters in the sentence, d h The number of feature dimensions for each character; The obtained sentence vector representation H with contextual semantics is input into the self-attention module for feature extraction to obtain H end , the formula is: H end =Self-Attention(H) where d k is the number of K vector dimensions; The final eigenvector H end , this paper inputs it into CRF for label classification and obtains the final character label y; y=CRF(H end )。 In the small model, it is necessary to calculate the score Sc(DT,y) of the output tag sequence y corresponding to the sentence DT; Where A is the tag transition probability matrix, where P i,yi Represents a single character mapped to a tag yi The unnormalized probability of n is the length of the character sequence. Using the softmax function, a probability value is defined for each correct tag sequence y, Y x Represents all tag sequences; In training, we need to maximize the likelihood probability, so we use the log-likelihood; Finally, the loss function in CRF is used to calculate the loss LOSS, which is defined as: LOSS = -log(p(y|X)).
6. A semi-supervised entity extraction method for citrus knowledge graph according to claim 5, characterized in that: The specific implementation process of step (4) is as follows: After a round of training, the labeled validation set data dev The prediction ability of the small BERT-BiLSTM-CRF model with self-attention is verified, and the prediction result of the labeled validation set is obtained. dev .
7. A semi-supervised entity extraction method for citrus knowledge graph according to claim 6, characterized in that: The specific implementation process of step (5) is as follows: Results for the validation set data with Ground Truth labels dev After processing, if it is found that there is an entity tag prediction error, the sentence with the wrong prediction is input into the big model that has been prompted in step (1) to prompt it again, so as to update, enrich and stimulate the knowledge in the big model. For the wrong prediction, "entity word 1" does not belong to label 1.
8. A semi-supervised entity extraction method for citrus knowledge graph according to claim 7, characterized in that: The specific implementation process of step (6) is as follows: Repeat the process from step (2) to step (5) until the preset training rounds are reached or the verification result accuracy reaches the highest value.