A geological named entity recognition method based on large language model and prompt learning
By using a prompt learning method and combining geological knowledge to optimize the geological named entity recognition of a large language model, the accuracy problem of traditional methods in identifying geological reports is solved, and higher recognition accuracy and recall rate are achieved.
Patent Information
- Application Number
- CN202411381124.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-09-26
- Estimated Expiration
- 2044-09-30
AI Technical Summary
Traditional methods are difficult to efficiently and accurately identify named entities in geological reports, especially in the case of small sample data and complex sentence structures. Directly applying large language models leads to the problem of insufficient professional knowledge.
A prompt-based learning method is used to construct a dataset and perform manual annotation, sort out incorrectly identified entities, integrate geological domain knowledge, and add secondary category verification to improve recognition accuracy.
It significantly improves the accuracy and recall of geological named entity recognition and reduces the error recognition rate, especially when processing complex sentences and small sample data.
Smart Images
Figure CN119337880B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of geological text recognition, and in particular to a large language model geological named entity recognition method based on prompt learning. Background Art
[0002] Geological named entity recognition (NER) is a fundamental and critical component of geological research and practice. Geological entities, such as rocks, minerals, and geological structures, frequently appear in geological reports. Accurately identifying and classifying these entities is crucial for analyzing these reports and conducting data mining. However, accurately identifying NERs in geological reports presents significant challenges. Geological report texts are often highly specialized and feature complex and diverse terminology, making traditional NER methods incapable of meeting the requirements for efficient and accurate recognition.
[0003] Currently, the application of intelligent technologies in geological named entity recognition (NER) holds broad prospects and urgent demands. Domestic and international scholars have conducted extensive research on improving the accuracy of NER. For example, while rule-based and statistical methods have made progress in certain specific applications, they still face challenges such as difficult maintenance and poor generalization. Machine learning-based methods, such as conditional random fields (CRFs) and traditional deep learning models, have shown some improvement but rely on large-scale annotated data for verification, resulting in high annotation costs and difficulty adapting to the ever-changing landscape of new geological knowledge. In recent years, with the development of large language models (such as BERT and GPT), these models have demonstrated excellent performance in natural language processing tasks. However, the direct application of large language models to geological NER still faces challenges such as insufficient expertise and poor recognition. For example, some studies have attempted to apply BERT to entity recognition in geological reports, but the results have been limited when dealing with small sample data and complex sentence structures. Summary of the Invention
[0004] To solve the above technical problems, the present invention provides a method for named entity recognition using a large language model based on prompt learning, comprising the steps of:
[0005] S1: Select sentences containing geological entities from geological reports, construct a dataset, divide it into a test set and a training set according to a certain ratio, and manually annotate the test set;
[0006] S2: Construct a prompt containing the task description, labeling rules, and demonstration examples. Input the prompt and a random sentence from the training set into the large model to make it output the labeling result.
[0007] S3: sort out the entities that are most frequently mislabeled and inaccurately recognized by the large model for various geological entities. Based on these errors, selectively integrate some geological domain knowledge into the previous prompts. Then input the improved prompts together with a random sentence in the test set into the large model to obtain a new output.
[0008] S4. Construct a prompt for secondary category verification, extract various entities from the new output obtained in step S3, and make the large model select a category as the category of the entity in the given category, thereby obtaining the final recognition result;
[0009] S5: Use the test dataset to evaluate the final recognition results of the model.
[0010] Furthermore, the specific method of step S1 is as follows:
[0011] S11: Obtain multiple regional geological report document datasets;
[0012] S12: Using periods as delimiters, the document is broken down into multiple sentence files. Sentences that may contain various geological entities are manually screened to form a dataset.
[0013] S13: Use the dataset as the test set D test , and take N samples from it to form the training set D train , and the test set D test Perform manual annotation.
[0014] In the preferred solution, the specific method of step S2 is as follows:
[0015] S21: Construct task description prompts based on the task requirements of marking different types of geological entities in the input sentence with different specific symbols task_desc and annotation rules Prompt label_rule ;
[0016] S22: Construct K sentences containing various geological entities as example samples D demo , and follow the annotation rules Prompt label_rule , manually mark out the various geological entities in these K sentences and obtain the labeled dataset D labeled , thus obtaining the example set D demonstration , the formula is as follows:
[0017]
[0018] S23: Task Description Prompt task_desc Prompt with annotation rules label_rule And the example set D demonstrationIt is concatenated with the input and fed into the large language model to obtain the output. The formula is as follows:
[0019] Output=LLM(Prompt task_desc ,Prompt label_rule ,D demonstration ,Input).
[0020] In the preferred solution, the specific method of step S3 is as follows:
[0021] S31: Take each sentence in the training set as input, execute steps S21 to S23, and get the output set D output ;
[0022] S32: Manually sorted output set D output The M entities with the most misidentifications for each type of geological entity are used as negative sample sets, where misidentified entities refer to entities that do not belong to the class but are marked as such. At the same time, the easily missed and over-added modifications corresponding to the entity classes that are easy to identify inaccurately are sorted out. Construct a guidance prompt based on error analysis error_based ;
[0023] S33: Task Description Prompt task_desc Prompt with annotation rules label_rule , Guidance Prompt based on error analysis error_based And the example set D demonstration It is concatenated with the input Input and fed into the large language model to obtain the new output NewOutput. The formula is as follows:
[0024] NewOutput=LLM(Prompt task_desc ,Prompt label_rule ,Prompt error_based ,D demonstration ,Input).
[0025] In the preferred solution, the specific method of step S4 is as follows:
[0026] S41: According to the marking rules Prompt label_rule Extract various geological entities Entities in the new output NewOutput types , where types is all categories;
[0027] S42: Construct task description VerifyPrompt based on the task requirement of secondary verification of the type of an entity task_desc , and the output format requires VerifyPrompt format_desc,The task requirement is to select one from all types and "others" as the final type of the entity. Adding the type of "others" can effectively reduce the error of identifying non-entities as a certain type of entity;
[0028] S43: VerifyPrompt the task description task_desc and output format requirements VerifyPrompt format_desc And various entities to be verified types The verification result VerifyOutput is obtained by splicing and inputting the large language model. The formula is as follows:
[0029] VerifyOutput=LLM(VerifyPrompt task_desc ,VerifyPrompt format_desc ,Entities types ).
[0030] In the preferred solution, the specific method of step S5 is as follows:
[0031] S51: NewOutput is obtained for the input sentence Input, thereby obtaining the verification result VerifyOutput, based on which the final recognition result finalOutput is obtained, which is compared with the manual annotation result Annotation;
[0032] S52: Compare each class separately. If the finalOutput contains an entity in a class, but the class in the annotation does not contain the entity, then the number of false positive examples fp increases by one. If the class in the annotation contains the entity, then the number of true positive examples tp increases by one. If the class in the annotation contains the entity but the class in the finalOutput does not contain the entity, then the number of false negative examples fn increases by one.
[0033] S53: For each geological category, record the fp, fn, and tp of all samples in the test dataset as the input sentence Input, and sum them to obtain FP, TP, and FN. The formula is as follows:
[0034]
[0035] Where TestNum is the total number of test set samples, fp i is the number of false positive examples corresponding to the i-th input sentence Input. The calculation of TP and FN is similar to it;
[0036] S54: For each geological category, calculate the evaluation indicators, Precision, Recall, and F-score. The formula is as follows:
[0037]
[0038] Precision indicates how many of the samples predicted as positive are actually positive samples, and recall indicates how many of the positive examples in the samples are correctly predicted. F-score is a comprehensive evaluation indicator of these two indicators. The larger the value, the better the effect.
[0039] Beneficial effects: The present invention converts the entity extraction task into an entity labeling task, identifies various geological entities at one time, and uses different symbols for labeling each type of entity, such as angle brackets, which is conducive to alleviating the problem of incorrect identification of entity categories caused by the large model hallucination problem and the problem of repeated identification of nested entities; selects a part from the test set for preliminary identification, sorts out common errors in the identification results, and increases geological knowledge based on these errors, thereby reducing the problem of incorrect identification caused by lack of geological knowledge; adds secondary category verification, requiring one category to be selected as the category of the entity from several given categories, further reducing the error recognition rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;
[0041] Figure 2 This is an example diagram of the BIOES annotation method;
[0042] Figure 3 This is a screenshot of the recognition indicator results of GPT-3.5;
[0043] Figure 4 This is a screenshot of the recognition indicator results of the GPT-3.5+ method;
[0044] Figure 5 This is a screenshot of GPT-4o's recognition indicator results;
[0045] Figure 6 This is a screenshot of the recognition indicator results of the GPT-4o+ method. DETAILED DESCRIPTION
[0046] like Figures 1-2 As shown, an embodiment of the present invention discloses a large language model geological named entity recognition method based on prompt learning, comprising the steps of:
[0047] S1: Create a dataset. This example dataset is derived from geological report text, primarily from the Nima District Survey Report, the Zaduo County Regional Geological Survey Report, the Wanyangchun City Regional Geological Survey Report in Guangdong Province, and the Jinniu Town Gaoqiao Regional Geological Survey Report. Using periods as delimiters, convert a single document file into multiple txt files consisting of single sentences. Manually select sentence files containing rock or mineral entities to form the dataset.
[0048] S2: Select 200 samples from the dataset as the test dataset D test , randomly select 100 samples from them as the training set D train For the test set D test ,manually annotate the rock entities and mineral entities in the sentence.,BIOES annotation method is used to indicate the beginning (Begin),,the middle (Inside), the end (End), the single-word entity (Single), and the non-entity (Other). Figure 2 An example of annotation is shown.
[0049] S3: Construct a prompt containing task description, labeling rules, and demonstration examples, and compare the prompt with the training set D train The sentence to be recognized is input into the big model together with the input, so that it can output the labeled sentence. The specific steps are as follows:
[0050] S31: Based on the task requirement of labeling the rock and mineral entities in the input sentence according to the labeling rules, construct the task description Prompt task_desc and annotation rules Prompt label_rule , where Prompt task_desc The contents are as follows:
[0051] "You need to label all the rock entities and mineral entities in the given sentence."
[0052] Marking Rules Prompt label_rule The content is as follows:
[0053] The labeling rules are as follows: 1. Use angle brackets '<' and '>' to mark rock entities. 2. Use square brackets '[' and ']' to mark mineral entities. 3. If no rock or mineral entity is identified, output the original sentence directly.
[0054] S32: Construct K sentences containing both rock and mineral entities as example samples D demo , here K value is 2, and according to the marking rule Prompt label_rule , manually mark the rock and mineral entities in these K sentences and obtain the labeled dataset D labeled, thus obtaining the example set D demonstration , the formula is as follows:
[0055]
[0056] Example Set D demonstration Specifically:
[0057] Here are some examples:
[0058] Example 1:
[0059] Input: The main rock types in the area are quartz monzonite and granite, and common minerals include quartz and feldspar.
[0060] Output: The main rock types in the area are <Quartz Monzonite> and <Granite>, and common minerals include [Quartz] and [Feldspar].
[0061] Example 2:
[0062] Input: Basalt is a common igneous rock containing minerals such as olivine and pyroxene.
[0063] Output: <Basalt> is a common igneous rock containing minerals such as [Olivine] and [Pyroxene].
[0064] S33: Task Description Prompt task_desc Prompt with annotation rules label_rule And the example set D demonstration It is concatenated with the input and fed into the large language model to obtain the output. The formula is as follows:
[0065] Output=LLM(Prompt task_desc ,Prompt label_rule ,D demonstration ,Input)
[0066] The following is an example input and output:
[0067] Input: "The Zhuchawomeng dike group usually does not contain quartz. Some plutons (Qilongwulong and Malgongnan) and most dikes have porphyritic textures, with phenocrysts being plagioclase."
[0068] Output: "The Zhuchawomeng and other <vein groups> usually do not contain [quartz]. Some <rock bodies> (Qilongwulong and Malgonan) and most <veins> have porphyritic structures, with phenocrysts being [plagioclase]."
[0069] S4: Refine the large model. For the most frequently mislabeled rock and mineral entities, add these entities to the previously constructed prompts, and input the improved prompts together with the sentences to be labeled in the test set into the large model to obtain new outputs. The specific steps are as follows:
[0070] S41: training set D train Each sample in is taken as input, and steps S31 to S33 are executed to obtain the output set D output .
[0071] S42: Manually sorted output set D output The M and N entities with the most misidentifications corresponding to rock and mineral entities are used as negative sample sets. Here, M is 9 and N is 3. Misidentified entities refer to entities that do not originally belong to the class but are marked as such. At the same time, modifications that are easily added or omitted when identifying rock entities are sorted out. Thus, a guidance prompt based on error analysis is constructed. error_based .
[0072] S43: Task Description Prompt task_desc Prompt with annotation rules label_rule , Guidance Prompt based on error analysis error_based And the example set D demonstration It is concatenated with the input Input and fed into the large language model to obtain the new output NewOutput. The formula is as follows:
[0073] NewOutput=LLM(Prompt task_desc ,Prompt label_rule ,Prompt error_based ,D demonstration ,Input)
[0074] Among them, the guidance prompt based on error analysis error_based The content is as follows:
[0075] Typical entities that are easily misidentified as rocks are as follows: rock mass, rock vein, rock group, rock belt, rock, andesite, stratum (XXX group, XXX group, XXX series), subzone, pituitary
[0076] Typical entities that are easily misidentified as minerals are as follows: mineral, Fe, FeO
[0077] The following modifiers are not considered part of the rock entity: location (the rock entity in "Nabanaiga Diorite" should be "Diorite"), stratigraphy (the rock entity in "Shenla Formation Volcanic Rock" should be "Volcanic Rock"), sedimentary facies (the rock entity in "Lagoonary Sand Shale" should be "Sand Shale")
[0078] The following modifiers should be considered part of the rock entity: color ("dark gray" as part of "dark gray calcareous shale"), minor minerals ("dacite" as part of "dacite crystal fragment ignimbrite"), texture ("fine-grained" as part of "medium-thin-bedded fine-grained dolomite"), and structure ("medium-thin-bedded" as part of "brown medium-thin-bedded silty slate")
[0079] The following is an example input Input and a new output NewOutput:
[0080] Input: "The Zhuchawomeng dike group usually does not contain quartz. Some plutons (Qilongwulong and Malgongnan) and most dikes have porphyritic textures, with phenocrysts being plagioclase."
[0081] New Output: "The Zhuchawomeng and other dike groups usually do not contain [quartz]. Some rock bodies (Qilongwulong and Malgonan) and most dikes have porphyritic textures, and the phenocrysts are [plagioclase]."
[0082] S5: Constructing the hints for secondary category verification and inputting the hints together with the extracted rock and mineral entities into the large model so that it outputs the final recognition result;
[0083] S51: According to the marking rules Prompt label_rule Extract various geological entities Entities in the new output NewOutput types , where types contains two categories: rock and mineral. For example, for the new output NewOutput:
[0084] "The main lithologies of the lower part are <andesite>, <brecciated andesite>, and a small amount of <pyroclastic rock>. The main lithologies of the upper part are <andesitic agglomerate lava>, (diopside) <andesite>, <dacite>, and a small amount of <sedimentary pyroclastic rock>."
[0085] The rock entities extracted are: andesite, brecciated andesite, pyroclastic rock, andesitic agglomerated lava, andesite, dacite, and sedimentary pyroclastic rock.
[0086] The extracted mineral entities are: diopside
[0087] S52: Construct task description VerifyPrompt based on the task requirement of secondary verification of a certain entity type task_desc , and the output format requires VerifyPrompt format_desc Secondary category verification is to verify the type of the recognized entity again, requiring one type from all types and "other" as the final type of the entity. Adding the "other" type can effectively reduce the error of misidentifying non-entities as entities of a certain type. The content is as follows:
[0088] "You are required to identify the category of each entity. The category must be one of "Rock", "Mineral", or "Other". The answer must be the category of each entity, separated by commas."
[0089] S53: VerifyPrompt the task description task_desc and output format requirements VerifyPrompt format_desc And various entities to be verified types The verification result VerifyOutput is obtained by splicing and inputting the large language model. The formula is as follows:
[0090] VerifyOutput=LLM(VerifyPrompt task_desc ,VerifyPrompt format_desc ,Entities types ).
[0091] For example, for various entities to be verified types :Andesite, brecciated andesite, pyroclastic rock, andesitic agglomerate lava, andesite, dacite, sedimentary pyroclastic rock, Anshan Group, diopside
[0092] Verification resultVerifyOutput: rock, rock, rock, rock, rock, rock, rock, other, mineral;
[0093] S6: Use the test data set to evaluate the final recognition results of the model. The specific steps are as follows:
[0094] S61: NewOutput is obtained for the input sentence Input, thereby obtaining the verification result VerifyOutput, based on which the final recognition result finalOutput is obtained, and it is compared with the manual annotation result Annotation;
[0095] For example, for VerifyOutput: rock, rock, rock, rock, rock, rock, rock, other, mineral,
[0096] The finalOutput is: Rocks: andesite, brecciated andesite, pyroclastic rock, andesitic agglomerate lava, andesite, dacite, sedimentary pyroclastic rock; Minerals: diopside
[0097] S62: Calculate the results for rock and mineral classes separately. For example, for rock: if the rock class in the finalOutput contains an entity, but the rock class in the annotation does not contain the entity, then the number of false positives fp increases by one; if the rock class in the annotation contains the entity, then the number of true positives tp increases by one; if the rock class in the annotation contains the entity, but the rock class in the finalOutput does not contain the entity, then the number of false negatives fn increases by one.
[0098] S63: For rock and mineral classes, record the fp, fn, and tp of all samples in the test data set as input, and sum them to obtain FP, TP, and FN. The formula is as follows:
[0099]
[0100] Where TestNum is the total number of test set samples, which is 100 here, and fp i is the number of false positive examples corresponding to the i-th input sentence Input. The calculation of TP and FN is similar.
[0101] S64: Calculate the evaluation indicators of rock and mineral categories, Precision, Recall and F-score respectively. The formulas are as follows:
[0102]
[0103] Precision indicates how many of the samples predicted as positive are actually positive samples, and recall indicates how many of the positive examples in the samples are correctly predicted. F-score is a comprehensive evaluation indicator of these two indicators. The larger the value, the better the effect.
[0104] Table 1: Results of rock and mineral identification by various models
[0105]
[0106] The experimental results are shown in Table 1 above. The GPT3.5+ system and the GPT-4o+ system refer to models that use the GPT-3.5-turbo and GPT-4o versions of the GPT model, respectively, and are tested according to the specific steps of the system of the present invention. The GPT-3.5 and GPT-4o models use the GPT-3.5-turbo and GPT-4o versions of the GPT model, respectively, and are fed with the following prompts, where {TEXT} refers to a sentence containing rock or mineral entities that the model needs to recognize:
[0107] "You are required to identify all rock entities and mineral entities in the given sentence. The answer format is: Rock entities: entity 1, entity 2; Mineral entities: entity 1, entity 2. If there are none, answer: Rock entities: none; Mineral entities: none.
[0108] Sentence: {TEXT}"
[0109] The results in Table 2 show that: (1) Compared with GPT-3.5, the GPT-3.5+ system has an approximately 30% improvement in rock recognition precision, an approximately 3% improvement in recall, and an approximately 15% improvement in F1-score; the GPT-3.5+ system has an approximately 28% improvement in mineral recognition precision, an approximately 6% reduction in recall, and an approximately 5% improvement in F1-score. (2) Compared with GPT-4o, the GPT-4o+ system has an approximately 23% improvement in rock recognition precision, an approximately 10% improvement in recall, and an approximately 17% improvement in F1-score; the GPT-4o system has an approximately 10% improvement in mineral recognition precision, an approximately 6% reduction in recall, and an approximately 8% improvement in F1-score.
[0110] like Figures 3 to 6 As shown in the figure, for different versions of the GPT model, the method of the system of the present invention can improve the recognition accuracy and effect of rocks and mineral entities. When the effect is the best, the F1-score of rock recognition is improved by about 29%, and the F1-score of mineral recognition is improved by about 27%. This result strongly proves the effectiveness of the system.
[0111] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.
Claims
1. A large language model geological named entity recognition method based on prompt learning, characterized in that: The following steps are involved: S1. Select sentences containing geological entities from geological reports, construct a dataset, divide it into a test set and a training set according to a certain ratio, and manually annotate the test set; S2. Construct a prompt containing the task description, labeling rules, and demonstration examples. Input the prompt and a random sentence from the training set into the large model to make it output the labeling result. S3: sort out the entities that are most frequently mislabeled and inaccurately identified by the large model, select relevant geological domain knowledge based on the sorted results and integrate it into the prompt for improvement, and then input the improved prompt together with a random sentence in the test set into the large model to obtain a new output; S4. Construct a prompt for secondary category verification, extract various entities from the new output obtained in step S3, and make the large model select a category as the category of the entity in the given category, thereby obtaining the final recognition result; S5. Use the test dataset to evaluate the final recognition results of the large model.
2. The method for geological named entity recognition based on a large language model and prompt learning according to claim 1, wherein: Step S1 further includes the following steps: S11. Obtain several regional geological report documents; S12, using periods as delimiters, decompose the document into several sentence files, and manually screen out sentences that may contain various geological entities to form a data set; S13. Use the dataset as the test set D test , and take N samples from it to form the training set D train , and the test set D test Perform manual annotation.
3. The method for geological named entity recognition based on a large language model and prompt learning according to claim 1 is characterized by: Step S2 includes the following steps: S21. Construct a task description prompt based on the task requirements of marking each category of geological entities in the input sentence with different specific symbols. task_desc and annotation rules Prompt label_rule ; S22. Construct K sentences containing various geological entities as example samples D demo , and follow the annotation rules Prompt label_rule , manually mark out the various geological entities in these K sentences and obtain the labeled dataset D labeled , thus obtaining the example set D demonstration , the formula is: S23, Task Description Prompt task_desc Prompt with annotation rules label_rule And the example set D demonstration It is concatenated with the input and fed into the large language model to obtain the output. The formula is: Output=LLM(Prompt task_desc ,Prompt label_rule ,D demonstration ,Input)。 4. The method for geological named entity recognition based on large language model and prompt learning according to claim 3 is characterized in that: Step S3 further includes the following steps: S31: Take each sentence in the training set as input, execute steps S21 to S23, and get the output set D output ; S32: Manually sorted output set D output The M entities with the most misidentifications for each type of geological entity are used as negative sample sets, where misidentified entities refer to entities that do not originally belong to the class but are marked as such. At the same time, the easily missed and over-added modifications corresponding to the entity classes that are easy to identify inaccurately are sorted out, and a guidance prompt based on error analysis is constructed. error_based ; S33: Task Description Prompt task_desc Prompt with annotation rules label_rule , Guidance Prompt based on error analysis error_based And the example set D demonstration It is concatenated with the input and fed into the large language model to obtain the new output NewOutput. The formula is: NewOutput=LLM(Prompt task_desc ,Prompt label_rule ,Prompt error_based ,D demonstration ,Input)。 5. The method for geological named entity recognition based on a large language model and prompt learning according to claim 1, wherein: Step S4 further includes the following steps: S41, according to the marking rules Prompt label_rule Extract various geological entities Entities in the new output NewOutput types , where types is all categories; S42. Construct the task description VerifyPrompt based on the task requirement of verifying whether an entity belongs to a certain category. task_desc , and the output format requires VerifyPrompt format_desc ; S43, set the task description to VerifyPrompt task_desc and output format requirements VerifyPrompt format_desc And various entities to be verified types The verification result VerifyOutput is obtained by splicing and inputting the large language model. The formula is: VerifyOutput=LLM(VerifyPrompt task_desc ,VerifyPrompt format_desc ,Entities types )。 6. The method for geological named entity recognition based on a large language model and prompt learning according to claim 1, wherein: Step S5 further includes the following steps: S51, obtain NewOutput for the input sentence Input, thereby obtaining the verification result VerifyOutput, obtain the final recognition result finalOutput based on the verification result, and compare the final recognition result with the manual annotation result Annotation; S52. Compare each class separately: when the finalOutput contains an entity in a class, but the class in the annotation does not contain the entity, the number of false positive examples fp increases by one; when the class in the annotation contains the entity, the number of true positive examples tp increases by one; when the class in the annotation contains the entity, but the class in the finalOutput does not contain the entity, the number of false negative examples fn increases by one; S53. For each geological category, record the fp, fn, and tp of all samples in the test dataset as the input sentence Input, and sum them to obtain FP, TP, and FN. The formula is: Where TestNum is the total number of test set samples, fp i is the number of false positive examples corresponding to the i-th input sentence Input, tp i is the number of true examples corresponding to the i-th input sentence Input, fn i is the number of false negative examples corresponding to the i-th input sentence Input, S54. For each geological category, calculate the evaluation indicators, Precision, Recall, and F-score. The formula is: Among them, Precision represents the proportion of actual positive samples among samples predicted to be positive, Recall represents the proportion of positive examples in samples that are predicted correctly, and F-score is a comprehensive evaluation indicator of precision and recall.