Big language model geologic named entity recognition method based on prompt learning

Abstract

The invention provides a big language model geologic named entity recognition method based on prompt learning. The method comprises the following steps: constructing a data set composed of sentences containing geologic entities; constructing prompts aiming at the task of simultaneously labeling the geological entities, wherein the prompts comprise a labeling rule, an example guide part and the like; arranging errors in an output result of the large language model, and analyzing and constructing a prompt of a tagged geological entity fused with geological knowledge according to the errors; various geological entities in the output result are extracted; and constructing a prompt for entity category verification, and performing secondary category verification on the extracted entity. According to the method, an entity extraction task is converted into various entity tasks marked by using different symbols at the same time, guidance of geological knowledge is added according to error analysis, entity category verification is added, one category is selected from a plurality of given categories to serve as an entity category, and the entity category is extracted. The illusion problem of a large model, the geological field nested entity recognition error problem and the geological field knowledge lacking problem are effectively relieved, and the geological entity recognition efficiency and accuracy are improved.

Description

Technical Field

[0001] The invention relates to the field of geological text recognition, and in particular to a large language model geological named entity recognition method based on prompt learning. Background Art

[0002] In the research and practice of geology, geological named entity recognition is a basic and key link. Geological entity types include rocks, minerals, geological structures, etc., which often appear in geological reports. Accurately identifying and classifying these entities is of great significance for analyzing geological reports and data mining. However, accurately identifying named entities in geological reports faces huge challenges. Geological report texts are usually highly professional and have complex and diverse terms. Traditional named entity recognition methods are difficult to meet the needs of efficient and accurate recognition.

[0003] At present, the application of intelligent technology in geological named entity recognition has broad prospects and urgent needs. Domestic and foreign scholars have conducted a lot of research on improving the accuracy of named entity recognition. For example, although rule-based and statistical methods have made progress in some specific applications, they still have problems such as difficult maintenance and poor generalization ability. Machine learning-based methods, such as conditional random fields (CRFs) and traditional deep learning models, have improved, but they rely on verification of large-scale annotated data, have high annotation costs, and are difficult to adapt to the ever-changing new knowledge in the geological field. In recent years, with the development of large language models (such as BERT and GPT), these models have performed well in natural language processing tasks. However, the direct application of large language models to geological named entity recognition still faces the problems of insufficient professional knowledge and poor recognition. For example, some studies have tried to apply BERT to entity recognition in geological reports, but the effect is limited when dealing with small sample data and complex sentences. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a large language model named entity recognition method based on prompt learning, comprising the steps of:

[0005] S1: Select sentences containing geological entities from geological reports, construct a dataset, divide it into a test set and a training set according to a certain ratio, and manually annotate the test set;

[0006] S2: Construct a prompt containing a task description, labeling rules, and demonstration examples, and input the prompt and a random sentence in the training set into the large model to make it output the labeling result;

[0007] S3: Sort out the entities that are most often mislabeled and inaccurately recognized by the big model for various geological entities. Based on these errors, selectively integrate some geological domain knowledge into the previous prompts, and input the improved prompts together with a random sentence in the test set into the big model to obtain new outputs.

[0008] S4: Construct a prompt for secondary category verification, extract various entities from the annotation results, and let the large model select one category from several given categories as the category of the entity, so as to obtain the final recognition result;

[0009] S5: Use the test data set to evaluate the final recognition results of the model.

[0010] Furthermore, the specific method of step S1 is as follows:

[0011] S11: Obtain multiple regional geological report document datasets;

[0012] S12: Using the period as a separator, the document is decomposed into multiple sentence files. And sentences that may contain various geological entities are manually screened to form a data set;

[0013] S13: Use the dataset as the test set D test , and take N samples from it to form the training set D train , and on the test set D test Perform manual annotation.

[0014] In the preferred solution, the specific method of step S2 is as follows:

[0015] S21: Construct task description prompts based on task requirements of different categories of geological entities in the input sentence marked with different specific symbols task_desc and annotation rule prompt label_rule ;

[0016] S22: Construct K sentences containing various geological entities as example samples D demo , and follow the annotation rules Prompt label_rule , manually annotate the various geological entities in these K sentences and obtain the annotated dataset D labeled , thus obtaining the example set D demonstration , the formula is as follows:

[0017]

[0018] S23: Task Description Prompt task_desc Prompt with annotation rules label_rule And the example set D demonstrationIt is concatenated with the input and fed into the large language model to get the output. The formula is as follows:

[0019] Output = LLM (Prompt task_desc ,Prompt label_rule ,D demonstration ,Input).

[0020] In the preferred solution, the specific method of step S3 is as follows:

[0021] S31: Take each sentence in the training set as input, execute steps S21 to S23, and get the output set D output ;

[0022] S32: Manually sorted output set D output The M entities with the most misidentifications for each type of geological entity are used as negative sample sets, where misidentified entities refer to entities that do not originally belong to the class but are marked as such. At the same time, the easily missed and over-added modifications corresponding to the entity categories that are easy to identify inaccurately are sorted out. Construct a guidance prompt based on error analysis error_based ;

[0023] S33: Task Description Prompt task_desc Prompt with annotation rules label_rule , Guidance Prompt based on error analysis error_based And the example set D demonstration It is concatenated with the input Input and input into the large language model to obtain the new output NewOutput. The formula is as follows:

[0024] NewOutput = LLM(Prompt task_desc ,Prompt label_rule ,Prompt error_based ,D demonstration ,Input).

[0025] In the preferred solution, the specific method of step S4 is as follows:

[0026] S41: According to the annotation rules Prompt label_rule Extract various geological entities Entities in the new output NewOutput types , where types are all categories;

[0027] S42: Construct task description VerifyPrompt based on task requirements for secondary verification of an entity type task_desc , and the output format requires VerifyPrompt format_desc,The task requires to select one from all types and "others" as the final type of the entity. Adding the type of "others" can effectively reduce the misidentification of non-entities as entities of a certain type;

[0028] S43: VerifyPrompt task description task_desc and output format requires VerifyPrompt format_desc And various entities to be verified types Concatenate and input the large language model to get the verification result VerifyOutput. The formula is as follows:

[0029] VerifyOutput=LLM(VerifyPrompt task_desc ,VerifyPrompt format_desc ,Entities types ).

[0030] In the preferred solution, the specific method of step S5 is as follows:

[0031] S51: NewOutput is obtained for the input sentence Input, thereby obtaining the verification result VerifyOutput, and the final recognition result finalOutput is obtained based on it, which is compared with the manual annotation result Annotation;

[0032] S52: Compare each class separately. If the finalOutput contains an entity in a class, but the class in the Annotation does not contain the entity, the number of false positive examples fp increases by one. If the class in the Annotation contains the entity, the number of true positive examples tp increases by one. If the class in the Annotation contains the entity but the class in the finalOutput does not contain the entity, the number of false negative examples fn increases by one.

[0033] S53: For each geological category, record the fp, fn and tp of all samples in the test data set as the input sentence Input, and sum them up to get FP, TP and FN. The formula is as follows:

[0034]

[0035] Where TestNum is the total number of test set samples, fp i is the number of false positive examples corresponding to the i-th input sentence Input. The calculation of TP and FN is similar to it;

[0036] S54: For each geological category, calculate the evaluation indicators, precision, recall and F-score. The formula is as follows:

[0037]

[0038]

[0039]

[0040] Among them, Precision indicates how many of the samples predicted as positive are truly positive samples, Recall indicates how many positive examples in the samples are predicted correctly, and F-score is a comprehensive evaluation indicator of these two indicators. The larger the value, the better the effect.

[0041] Beneficial effects: The present invention converts the entity extraction task into an entity labeling task, identifies various geological entities at one time, and each type of entity is labeled with different symbols, such as angle brackets, etc., which is conducive to alleviating the problem of incorrect identification of entity categories caused by the large model hallucination problem and the problem of repeated identification of nested entities; selects a part from the test set for preliminary identification, sorts out common errors in the identification results, and increases geological knowledge based on these errors, thereby reducing the problem of incorrect identification caused by lack of geological knowledge; adds secondary category verification, requiring one category to be selected from several given categories as the category of the entity, further reducing the error recognition rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0043] Figure 2 This is an example diagram of the BIOES annotation method;

[0044] Figure 3 This is a screenshot of the recognition indicator results of GPT-3.5;

[0045] Figure 4 This is a screenshot of the recognition indicator results of the GPT-3.5+ method;

[0046] Figure 5 This is a screenshot of GPT-4o's recognition indicator results;

[0047] Figure 6 This is a screenshot of the recognition indicator results of the GPT-4o+ method. DETAILED DESCRIPTION

[0048] like Figures 1-2 As shown, a large language model geological named entity recognition method based on prompt learning disclosed in an embodiment of the present invention comprises the following steps:

[0049] S1: Make a dataset. This example dataset comes from geological report texts, and mainly includes the survey report of Nima District, the regional geological survey report of Zaduo County, the regional geological survey report of Wanyangchun City, Guangdong, and the regional geological survey report of Jinniu Town and Gaoqiao. Use a period as a delimiter to convert a document file into multiple txt files consisting of a single sentence. Manually select sentence files containing rock entities or mineral entities to form a dataset.

[0050] S2: Select 200 samples from the dataset as the test dataset D test , randomly select 100 samples from them as the training set D train For the test set D test , manually annotate the rock entities and mineral entities in the sentence. Using the BIOES annotation method, the beginning of the entity segment (Begin), the middle of the entity segment (Inside), the end of the entity segment (End), the single-word entity (Single), and the non-entity (Other) are indicated respectively. Figure 2 An example of annotation is shown.

[0051] S3: Construct a prompt containing task description, annotation rules, and demonstration examples, and compare the prompt with the training set D train The sentence to be recognized is input into the big model together with the Input, so that it can output the labeled sentence. The specific steps are as follows:

[0052] S31: Construct a task description prompt based on the task requirement of annotating the rock and mineral entities in the input sentence according to the annotation rules task_desc and annotation rule prompt label_rule , where Prompt task_desc The contents are as follows:

[0053] "You need to label all the rock entities and mineral entities in the given sentence."

[0054] Annotation Rules Prompt label_rule The content is as follows:

[0055] "The marking rules are as follows: 1. Use angle brackets '<' and '>' to mark rock entities. 2. Use square brackets '[' and ']' to mark mineral entities. 3. If rock and mineral entities are not identified, the original sentence is output directly."

[0056] S32: Construct K sentences containing both rock and mineral entities as example samples D demo , here K value is 2, and according to the annotation rules Prompt label_rule , manually annotate the rock and mineral entities in these K sentences and obtain the annotated dataset Dlabeled , thus obtaining the example set D demonstration , the formula is as follows:

[0057]

[0058] Example Set D demonstration Specifically:

[0059] “Here are some examples:

[0060] Example 1:

[0061] Input: The main rock types in the area are quartz monzonite and granite, and common minerals include quartz and feldspar.

[0062] Output: The main rock types in the area are <Quartz Monzonite> and <Granite>, and common minerals include [Quartz] and [Feldspar].

[0063] Example 2:

[0064] Input: Basalt is a common igneous rock containing minerals such as olivine and pyroxene.

[0065] Output: <Basalt> is a common igneous rock containing minerals such as [olivine] and [pyroxene]. "

[0066] S33: Task Description Prompt task_desc Prompt with annotation rules label_rule And the example set D demonstration It is concatenated with the input and fed into the large language model to get the output. The formula is as follows:

[0067] Output = LLM (Prompt task_desc ,Prompt label_rule ,D demonstration ,Input)

[0068] The following is a sample input and output:

[0069] Input: "The Zhuchawomeng and other dyke groups usually do not contain quartz. Some rock bodies (Qilongwulong and Malgonan) and most dykes have porphyritic structures, and the phenocrysts are plagioclase."

[0070] Output: "Zhuchawomeng and other <vein groups> usually do not contain [quartz], some <rock bodies> (Qilongwulong, Malgonan) and most <veins> have porphyritic structures, and the phenocrysts are [plagioclase]"

[0071] S4: Refine the large model For the most frequently mislabeled rock entities and mineral entities, add these entities to the previously constructed prompts, and input the improved prompts together with the sentences to be labeled in the test set into the large model to obtain new outputs. The specific steps are as follows:

[0072] S41: Training set D train Each sample in is taken as input Input, and steps S31 to S33 are executed to obtain the output set D output .

[0073] S42: Manually sorted output set D output The M and N entities with the most misidentifications corresponding to the rock and mineral entities are used as negative sample sets. Here, M is 9 and N is 3. Misidentified entities refer to entities that do not originally belong to the category but are marked as such. At the same time, modifications that are easily added or omitted when identifying rock entities are sorted out. In this way, a guidance prompt based on error analysis is constructed. error_based .

[0074] S43: Task Description Prompt task_desc Prompt with annotation rules label_rule , Guidance Prompt based on error analysis error_based And the example set D demonstration It is concatenated with the input Input and input into the large language model to obtain the new output NewOutput. The formula is as follows:

[0075] NewOutput = LLM(Prompt task_desc ,Prompt label_rule ,Prompt error_based ,D demonstration ,Input)

[0076] Among them, the guidance prompt based on error analysis error_based The content is as follows:

[0077] "Typical entities that are easily misidentified as rocks are as follows: rock mass, rock vein, rock group, rock belt, rock, andesite, stratum (XXX group, XXX group, XXX series), subzone, pituitary

[0078] Typical entities that are easily misidentified as minerals are as follows: mineral, Fe, FeO

[0079] The following modifiers are not considered part of the rock entity: location (the rock entity in "Nabanaiga diorite" should be "diorite"), stratigraphy (the rock entity in "Shenla Formation volcanic rock" should be "volcanic rock"), sedimentary facies (the rock entity in "lagoonal sandstone shale" should be "sandstone shale")

[0080] The following modifiers should be considered as part of the rock entity: color ("dark gray" as part of "dark gray calcareous shale"), minor minerals ("dacite" as part of "dacite crystal fragment ignimbrite"), texture ("fine-grained" as part of "medium-thin-bedded fine-grained dolomite"), structure ("medium-thin-bedded" as part of "brown medium-thin-bedded silty slate")

[0081] The following is an example of an input Input and a new output NewOutput:

[0082] Input: "The Zhuchawomeng and other dyke groups usually do not contain quartz. Some rock bodies (Qilongwulong and Malgonan) and most dykes have porphyritic structures, and the phenocrysts are plagioclase."

[0083] New output: "Zhuchawomeng and other dyke groups usually do not contain [quartz], and some rock bodies (Qilongwulong, Malgonan) and most dykes have porphyritic structures, and the phenocrysts are [plagioclase]"

[0084] S5: constructing the prompts for secondary category verification, and inputting the prompts together with the extracted rock and mineral entities into the large model so that its output is the final recognition result;

[0085] S51: According to the annotation rules Prompt label_rule Extract various geological entities Entities in the new output NewOutput types , where types contains two categories: rock and mineral. For example, for the new output NewOutput:

[0086] "The main lithologies of the lower part are <andesite>, <brecciated andesite>, and a small amount of <pyroclastic rock>, and the main lithologies of the upper part are <andesite agglomerate lava>, ([diopside]) <andesite>, <dacite>, and a small amount of <sedimentary pyroclastic rock>."

[0087] The rock entities extracted are: andesite, brecciated andesite, pyroclastic rock, andesitic agglomerate lava, andesite, dacite, and sedimentary pyroclastic rock.

[0088] The extracted mineral entities are: Diopside

[0089] S52: Construct task description VerifyPrompt based on the task requirement of secondary verification of a certain entity type task_desc , and the output format requires VerifyPrompt format_desc Secondary category verification is to verify the type of the recognized entity again, requiring one to be selected from all types and "other" as the final type of the entity. Adding the "other" type can effectively reduce the misidentification of non-entities as entities of a certain type. The content is as follows:

[0090] "You are required to identify the category of each entity. The category must be any one of the three: "Rock", "Mineral", and "Other". The answer must be the category of each entity, connected by commas."

[0091] S53: VerifyPrompt the task description task_desc and output format requires VerifyPrompt format_desc And various entities to be verified types Concatenate and input the large language model to get the verification result VerifyOutput. The formula is as follows:

[0092] VerifyOutput=LLM(VerifyPrompt task_desc ,VerifyPrompt format_desc ,Entities types ).

[0093] For example, for various entities to be verified types : andesite, brecciated andesite, pyroclastic rock, andesitic agglomerate lava, andesite, dacite, sedimentary pyroclastic rock, Anshan Group, diopside

[0094] VerifyResultVerifyOutput: rock, rock, rock, rock, rock, rock, rock, other, mineral;

[0095] S6: Use the test data set to evaluate the final recognition results of the model. The specific steps are as follows:

[0096] S61: NewOutput is obtained for the input sentence Input, thereby obtaining the verification result VerifyOutput, and the final recognition result finalOutput is obtained based on it, which is compared with the manual annotation result Annotation;

[0097] For example, for VerifyOutput: rock, rock, rock, rock, rock, rock, rock, other, mineral,

[0098] The finalOutput is: Rocks: andesite, brecciated andesite, pyroclastic rock, andesitic agglomerate lava, andesite, dacite, sedimentary pyroclastic rock; Minerals: diopside

[0099] S62: Calculate the results of rock and mineral separately. For example, for rock: if the rock in finalOutput contains an entity, but the rock in Annotation does not contain the entity, then the number of false positive examples fp increases by one; if the rock in Annotation contains the entity, then the number of true positive examples tp increases by one; if the rock in Annotation contains the entity, but the rock in finalOutput does not contain the entity, then the number of false negative examples fn increases by one.

[0100] S63: For rock and mineral types, record the fp, fn and tp of all samples in the test data set as input respectively, and sum them to get FP, TP and FN. The formula is as follows:

[0101]

[0102] Where TestNum is the total number of samples in the test set, which is 100 here, and fp i is the number of false positive examples corresponding to the i-th input sentence Input. The calculation of TP and FN is similar.

[0103] S64: Calculate the evaluation indicators of rock and mineral categories, precision, recall and F-score respectively. The formulas are as follows:

[0104]

[0105]

[0106]

[0107] Among them, Precision indicates how many of the samples predicted as positive are truly positive samples, Recall indicates how many positive examples in the samples are predicted correctly, and F-score is a comprehensive evaluation indicator of these two indicators. The larger the value, the better the effect.

[0108] Table 1: Results of rock and mineral identification by various models

[0109]

[0110] The experimental results are shown in Table 1 above. The model GPT3.5+ system and GPT-4o+ system refer to the models that use the GPT models of gpt-3.5-turbo and gpt-4o versions respectively and conduct experiments according to the specific steps of the system of the present invention. The models GPT-3.5 and GPT-4o refer to the GPT models of gpt-3.5-turbo and gpt-4o versions respectively and input the following prompts to them, where {TEXT} refers to the sentence containing rock or mineral entities that need to be recognized by the model:

[0111] "You are required to identify all the rock entities and mineral entities in the given sentence. The answer format is: Rock entity: entity 1, entity 2; Mineral entity: entity 1, entity 2. If there is none, answer: Rock entity: none; Mineral entity: none.

[0112] Sentence: {TEXT}"

[0113] From the results in Table 2, we can see that: (1) Compared with GPT-3.5, the precision of rock recognition by the GPT-3.5+ system is improved by about 30%, the recall rate is improved by about 3%, and the F1-score is improved by about 15%; the precision of mineral recognition is improved by about 28%, the recall rate is reduced by about 6%, and the F1-score is improved by about 5%. (2) Compared with GPT-4o, the precision of rock recognition by the GPT-4o+ system is improved by about 23%, the recall rate is improved by about 10%, and the F1-score is improved by about 17%; the precision of mineral recognition is improved by about 10%, the recall rate is reduced by about 6%, and the F1-score is improved by about 8%.

[0114] like Figures 3 to 6 As shown, for different versions of the GPT model, the method of the system of the present invention can improve the recognition accuracy and effect of rocks and mineral entities. When the effect is the best, the F1-score of rock recognition is improved by about 29%, and the F1-score of mineral recognition is improved by about 27%. This result strongly proves the effectiveness of the system.

[0115] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large language model geological named entity recognition method based on prompt learning, characterized in that: The following steps are involved: S1. Select sentences containing geological entities from geological reports, construct a data set, divide it into a test set and a training set according to a certain ratio, and manually annotate the test set; S2, construct a prompt containing a task description, labeling rules, and demonstration examples, and input the prompt and a random sentence in the training set into the large model to make it output the labeling result; S3, sort out the entities that are most often mislabeled and inaccurately identified in the large model, select the corresponding geological field knowledge based on the sorted results and integrate them into the prompts for improvement, and input the improved prompts together with a random sentence in the test set into the large model to obtain a new output; S4, constructing a prompt for secondary category verification, extracting various entities in the annotation results, and making the large model select a category from the given categories as the category of the entity, thereby obtaining the final recognition result; S5. Use the test data set to evaluate the final recognition results of the large model.

2. According to claim 1, a large language model geological named entity recognition method based on prompt learning is characterized by: Step S1 also includes the following steps: S11. Obtain several regional geological report documents; S12, using the period as a delimiter, decomposing the document into several sentence files, and manually screening out sentences that may contain various geological entities to form a data set; S13. Use the data set as the test set D test , and take N samples from it to form the training set D train , and on the test set D test Perform manual annotation.

3. The method for geological named entity recognition based on large language model and prompt learning according to claim 1 is characterized in that: Step S2 includes the following steps: S21. Construct task description prompts based on the task requirements of marking various types of geological entities in the input sentence with different specific symbols. task_desc and annotation rule prompt label_rule ; S22. Construct K sentences containing various geological entities as example samples D demo , and follow the annotation rules Prompt label_rule , manually annotate the various geological entities in these K sentences and obtain the annotated dataset D labeled , thus obtaining the example set D demonstration , the formula is: S23. Prompt the task description task_desc Prompt with annotation rules label_rule And the example set D demonstration It is concatenated with the input and input into the large language model to get the output Output. The formula is: Output=LLM(Prompt task_desc ,Prompt label_rule ,D demonstration ,Input)。 4. The method for geological named entity recognition based on large language model and prompt learning according to claim 3 is characterized in that: Step S3 also includes the following steps: S31: Take each sentence in the training set as input, execute steps S21 to S23, and get the output set D output ; S32: Manually sorted output set D output The M entities with the most misidentifications corresponding to each type of geological entity in the dataset are taken as negative sample sets, where misidentified entities refer to entities that do not originally belong to the class but are marked as such. At the same time, the easily missed and over-added modifications corresponding to the easily recognized inaccurate entity categories are sorted out to construct a guidance prompt based on error analysis. error_based ; S33: Task Description Prompt task_desc Prompt with annotation rules label_rule , Guidance Prompt based on error analysis error_based And the example set D demonstration It is concatenated with the input Input and input into the large language model to obtain the new output NewOutput. The formula is: NewOutput=LLM(Prompt task_desc ,Prompt label_rule ,Prompt error_based ,D demonstration ,Input)。 5. The method for geological named entity recognition using a large language model based on prompt learning according to claim 1, characterized in that: Step S4 also includes the following steps: S41. According to the marking rules Prompt label_rule Extract various geological entities Entities in the new output NewOutput types , where types are all categories; S42. Construct the task description VerifyPrompt according to the task requirement of verifying whether an entity belongs to a certain category task_desc , and the output format requires VerifyPrompt format_desc ; S43, VerifyPrompt task description task_desc and output format requires VerifyPrompt format_desc And various entities to be verified types Concatenate and input the large language model to get the verification result VerifyOutput, the formula is: VerifyOutput=LLM(VerifyPrompt task_desc ,VerifyPrompt format_desc ,Entities types )。 6. The method for geological named entity recognition using a large language model based on prompt learning according to claim 1, characterized in that: Step S5 also includes the following steps: S51, NewOutput is obtained for the input sentence Input, thereby obtaining a verification result VerifyOutput, and a final recognition result finalOutput is obtained according to the verification result, and the final recognition result is compared with the manual annotation result Annotation; S52. Compare each class separately: when the finalOutput contains an entity in a class, but the class in the Annotation does not contain the entity, the number of false positive examples fp increases by one; when the class in the Annotation contains the entity, the number of true positive examples tp increases by one; when the class in the Annotation contains the entity but the class in the finalOutput does not contain the entity, the number of false negative examples fn increases by one; S53. For each geological category, record the fp, fn and tp of all samples in the test data set as the input sentence Input, and sum them up to get FP, TP and FN. The formula is: Where TestNum is the total number of test set samples, fp i is the number of false positive examples corresponding to the i-th input sentence Input, tp i is the number of true examples corresponding to the i-th input sentence Input, fn i is the number of false negative examples corresponding to the i-th input sentence Input, S54. For each geological category, calculate the evaluation indicators, precision, recall and F-score. The formula is: Among them, precision Precision represents the proportion of actual positive samples among samples predicted to be positive, recall Recall represents the proportion of positive examples predicted correctly in samples, and F-score is a comprehensive evaluation indicator of precision and recall.

Images