Intelligent verification and correction method for named entity recognition based on large language model in financial field

Abstract

The invention discloses a named entity recognition intelligent verification and correction method based on a large language model in the financial field. In order to overcome the defects of an existing small model named entity recognition method, the invention provides a method based on the combination of a large language model and an NER small model in the financial field to correct a named entity recognition result, and the NER accuracy is improved. According to the technical scheme for verifying and correcting the result of the NER small model by using the financial field language large model, the accuracy and generalization ability of the financial NER can be remarkably improved, and efficient and reliable named entity recognition in a complex and changeable financial text is ensured.

Description

Technical Field

[0001] The present invention relates to an intelligent verification and correction method for named entity recognition based on a large language model in the financial field, belonging to the field of Internet technology. Background Art

[0002] Named Entity Recognition (NER) is an important task in Natural Language Processing (NLP), aiming to identify entities with specific meanings from text, such as person names, place names, organization names, etc. In the financial field, NER technology is widely used in multiple aspects such as document processing, risk assessment, compliance inspection, intelligent customer service, etc. For example, extracting key information such as company names, transaction amounts, dates, etc. from bank reports is crucial for improving work efficiency and accuracy.

[0003] Although traditional NER small models have made significant progress in general fields, there are still the following four problems when applied in the financial field:

[0004] 1) Lack of professional knowledge: Traditional NER small models are usually trained based on general corpora and lack professional knowledge in the financial field. Financial texts contain a large number of professional terms, industry-specific vocabulary, and complex sentence structures, which are not common in general corpora, resulting in poor generalization ability of the model in the financial field.

[0005] 2) Data sparsity: The labeled data in the financial field is relatively scarce and the acquisition cost is high. Traditional small models are difficult to learn sufficient features in the case of data sparsity, thus affecting the recognition accuracy.

[0006] 3) Difficulty in dealing with complex texts: Financial texts usually contain complex sentence structures and multi-layer nested entity relationships, and traditional small models are difficult to effectively capture these complexities, resulting in inaccurate recognition results.

[0007] 4) Poor domain adaptability: The financial field changes rapidly, and new terms and concepts emerge continuously. Traditional small models are difficult to quickly adapt to these changes and need to be updated and retrained frequently.

[0008] Currently, the following are the main methods for improving the accuracy of NER in the financial field:

[0009] 1) The Chinese invention patent with the publication number CN117113997A, which was published on November 24, 2023, adopts data augmentation or data fusion. By vectorizing or lexicalizing the data at the input layer, the information representation ability of the data is enhanced. This method relies on high-quality dictionaries and pre-trained models, and it is challenging to ensure the effective fusion of context information and dictionary representations. For entities not appearing in the dictionary, the recognition effect is not good.

[0010] 2) The Chinese patent application with the publication number CN118313382A, which was published on July 9, 2024, adopted model architecture improvement. By improving the model architecture, the accuracy of named entity recognition was enhanced. For example, a bidirectional recurrent neural network (LSTM or GRU) was used to capture context information; a conditional random field (CRF) layer was added to the output layer of the model to better handle sequence labeling tasks; an attention mechanism was introduced to enhance the model's attention to important features, etc. The above methods increased the model complexity, required additional data annotation, and the addition of algorithms such as the attention mechanism made the model less interpretable. Similarly, since the above are all supervised methods, when encountering unseen entities, the generalization ability of the model is poor.

[0011] 3) The Chinese patent with the publication number CN117436449A, which was published on January 23, 2024, adopted a third type of method to guide the model to learn correct entity labels through reinforcement learning or by setting up a suitable reward mechanism. By using reinforcement learning to discard low-quality annotations, the adaptability and recognition accuracy of the model were improved. The data preprocessing instance selector for reinforcement learning may be difficult to design and optimize, and it is difficult to effectively screen out low-quality annotations.

[0012] 4) The Chinese patent application with the publication number CN118153576A, which was published on June 7, 2024, first classified the text and then performed NER. First, the field to which the text belongs was determined, and then it was processed based on a pre-trained financial NER model in that field to output the target business field label and entity recognition label. This method requires prior classification of the financial text to be recognized, such as stock trading, credit assessment, etc., which increases the workload; then it is sent into different NER models, that is, this method requires multiple pre-trained NER models suitable for different financial fields to handle NER for different financial field texts, consuming a large amount of resources.

[0013] First, existing technical solutions rely on high-quality dictionaries and pre-trained models and have difficulties in ensuring the effective integration of context information with dictionary representations, resulting in poor entity recognition effects for entities not appearing in the dictionary. Second, although existing technical solutions using improved model architectures (such as using bidirectional LSTM / GRU, CRF layers, and attention mechanisms) can improve accuracy, they increase model complexity, require additional data annotation, and the introduction of algorithms such as attention mechanisms makes the model less interpretable and has limited generalization ability. In addition, although the reinforcement learning method adopted by existing technical solutions can improve the adaptability and recognition accuracy of the model by setting appropriate reward mechanisms, the design of its data preprocessing instance selector and the screening of low-quality annotations are difficult, and it is difficult to effectively screen out low-quality annotations. Moreover, the method of performing NER after text classification in existing technical solutions requires prior classification of the text (such as stock trading, credit assessment, etc.), increasing the workload and requiring pre-training of multiple NER models applicable to different financial fields, consuming a large amount of resources. Finally, the terms in the financial field are complex and constantly changing, the model needs to be continuously updated to adapt to new terms, and at the same time, the problem of data imbalance leads to weak entity recognition ability for some entities, high requirements for context understanding ability, and multilingual support is also a challenge. These technical problems jointly limit the accuracy and generalization ability of existing financial NER technologies. Summary of the Invention

[0014] The technical problem to be solved by the present invention is that existing small models have poor recognition effects and insufficient generalization ability in the field of financial NER. Specifically: they rely on high-quality dictionaries and pre-trained models and have poor recognition of entities that do not appear; although improving the model architecture improves accuracy, it increases complexity and data requirements and has poor interpretability; the reinforcement learning method is difficult to screen out low-quality annotations; performing NER after text classification increases the workload and resource consumption.

[0015] To solve the above technical problems, the technical solution of the present invention is to disclose an intelligent verification and correction method for named entity recognition based on a large language model in the financial field, which is characterized by including the following steps:

[0016] Step 1, data construction to generate a large amount of annotated data;

[0017] Step 2, instruction adjustment is carried out under the guidance of the data constructed in Step 1 to enable the model to understand and execute specific task instructions. Through instruction adjustment, the general ability of the LLM is transformed into a specific ability focusing on financial field NER, so that the LLM can achieve higher accuracy and stronger generalization ability in the financial field NER task;

[0018] Step 3: Supervised Fine-tuning Based on the instruction tuning, by constructing a dataset containing various error types, the model learns how to identify and correct these errors. Meanwhile, leveraging the context understanding ability of the LLM, the accuracy of the model in dealing with ambiguous or vague entities is improved;

[0019] Step 4: Evaluate the fine-tuned LLM to ensure that the NER effect of the LLM in the financial field is significantly improved compared to other large language models;

[0020] Step 5: Conduct a deterministic evaluation of the NER results of the small model;

[0021] Step 6: The LLM verifies and corrects the results of the traditional NER small model;

[0022] Step 7: Feedback and Iteration:

[0023] Add the results corrected by the LLM to the training dataset of the NER small model for subsequent training and optimization. Through continuous iteration, improve the performance of the NER small model in the financial field.

[0024] Preferably, Step 1 includes the following steps:

[0025] Step 101: Extract text using a large corpus, where the text covers finance;

[0026] Step 102: Chunk the text collected in Step 101 into paragraphs with a maximum length of 256 tokens, and randomly sample several paragraphs as the input for the large language model;

[0027] Step 103: Use the large language model to generate entity mentions and their associated types based on the paragraphs obtained in Step 102;

[0028] Step 104: Clean the data obtained in Step 103, filtering out unparseable outputs and inapplicable entities;

[0029] Step 105: Classify the entities obtained in Step 105, count the top N entity types in each frequency range to construct the data and understand the entity type distribution, facilitating the subsequent instruction tuning process.

[0030] Preferably, in Step 103, fix the Temperature to 0 during the generation process to ensure the stability of the generated entity mentions and types.

[0031] Preferably, Step 2 specifically includes the following steps:

[0032] Step 201: Convert the data obtained in Step 1 into a conversational template, and each template contains:

[0033] A) System message: Prompt the LLM that it is a powerful information extraction system;

[0034] B) Prompt: Given a passage of text, prompt the LLM's task is to extract all entities and determine their entity types;

[0035] C) Output: The output is a list of tuples in the following format: [("Entity 1", "Type of Entity 1"), ……];

[0036] D) User: Make a natural language query based on the entity type;

[0037] E) Assistant: Generate a JSON list containing the corresponding entity mentions according to the user's query.

[0038] Step 202: Convert the entity type Ti of each entity that appears in the data obtained in Step 201 into a natural language query;

[0039] Step 203: Adjust the LLM to generate a structured output Yi in the form of a JSON containing all entities corresponding to Ti in the paragraph;

[0040] Step 204: Extract negative entity types as queries from the set of entity types that do not appear in the article, and set the expected output to an empty JSON.

[0041] Preferably, the specific steps of Step 3 are as follows:

[0042] Step 301: Construct a dataset for the three cases of entity type error, entity name error, and missing entity error;

[0043] Step 302: Manually check the above labels and convert them into natural language format;

[0044] Step 303: Use the LLM combined with the manually labeled dataset to fine-tune the LLM. Different hyperparameters can be considered to find the best fine-tuning strategy.

[0045] Preferably, in Step 301, the entity type error means that ambiguous entities are prone to type errors;

[0046] The entity name error means that there are extra symbols or the proper name cannot be accurately recognized;

[0047] The missing entity error means that the entities in the paragraph are not recognized.

[0048] Preferably, in Step 301, the specific method of constructing the dataset is: Input the original text and the previous recognition results into the LLM, let the LLM judge whether there are entity type errors, entity name errors or missing entity errors, and label the errors.

[0049] Preferably, step 4 specifically includes the following steps:

[0050] Step 401: Use strict entity-level micro F1 as the evaluation metric, requiring the entity type and boundaries to exactly match the ground truth;

[0051] Step 402: Compare the fine-tuned LLM with existing large models to evaluate its NER performance in the financial domain;

[0052] Step 403: Through ablation experiments, evaluate the contributions of instruction tuning and negative sampling in step 2 to the model performance, and at the same time evaluate the impacts of different fine-tuning strategies in step 3 on the model performance.

[0053] Preferably, step 5 specifically includes the following steps:

[0054] Step 501: Use the LLM for probability estimation: Input the entities identified by the NER small model and their context information into the LLM, and use the LLM to calculate the probability or confidence score of each entity identified by NER.

[0055] Step 502: Use the LLM for semantic consistency checking: Input the entities identified by the NER small model and their context information into the LLM, and use the LLM to evaluate the semantic consistency between the entities and the context.

[0056] Step 503: Combine the results of probability estimation and semantic consistency checking to obtain the final certainty score.

[0057] Preferably, step 6 specifically includes the following steps:

[0058] Step 601: Analyze the certainty scores obtained in step 5, set a threshold, and consider the recognition results below the threshold as uncertain, while the recognition results above the threshold can be directly accepted;

[0059] Step 602: For the recognition results with low certainty, the LLM will perform further verification:

[0060] First, utilize the semantic understanding ability of the LLM to verify whether the recognition results of the NER small model conform to the context semantics;

[0061] Then query the financial domain database to verify whether the recognition results of the NER small model are consistent with known facts;

[0062] Finally, apply the specific rules in the financial domain to check whether the recognition results of the NER small model conform to the rules;

[0063] Step 603: If the verification result indicates that the recognition result of the NER small model is incorrect, the LLM will provide a correction: The LLM will re-recognize the partial text of the context of the entity, and use the context information to adjust or modify the recognition result of the NER small model to make it more consistent with the context semantics.

[0064] Aiming at the deficiencies of the existing small model named entity recognition method, the present invention proposes a method combining a large language model in the financial field and a NER small model to correct the named entity recognition result and improve the NER accuracy. The technical solution proposed by the present invention, which uses a large language model in the financial field to verify and correct the result of the NER small model, can significantly improve the accuracy and generalization ability of financial NER, and ensure efficient and reliable named entity recognition in complex and changeable financial texts.

[0065] Compared with the existing technical solutions, the technical solution proposed by the present invention to use the LLM to verify and correct the recognition result of the traditional NER small model has high universality and accuracy, which is reflected in:

[0066] 1) Improve accuracy: Improve the accuracy of the NER task through the domain-adaptive LLM, especially in the financial field;

[0067] 2) Enhance robustness: When the certainty of the recognition result of the small model is low, the LLM can provide a correction to enhance the result robustness of the entire system;

[0068] 3) Improve professionalism: By combining the traditional NER small model and the LLM, the NER system can handle more complex professional field problems, and improve the accuracy and generalization of the system in the financial field.

[0069] Based on the technical solution proposed by the present invention, it has been put into use in the production environment, and the accuracy rate is about 95%. Brief Description of the Drawings

[0070] Figure 1 It is a flow chart of the technical solution of the present invention;

[0071] Figure 2 It is a flow chart of the LLM correcting the NER result of the small model proposed by the present invention. Detailed Embodiments

[0072] The following further elaborates the present invention in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

[0073] An intelligent verification and correction method for named entity recognition of large models in the financial field disclosed in the embodiments of the present invention specifically includes the following steps:

[0074] Step 1: Data construction. The role of data construction is to generate a large amount of labeled data, improve data diversity, and provide data for instruction adjustment. It further includes the following steps:

[0075] Step 101: Extract texts using a large corpus, including news reports, financial analyses, market analysis reports, etc., to ensure that the texts cover content in fields such as finance, and complete data collection.

[0076] Step 102: Divide the above texts into paragraphs with a maximum length of 256 tokens, and randomly sample 50,000 paragraphs as the input for the large language model.

[0077] Step 103: Use large language models such as ChatGPT to generate entity mentions and their associated types based on the paragraphs input above. During the generation process, fix the Temperature at 0 to ensure that the generated entity mentions and types are stable.

[0078] Step 104: Clean the above output, filtering out unparsable outputs and inapplicable entities, such as entities classified as "ELSE" like None, NA, MISC, and ELSE.

[0079] Step 105: Classify the above entities, count the top 10 entity types in each frequency range, to construct data and understand the entity type distribution, facilitating the subsequent instruction adjustment process.

[0080] Step 2: Instruction adjustment. The role of instruction adjustment is to enable the model to understand and execute specific task instructions. Through instruction adjustment, transform the general capabilities of the LLM into specific capabilities focused on NER in the financial field, so that the LLM can achieve higher accuracy and stronger generalization ability in the NER task in the financial field. It specifically includes the following steps:

[0081] Step 201: Convert the data obtained in Step 1 into a conversational template. Each template contains:

[0082] A) System message: Prompt the LLM that it is a powerful information extraction system;

[0083] B) Prompt: Given a passage of text, prompt the LLM that its task is to extract all entities and determine their entity types;

[0084] C) Output: The output is a list tuple in the following format: [("entity 1","type of entity 1"),……];

[0085] D) User: Conduct a natural language query based on entity types;

[0086] E) Assistant: Generate a JSON list containing corresponding entity mentions based on the user's query.

[0087] Step 202: Convert the type Ti of each entity that appears in the above output into a natural language query. For example, for the entity type "PERSON", the query could be "Which people are mentioned in the text?".

[0088] Step 203: Adjust the LLM to generate a structured output Yi in the form of a JSON containing all entities corresponding to Ti in the paragraph.

[0089] Step 204: Extract negative entity types as queries from the set of all entity types that do not appear in the article and set the expected output to an empty JSON.

[0090] Step 3: Supervised fine-tuning. Based on instruction tuning, by constructing a dataset containing various error types, the model can learn how to identify and correct these errors. At the same time, using the context understanding ability of the LLM, improve the accuracy of the model when dealing with ambiguous or vague entities. Specifically, it includes the following steps:

[0091] Step 301: Construct a dataset for three cases: entity type error, entity name error, and missing entity error. Specifically, for entity type error: Ambiguous entities are prone to type errors, such as misjudging "Chrome" as a company, etc.;

[0092] For entity name error: There are extra symbols or the proper name cannot be accurately recognized. For example, "FAW-Volkswagen" is recognized as "FAW-Volkswagen Volkswagen", "Baidu" is recognized as "Baidu Apollo", and "One Hundred Years of Solitude" is recognized as "One Hundred Years of Solitude》", etc.;

[0093] For missing entity error: The entity in the paragraph is not recognized, such as not recognizing "Huawei" in "Huawei is a company".

[0094] The specific method for constructing the dataset is: Input the original text and the previous recognition results into the LLM, let the LLM judge whether there are the above three types of errors, and label the errors.

[0095] Step 302: Manually check the above labels and convert them into natural language format.

[0096] Step 303: Use the above LLM combined with the manually annotated dataset to fine-tune the LLM. Different hyperparameters can be considered to find the best fine-tuning strategy.

[0097] Step 4: Evaluate the LLM after the above fine-tuning, which specifically includes the following steps:

[0098] Step 401: Use strict entity-level micro F1 as the evaluation metric, requiring that the entity types and boundaries match the ground truth exactly.

[0099] Step 402: Compare the above LLM with existing large models such as GhatGPT, Vicuna, InstructUIE, etc., and evaluate its NER effect in the financial field.

[0100] Step 403: Through ablation experiments, evaluate the contributions of components such as instruction tuning and negative sampling in Step 2 to the model performance, and at the same time evaluate the impacts of different fine-tuning strategies in Step 3 on the model performance.

[0101] After Step 4, ensure that the NER effect of the above LLM in the financial field is greatly improved compared with other large language models. Through Step 2, Step 3, and Step 4, an LLM with error correction ability and dedicated to NER in the financial field is obtained.

[0102] Step 5: Conduct a deterministic evaluation of the NER results of the small model, which specifically includes the following steps:

[0103] Step 501: Use the LLM for probability estimation: Input the entities identified by the NER small model and their context information into the LLM, and use the LLM to calculate the probability or confidence score of each entity identified by NER.

[0104] Step 502: Use the LLM for semantic consistency check: Input the entities identified by the NER small model and their context information into the LLM, and use the LLM to evaluate the semantic consistency between the entities and the context.

[0105] Step 503: Combine the results of probability estimation and semantic consistency check to obtain the final deterministic score.

[0106] Step 6: The LLM verifies and corrects the results of the traditional NER small model, which specifically includes the following steps:

[0107] Step 601: Analyze the above deterministic score, set a threshold, and the recognition results below this threshold are regarded as uncertain, while the recognition results above this threshold can be directly accepted.

[0108] Step 602: For the recognition results with low certainty, the LLM will conduct further verification:

[0109] First, utilize the semantic understanding ability of the LLM to verify whether the recognition results of the NER small model conform to the context semantics;

[0110] Then query the financial domain database to verify whether the recognition results of the NER small model are consistent with known facts;

[0111] Finally, apply the specific rules in the financial domain to check whether the recognition results of the NER small model comply with the rules.

[0112] Step 603: If the verification results indicate that the recognition results of the NER small model are incorrect, the LLM will provide corrections: The LLM will re-recognize the partial text of the context of the entity, and use the context information to adjust or modify the recognition results of the NER small model to make it more consistent with the context semantics. For example: The input text is "On January 1, 2023, ABC Bank and XYZ Company reached a cooperation agreement." The traditional NER small model recognized "XYZ Company" as an organization name, but the certainty score of this recognition result is low. The LLM will combine the context semantics, database query, and rule checking to verify whether "XYZ Company" is a real financial institution. If the verification results of the LLM indicate that the NER small model recognition is incorrect, it will provide correction suggestions, such as correcting "XYZ Company" to "XYZ Group".

[0113] Step 7: Feedback and iteration: Add the results corrected by the above LLM to the training dataset of the NER small model for subsequent training and optimization. Through continuous iteration, improve the performance of the NER small model in the financial domain.

Claims

1. A method for intelligent verification and correction of named entity recognition based on a large language model in the financial field, characterized in that: The following steps are involved: Step 1: Data construction to generate a large amount of labeled data; Step 2: Adjust the instructions based on the data constructed in step 1, so that the model can understand and execute specific task instructions. Through instruction adjustment, the general capabilities of LLM are transformed into specific capabilities focused on financial NER, so that LLM can achieve higher accuracy and stronger generalization ability in financial NER tasks. Step 3: Supervised fine-tuning Based on the instruction adjustment, by constructing a dataset containing various error types, the model learns how to identify and correct these errors. At the same time, the context understanding ability of LLM is used to improve the accuracy of the model when dealing with fuzzy or ambiguous entities. Step 4: Evaluate the fine-tuned LLM to ensure that the NER effect of LLM in the financial field is significantly improved compared with other large language models; Step 5: Conduct deterministic evaluation on the NER results of the small model; Step 6: LLM verifies and corrects the results of the traditional NER small model; Step 7: Feedback and iteration: The LLM-corrected results are added to the training data set of the NER small model for subsequent training and optimization. Through continuous iteration, the performance of the NER small model in the financial field is improved.

2. The method for intelligent verification and correction of named entity recognition based on a large language model in the financial field as claimed in claim 1, characterized in that: The step 1 comprises the following steps: Step 101, extracting text using a large corpus, where the text covers finance; Step 102: Divide the text collected in step 101 into paragraphs with a maximum length of 256 tags, and randomly select a number of paragraphs as inputs of the large language model; Step 103: Generate entity mentions and their associated types based on the paragraphs obtained in step 102 using a large language model; Step 104: clean the data obtained in step 103 to filter out unparseable outputs and inapplicable entities; Step 105 , classify the entities obtained in step 105 into types, and count the top N entity types in each frequency range to construct data and understand the entity type distribution, so as to guide the subsequent instruction adjustment process.

3. The intelligent verification and correction method for named entity recognition based on a large language model in the financial field as claimed in claim 2, characterized in that: In step 103, Temperature is fixed to 0 during the generation process to ensure that the generated entity mentions and types are stable.

4. The method for intelligent verification and correction of named entity recognition based on a large language model in the financial field as claimed in claim 1, characterized in that: The step 2 specifically includes the following steps: Step 201: Convert the data obtained in step 1 into a conversational template, each template comprising: A) System message: Tip: LLM is a powerful information extraction system; B) Hint: Given a text, the task of the LLM is to extract all entities and determine their entity types; C) Output: The output is a list tuple in the following format: [("Entity 1", "Entity 1 type"), ...]; D) User: natural language query based on entity type; E) Assistant: Generates a JSON list containing corresponding entity mentions based on the user query. Step 202: convert the type Ti of each entity appearing in the data obtained in step 201 into a natural language query; Step 203, adjust the LLM to generate a structured output Yi in the form of JSON containing all entities corresponding to Ti in the paragraph; Step 204: extract negative entity types from the set of all entity types that do not appear in the article as a query, and set the expected output to an empty JSON.

5. The method for intelligent verification and correction of named entity recognition based on a large language model in the financial field as claimed in claim 1, characterized in that: The step 3 specifically includes the following steps: Step 301, constructing a data set for three situations: entity type error, entity name error, and missing entity error; Step 302: manually check the above labels and convert them into natural language format; Step 303: Fine-tune the LLM using the LLM combined with the manually annotated dataset. Different hyperparameters may be considered to find the best fine-tuning strategy.

6. The method for intelligent verification and correction of named entity recognition based on a large language model in the financial field as claimed in claim 5, characterized in that: In step 301, the entity type error is that an ambiguous entity is prone to type errors; The entity name error is the presence of extra symbols or the inability to accurately identify the proper name; Missing entity errors occur when an entity in a paragraph is not recognized.

7. The method for intelligent verification and correction of named entity recognition based on a large language model in the financial field as claimed in claim 5, characterized in that: In step 301, the specific method of constructing the data set is: input the original text and the previous recognition results to the LLM, let the LLM determine whether there is an entity type error, entity name error or missing entity error, and label the error.

8. The method for intelligent verification and correction of named entity recognition based on a large language model in the financial field as claimed in claim 1, characterized in that: The step 4 specifically comprises the following steps: Step 401: Use strict entity-level micro-F1 as the evaluation metric, requiring that entity types and boundaries fully match the basic facts; Step 402: compare the fine-tuned LLM with the existing large model to evaluate its NER effect in the financial field; Step 403: Through ablation experiments, evaluate the contribution of instruction adjustment and negative sampling in step 2 to model performance, and evaluate the impact of different fine-tuning strategies in step 3 on model performance.

9. The method for intelligent verification and correction of named entity recognition based on a large language model in the financial field as claimed in claim 1, characterized in that: The step 5 specifically comprises the following steps: Step 501, use LLM to perform probability estimation: input the entities recognized by the NER small model and their context information into the LLM, and use the LLM to calculate the probability or confidence score of each entity recognized by the NER. Step 502: Use LLM to perform semantic consistency check: input the entity identified by the NER mini-model and its context information into LLM, and use LLM to evaluate the semantic consistency between the entity and the context. Step 503: Combine the results of the probability estimation and the semantic consistency check to obtain a final certainty score.

10. The method for intelligent verification and correction of named entity recognition based on a large language model in the financial field according to claim 1, characterized in that: The step 6 specifically comprises the following steps: Step 601: Analyze the certainty score obtained in step 5 and set a threshold. Recognition results below the threshold are considered uncertain, and recognition results above the threshold can be directly accepted. Step 602: For recognition results with low certainty, LLM will perform further verification: First, the semantic understanding ability of LLM is used to verify whether the recognition results of the NER small model conform to the context semantics; Then query the financial database to verify whether the recognition results of the NER small model are consistent with the known facts; Finally, apply specific rules in the financial field to check whether the recognition results of the NER small model meet the rules; Step 603: If the verification result shows that the recognition result of the NER small model is incorrect, LLM will provide corrections: LLM will re-recognize the contextual text of the entity and use the contextual information to adjust or modify the recognition result of the NER small model to make it more consistent with the contextual semantics.

Images