Financial case information mining method and system based on large language model

Abstract

The invention provides a financial case information mining system and method based on a large language model, and the method comprises the steps: carrying out the preprocessing of a financial case file through a financial law large language model, and carrying out the entity recognition of the preprocessing data through the financial law large language model and a constructed financial law knowledge graph; and the man-machine loopback module intelligently analyzes and determines the entity recognition result, performs entity relationship recognition according to the determined entity recognition result by using a financial law big language model and a financial law knowledge graph, intelligently analyzes and determines the entity relationship recognition result, and outputs the recognized entity and entity relationship. The accuracy and reliability of the identification result are ensured while the identification efficiency is improved, the omission condition of the entity relationship is avoided as far as possible, and the financial case handling accuracy requirement is met.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a financial case information mining method and system based on a large language model. Background Art

[0002] As my country's financial market develops rapidly, financial misconduct activities are also increasing, which puts higher requirements on institutions that handle financial-related issues. In the traditional financial case handling process, the analysis method of the entities involved and the relationship between them often relies on manual review. Most financial cases have large volumes of documents, even up to several hundred pages. Manual review is inefficient and easy to miss important information. Summary of the invention

[0003] Based on the above problems, the present invention provides a financial case information mining system and method based on a large language model, aiming to solve the technical problems in the prior art such as low efficiency and easy omissions in manual review of financial cases.

[0004] A financial case information mining system based on a large language model, comprising:

[0005] The input and preprocessing module is used to input the financial case files to be mined, and use the fine-tuned financial legal language model to preprocess the financial case files to be mined to obtain preprocessed data;

[0006] The entity recognition module is connected to the input and preprocessing modules and is used to perform entity recognition on the preprocessed data using the financial law large language model and the constructed financial law knowledge graph to obtain entity recognition results;

[0007] The human-machine loop module is connected to the entity recognition module and is used to: use the trained local small model to perform intelligent analysis on the entity recognition results and manually determine the final results of entity recognition;

[0008] The entity relationship recognition module is connected to the human-computer loopback module and is used to: use the financial law large language model and the financial law knowledge graph to perform entity relationship recognition on the preprocessed data according to the final result of entity recognition to obtain the entity relationship recognition result;

[0009] The human-machine loopback module is also used to: use the local small model to intelligently analyze the entity relationship recognition results, and manually determine the final results of entity relationship recognition;

[0010] The result output display module is connected to the human-computer loopback module and is used to jointly display the final results of entity recognition and entity relationship recognition.

[0011] Furthermore, the system also includes:

[0012] A local personalized word library is used to store the collected personalized knowledge data, which is financial related data within a predetermined range;

[0013] Personalized graph library, used to store personalized knowledge graphs built based on personalized knowledge data;

[0014] Financial law graph library, used to store financial law knowledge graph;

[0015] The entity recognition module is connected to the local personalized vocabulary, personalized graph library and financial legal graph library, including:

[0016] A first entity recognition unit, used to jointly use the financial law language model and the personalized knowledge data in the local personalized vocabulary to perform entity recognition on the preprocessed data to obtain a first entity recognition result;

[0017] A second entity recognition unit is connected to the first entity recognition unit and is used to: jointly use the financial legal knowledge graph and the personalized knowledge graph to verify and match the first entity recognition result to obtain a second entity recognition result;

[0018] The entity recognition result includes a first entity recognition result and a second entity recognition result.

[0019] Furthermore, the system also includes:

[0020] A local personalized word library is used to store the collected personalized knowledge data, which is financial related data within a predetermined range;

[0021] Personalized graph library, used to store personalized knowledge graphs built based on personalized knowledge data;

[0022] Financial law graph library, used to store financial law knowledge graph;

[0023] The entity relationship recognition module is connected to the local personalized vocabulary, personalized graph library and financial legal graph library, including:

[0024] The first relationship recognition unit is used to: jointly use the financial law language model and the personalized knowledge data in the local personalized vocabulary, perform entity relationship recognition on the preprocessed data according to the final result of entity recognition, and obtain a first entity relationship recognition result;

[0025] A second relationship identification unit, connected to the first relationship identification unit, is used to: jointly use the financial legal knowledge graph and the personalized knowledge graph to verify and match the first entity relationship identification result to obtain a second entity relationship identification result;

[0026] The entity relationship recognition result includes a first entity relationship recognition result and a second entity relationship recognition result.

[0027] Furthermore, the human-machine loopback module includes:

[0028] An intelligent processing unit deploys a local small model, and is used to analyze the first entity recognition result and the second entity recognition result using the local small model to obtain a comprehensive entity recognition result;

[0029] The human-computer interaction unit is connected to the intelligent processing unit and is used to:

[0030] The entity comprehensive recognition result is displayed, and the entity comprehensive recognition result is manually adjusted to obtain the entity recognition adjustment result;

[0031] When it is considered that there is no need to adjust the entity comprehensive recognition result manually, the entity comprehensive recognition result is confirmed as the final entity recognition result;

[0032] When it is manually considered that the entity recognition adjustment result does not need to be iterated any further, the entity recognition adjustment result is confirmed as the final entity recognition result;

[0033] When it is manually considered that the entity recognition adjustment result needs to be iterated again, the entity recognition adjustment result is added to the data as the entity recognition iteration;

[0034] The intelligent processing unit is also used to:

[0035] Pass the final result of entity recognition to the entity relationship recognition module;

[0036] Passing entity recognition iteration data to the entity recognition module so that the entity recognition module performs an iterative process;

[0037] The entity recognition module is also used to: use the financial law big language model and the constructed financial law knowledge graph, combine entity recognition to iteratively add data, perform entity recognition on the preprocessed data again, and obtain entity recognition results.

[0038] Furthermore, the system also includes a local personalized word library, which is used to store the collected personalized knowledge data, and the personalized knowledge data is financial related data within a predetermined range;

[0039] The human-computer loopback module connects to the local personalized vocabulary, including:

[0040] The intelligent processing unit deploys a local small model, and is used to: use the local small model to analyze the first entity recognition result using the local personalized vocabulary to obtain a third entity recognition result, and provide an entity comprehensive recognition result based on the first entity recognition result, the second entity recognition result, and the third entity recognition result;

[0041] The human-computer interaction unit is connected to the intelligent processing unit and is used to: display the entity comprehensive recognition result, and manually determine the final entity recognition result based on the entity comprehensive recognition result.

[0042] Furthermore, the human-machine loopback module includes:

[0043] An intelligent processing unit deploys a local small model and is used to use the local small model to analyze the first entity relationship recognition result and the second entity relationship recognition result to obtain a comprehensive entity relationship recognition result;

[0044] The human-computer interaction unit is connected to the intelligent processing unit and is used to:

[0045] The entity relationship comprehensive recognition result is displayed, and the entity relationship comprehensive recognition result is manually adjusted to obtain the entity relationship recognition adjustment result;

[0046] When it is considered that there is no need to adjust the entity relationship comprehensive recognition result manually, the entity relationship comprehensive recognition result is confirmed as the final entity relationship recognition result;

[0047] When it is manually considered that the entity relationship recognition adjustment result does not need to be iterated any further, the entity relationship recognition adjustment result is confirmed as the final entity relationship recognition result;

[0048] When it is manually considered that the entity relationship recognition adjustment result needs to be iterated again, the entity relationship recognition adjustment result is added to the data as the entity relationship recognition iteration;

[0049] The intelligent processing unit is also used to:

[0050] Pass the final result of entity relationship recognition to the result output display module;

[0051] Passing entity relationship recognition iteration adding data to the entity relationship recognition module so that the entity relationship recognition module performs an iterative process;

[0052] The entity relationship recognition module is also used to: use the financial law big language model and the constructed financial law knowledge graph, combine entity relationship recognition to iteratively join data and entity recognition final results, perform entity relationship recognition on the preprocessed data, and obtain entity relationship recognition results.

[0053] Furthermore, the system also includes a local personalized word library, which is used to store the collected personalized knowledge data, and the personalized knowledge data is financial related data within a predetermined range; a human-computer loop module, connected to the local personalized word library, including:

[0054] The intelligent processing unit deploys a local small model and is used to: use the local small model to analyze the first entity relationship recognition result using the local personalized vocabulary to obtain a third entity relationship recognition result, and provide an entity relationship comprehensive recognition result based on the first entity relationship recognition result, the second entity relationship recognition result and the third entity relationship recognition result;

[0055] The human-computer interaction unit is connected to the intelligent processing unit and is used to: display the comprehensive recognition result of entity relationship and manually determine the final result of entity relationship recognition based on the comprehensive recognition result of entity relationship.

[0056] Furthermore, the system also includes:

[0057] The vocabulary management module connects the local personalized vocabulary and personalized graph library and is used for:

[0058] Update the personalized knowledge data in the local personalized vocabulary;

[0059] Building a personalized knowledge graph based on the personalized knowledge data, and updating the personalized knowledge graph according to the updated personalized knowledge data;

[0060] Optimize the prompt word engineering template of the financial and legal large language model based on the local personalized vocabulary.

[0061] Furthermore, personalized knowledge data includes financial related data extracted from financial cases that have been tried and decided by the courts;

[0062] The first relationship identification unit includes:

[0063] The second search sub-unit is used to use the financial law big language model to retrieve the financial related data of the same type of financial cases from the personalized word library according to the final result of entity recognition and the pre-processed data, and obtain the second search result;

[0064] The relationship identification subunit is connected to the second retrieval subunit and is used to: use the financial law large language model to obtain the first entity relationship identification result according to the final result of entity recognition, the preprocessed data and the second retrieval result.

[0065] A financial case information mining method based on a large language model, using the aforementioned financial case information mining system based on a large language model, comprising:

[0066] Step A1, inputting the financial case files to be mined, and using the fine-tuned financial legal language model to pre-process the financial case files to be mined, to obtain pre-processed data;

[0067] Step A2, using the financial law big language model and the constructed financial law knowledge graph to perform entity recognition on the preprocessed data to obtain entity recognition results;

[0068] Step A3, using a human-machine loopback module to intelligently analyze the entity relationship recognition results, and manually determine the final results of the entity relationship recognition;

[0069] Step A4, using the financial law big language model and the financial law knowledge graph, according to the final result of entity recognition, perform entity relationship recognition on the preprocessed data to obtain the entity relationship recognition result;

[0070] Step A5, using a human-machine loopback module to intelligently analyze the entity relationship recognition results, and manually determine the final results of the entity relationship recognition;

[0071] Step A6, jointly displaying the final results of entity recognition and entity relationship recognition.

[0072] The beneficial technical effect of the present invention is that the present invention combines a large language model, a knowledge graph, and human-computer loopback technology to perform intelligent analysis and manual supervision on entities and entity relationships in financial cases. While improving the recognition efficiency, it ensures the accuracy and reliability of the recognition results and avoids the omission of entities and entity relationships as much as possible, meeting the accuracy requirements of financial case handling. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1-6 A schematic diagram of a module of a financial case information mining system based on a large language model according to the present invention;

[0074] Figure 7 The present invention is a flowchart of the steps of a financial case information mining method based on a large language model. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0077] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0078] See also Figure 1 The present invention provides a financial case information mining system based on a large language model, comprising:

[0079] An input and preprocessing module (1) is used to input the financial case files to be mined, and to preprocess the financial case files to be mined using the fine-tuned financial legal language model to obtain preprocessed data;

[0080] An entity recognition module (2) is connected to the input and preprocessing modules (1) and is used to perform entity recognition on the preprocessed data using the financial law large language model and the constructed financial law knowledge graph to obtain an entity recognition result;

[0081] A human-machine loop module (3) is connected to the entity recognition module (2) and is used to: use the trained local small model to perform intelligent analysis on the entity recognition result, and manually determine the final result of the entity recognition;

[0082] The entity relationship recognition module (4) is connected to the human-computer loopback module (3) and is used to: use the financial law large language model and the financial law knowledge graph to perform entity relationship recognition on the preprocessed data according to the final result of entity recognition, and obtain the entity relationship recognition result;

[0083] The human-machine loop module (3) is also used to: use the local small model to intelligently analyze the entity relationship recognition results, and manually determine the final results of the entity relationship recognition;

[0084] The result output display module (5) is connected to the human-machine loopback module (3) and is used to jointly display the final result of entity recognition and the final result of entity relationship recognition. The present invention combines a large language model, a knowledge graph and a human-machine loopback technology to perform intelligent analysis and manual supervision recognition of entities and entity relationships in financial cases, thereby improving the recognition efficiency while ensuring the accuracy and reliability of the recognition results, and avoiding the omission of entities and entity relationships as much as possible, thereby meeting the accuracy requirements of financial case handling.

[0085] Specifically, in the input and preprocessing module (1), the financial law big language model creates prompt words according to the prompt word engineering template, and first preprocesses the input financial case file. The preprocessing content includes but is not limited to segmentation, cleaning, word segmentation and part-of-speech tagging.

[0086] For some financial case documents with very long content, the financial law big language model segments and slices the financial case documents into multiple segments of data for separate processing. For example, they can be divided based on the time of financial behavior or the events of financial behavior to ensure that the identification of the entity is not affected.

[0087] Some sentences that are not fluent, have omitted content (such as a certain person's name, etc.), erroneous content such as typos, repeated content, etc. are cleaned up to correct the influent content, supplement the omitted content (complete the name), correct the erroneous content, delete the repeated content, etc.

[0088] Perform word segmentation on some content to facilitate better entity recognition and entity relationship recognition. For example, a person involved in the case is called Zhang Sanhe. This name can be easily identified as Zhang San, and the word "he" is identified as a conjunction. Therefore, word segmentation is performed and Zhang Sanhe is treated as a person's name.

[0089] Some words are labeled as nouns, verbs, etc. to facilitate the subsequent identification of entities and entity relationships.

[0090] Entities include names of people, laws, regulations, company names, etc. in the financial case documents.

[0091] Specifically, the financial law big language model is obtained by fine-tuning the pre-trained general large language model using the collected textual knowledge data in the fields of law and finance (including but not limited to the text data of laws and regulations in my country, the text data of case law on the Judgment Documents Network, etc.) on the basis of the open source pre-trained general large language model (such as Tongyi Qianwen, GLM, etc.). Therefore, the financial law big language model obtained by fine-tuning the present invention is more suitable for processing text data in the specific field of financial law.

[0092] See also Figure 2 , further, the system also includes:

[0093] A local personalized word library (6), used to store the collected personalized knowledge data, where the personalized knowledge data is financial related data within a predetermined range;

[0094] Personalized graph library (7), used to store personalized knowledge graphs constructed based on personalized knowledge data;

[0095] Financial Law Graph Library (8), used to store financial law knowledge graphs;

[0096] The entity recognition module (2) is connected to the local personalized vocabulary, personalized graph library and financial legal graph library, including:

[0097] A first entity recognition unit (21) is used to jointly use the financial law language model and the personalized knowledge data in the local personalized vocabulary to perform entity recognition on the pre-processed data to obtain a first entity recognition result;

[0098] A second entity identification unit (22), connected to the first entity identification unit (21), is used to: jointly use the financial legal knowledge graph and the personalized knowledge graph to verify and match the first entity identification result to obtain a second entity identification result;

[0099] The entity recognition result includes a first entity recognition result and a second entity recognition result.

[0100] The local personalized word library can be personalized knowledge data added by the system users themselves, that is, the local personalized word library created by the users themselves, adding personalized knowledge data according to their own needs. Local laws and regulations in different regions are different, so users can add personalized knowledge data with regional characteristics according to their regions, such as some organization names, some financial product names, place names, names, local law names, etc. in this region. You can also add data extracted from some financial cases, such as case names, abbreviations, codes, codes, contract agreement file names, legal and regulatory names, place names, organization names, names, financial product names, and other entities, as well as the associations between entities, i.e., entity relationships. Because the local personalized word library has personalized characteristics such as regions, it helps the large language model to accurately and reliably identify entities and entity relationships.

[0101] The financial legal knowledge graph of the present invention records a large number of historical financial cases and related laws and regulations of the judicial and prosecutorial departments in the form of knowledge graphs, and provides a basis for verifying the identified entities and entity relationships.

[0102] See also Figure 3 , further, the system also includes:

[0103] A local personalized word library (6), used to store the collected personalized knowledge data, where the personalized knowledge data is financial related data within a predetermined range;

[0104] Personalized graph library (7), used to store personalized knowledge graphs constructed based on personalized knowledge data;

[0105] Financial Law Graph Library (8), used to store financial law knowledge graphs;

[0106] The entity relationship recognition module (4) is connected to the local personalized vocabulary, personalized graph library and financial legal graph library, including:

[0107] A first relationship recognition unit (41) is used to: jointly use the financial law language model and the personalized knowledge data in the local personalized vocabulary, perform entity relationship recognition on the pre-processed data according to the final result of entity recognition, and obtain a first entity relationship recognition result;

[0108] A second relationship identification unit (42), connected to the first relationship identification unit (41), is used to: jointly use the financial legal knowledge graph and the personalized knowledge graph to verify and match the first entity relationship identification result to obtain a second entity relationship identification result;

[0109] The entity relationship recognition result includes a first entity relationship recognition result and a second entity relationship recognition result.

[0110] Utilizing the financial law language model and local personalized vocabulary, the relationships between entities are extracted from the text, such as legal plaintiff-defendant relationships, capital flow relationships, kinship relationships, cooperative relationships, creditor-debtor relationships, etc. On this basis, the identified and mined entity relationships are matched and verified with the content in the financial law knowledge graph library.

[0111] Furthermore, the human-machine loopback module (3) comprises:

[0112] An intelligent processing unit (31) deploys a local small model and is used to analyze the first entity recognition result and the second entity recognition result using the local small model to obtain a comprehensive entity recognition result;

[0113] See also Figure 4 ,Furthermore, the human-machine loopback module (3) comprises:

[0114] The human-machine interaction unit (32) is connected to the intelligent processing unit (31) and is used for:

[0115] The entity comprehensive recognition result is displayed, and the entity comprehensive recognition result is manually adjusted to obtain the entity recognition adjustment result;

[0116] When it is considered that there is no need to adjust the entity comprehensive recognition result manually, the entity comprehensive recognition result is confirmed as the final entity recognition result;

[0117] When it is manually considered that the entity recognition adjustment result does not need to be iterated any further, the entity recognition adjustment result is confirmed as the final entity recognition result;

[0118] When it is manually considered that the entity recognition adjustment result needs to be iterated again, the entity recognition adjustment result is added to the data as the entity recognition iteration;

[0119] The intelligent processing unit (31) is also used for:

[0120] Pass the final result of entity recognition to the entity relationship recognition module;

[0121] Passing entity recognition iteration data to the entity recognition module so that the entity recognition module performs an iterative process;

[0122] The entity recognition module (2) is also used to: use the financial law large language model and the constructed financial law knowledge graph, combine entity recognition to iteratively add data, perform entity recognition on the pre-processed data again, and obtain entity recognition results.

[0123] Furthermore, the system also includes a local personalized word library (6), which is used to store the collected personalized knowledge data, and the personalized knowledge data is financial related data within a predetermined range;

[0124] The human-machine loopback module (3) is connected to the local personalized vocabulary (6), including:

[0125] The intelligent processing unit (31) deploys a local small model and is used to: use the local small model to analyze the first entity recognition result using the local personalized word library to obtain a third entity recognition result, and provide an entity comprehensive recognition result based on the first entity recognition result, the second entity recognition result and the third entity recognition result;

[0126] The human-computer interaction unit (32) is connected to the intelligent processing unit (31) and is used to display the entity comprehensive recognition result and manually determine the entity recognition final result based on the entity comprehensive recognition result.

[0127] The human-computer interaction unit (32) displays the first entity recognition result, the second entity recognition result, the third entity recognition result and the entity comprehensive recognition result;

[0128] The first entity recognition result includes the recognized entities and a confidence score for each entity;

[0129] The second entity recognition result includes the recognized entities and a confidence score for each entity;

[0130] The third entity recognition result includes the recognized entities and the confidence score for each entity;

[0131] The comprehensive entity recognition results include the entity, the confidence score for each entity, and the total confidence score for the overall entity recognition.

[0132] The confidence score represents the confidence score of whether the identified entity is a real and reliable entity.

[0133] The intelligent processing unit (31) integrates and analyzes the first entity recognition result, the second entity recognition result, and the third entity recognition result, for example, by weighting the confidence scores of the three entity recognition results to obtain a confidence score for each entity, and comprehensively analyzing to obtain a total confidence score for the overall entity recognition.

[0134] The human-computer interaction unit (32) sorts the entity comprehensive recognition results from low to high according to the confidence score of each entity, so that users can see the recognized entity with the lowest confidence for adjustment and modification.

[0135] The total confidence score of the overall entity recognition is given to help users understand the overall reliability of the entities recognized in this iteration. Combined with the confidence score of each entity and the amount of modification and adjustment made to the entities in the comprehensive entity recognition results, users can determine whether to choose to iterate again. If no further iteration is required, the final result of entity recognition can be determined and the entity relationship recognition can be continued for subsequent entity relationship recognition. Through continuous iterative recognition and manual adjustment, incorrect entity and entity relationship recognition and omission of entities and entity relationships can be avoided.

[0136] Specifically, for entities with low scores or not recognized, manual verification and supplementation can be performed in combination with the context. Users can modify the entities in the comprehensive entity recognition results (including correcting errors and supplementing entities missed by machine recognition), and can also adjust the confidence score corresponding to the entity. For example, the recognized entity A is correct, but the confidence score given is relatively low, such as 70 points (out of 100 points). At this time, the user can adjust the confidence score of entity A to 100 points, so that in subsequent iterations, the financial law language model and the financial law knowledge graph can assist in more accurate recognition of entities. Since manual adjustments at a certain point may affect the recognition of other entities, it is up to humans to decide whether to perform machine recognition and mining iterations again. The process of optimizing entity recognition and mining based on intelligent analysis of human-machine loops can be repeated for multiple rounds, which can improve the personalized applicability of the entities involved in the financial case files, reduce the "machine hallucination" of the large language model, and improve the accuracy of entity recognition and mining involved in financial cases.

[0137] Furthermore, the human-machine loopback module (3) comprises:

[0138] The intelligent processing unit (31) deploys a local small model and is used to use the local small model to analyze the first entity relationship recognition result and the second entity relationship recognition result to obtain a comprehensive entity relationship recognition result.

[0139] Furthermore, the human-machine loop module (3) includes: a human-machine interaction unit (32) connected to the intelligent processing unit (31) for:

[0140] The entity relationship comprehensive recognition result is displayed, and the entity relationship comprehensive recognition result is manually adjusted to obtain the entity relationship recognition adjustment result;

[0141] When it is considered that there is no need to adjust the entity relationship comprehensive recognition result manually, the entity relationship comprehensive recognition result is confirmed as the final entity relationship recognition result;

[0142] When it is manually considered that the entity relationship recognition adjustment result does not need to be iterated any further, the entity relationship recognition adjustment result is confirmed as the final entity relationship recognition result;

[0143] When it is manually considered that the entity relationship recognition adjustment result needs to be iterated again, the entity relationship recognition adjustment result is added to the data as the entity relationship recognition iteration;

[0144] The intelligent processing unit (31) is also used for:

[0145] Pass the final result of entity relationship recognition to the result output display module;

[0146] Passing entity relationship recognition iteration adding data to the entity relationship recognition module so that the entity relationship recognition module performs an iterative process;

[0147] The entity relationship recognition module (4) is also used to: use the financial law big language model and the constructed financial law knowledge graph, combine the entity relationship recognition iterative joining data and the final result of entity recognition, perform entity relationship recognition on the pre-processed data, and obtain the entity relationship recognition result.

[0148] Based on multiple rounds of dialogues between human and machine loops, intelligent analysis and optimization are performed using a combination of large language model prompt word engineering templates, personalized knowledge graph libraries, local small models, financial law large language models, and financial law knowledge graph libraries, thereby improving the accuracy and efficiency of mining entities and entity relationships involved in financial cases. While optimizing intelligent analysis based on human-machine loops, the word library management module can also be used to adjust and improve the content in the local personalized word library.

[0149] Furthermore, the system also includes a local personalized word library (6), which is used to store the collected personalized knowledge data, and the personalized knowledge data is financial related data within a predetermined range;

[0150] The human-machine loopback module (3) is connected to the local personalized vocabulary (6), including:

[0151] The intelligent processing unit (31) deploys a local small model and is used to: use the local small model to analyze the first entity relationship recognition result using the local personalized vocabulary to obtain a third entity relationship recognition result, and provide an entity relationship comprehensive recognition result based on the first entity relationship recognition result, the second entity relationship recognition result and the third entity relationship recognition result;

[0152] The human-computer interaction unit (32) is connected to the intelligent processing unit (31) and is used to: display the entity relationship comprehensive recognition result, and manually determine the entity relationship recognition final result based on the entity relationship comprehensive recognition result.

[0153] The human-computer interaction unit (32) displays the first entity relationship recognition result, the second entity relationship recognition result, the third entity relationship recognition result and the entity relationship comprehensive recognition result;

[0154] The first entity relationship recognition result includes the recognized entity relationships and a confidence score for each entity relationship;

[0155] The second entity relationship recognition result includes the recognized entity relationships and a confidence score for each entity relationship;

[0156] The third entity relationship recognition result includes the recognized entity relationship and the confidence score of each entity relationship;

[0157] The comprehensive entity relationship recognition results include entity relationships, confidence scores for each entity relationship, and the total confidence score for the overall entity relationship recognition.

[0158] The confidence score represents the confidence score of whether the identified entity relationship is a true and reliable entity relationship.

[0159] The intelligent processing unit (31) integrates and analyzes the first entity relationship recognition result, the second entity relationship recognition result, and the third entity relationship recognition result, for example, by weighting the confidence scores of the three entity relationship recognition results to obtain a confidence score for each entity relationship, and comprehensively analyzing to obtain a total confidence score for the overall entity relationship recognition.

[0160] The human-computer interaction unit (32) sorts the entity relationships identified according to the confidence score of each entity relationship from low to high based on the entity relationship comprehensive identification results, so that users can see the entity relationship identified with the lowest confidence level for adjustment and modification.

[0161] The confidence score of the overall entity relationship recognition is given to help users understand the overall reliability of the entity relationships recognized in this iteration. Combined with the confidence score of each entity relationship and the amount of modification and adjustment of the entity relationship in the comprehensive entity relationship recognition result, users can determine whether to choose to iterate again. If no further iteration is required, the final result of entity relationship recognition can be determined, and finally the visual output of entities and entity relationships is performed. Through continuous iterative recognition and manual adjustment, incorrect entity and entity relationship recognition and omission of entities and entity relationships can be avoided.

[0162] Specifically, for entity relationships with low scores or not recognized, manual verification and supplementation can be performed in combination with the context. Users can modify the entity relationships of the comprehensive entity relationship recognition results (including correcting errors and supplementing entities missed by machine recognition), and can also adjust the confidence score corresponding to the entity relationship. For example, the recognized entity relationship B is correct, but the confidence score given is relatively low, such as 70 points (out of 100 points). At this time, the user can adjust the confidence score of entity relationship B to 100 points, so that in subsequent iterations, the financial law language model and the financial law knowledge graph can be assisted to more accurately identify the entity relationship. Since the adjustment of a certain point by humans may affect the recognition of other entity relationships, it can be decided by humans whether to perform machine recognition and mining iterations again. The process of optimizing entity relationship recognition and mining based on intelligent analysis of human-machine loops can be repeated for multiple rounds, which can improve the personalized applicability of the entity relationships involved in the financial case files, reduce the "machine hallucination" of the large language model, and improve the accuracy of the recognition and mining of entity relationships involved in financial cases.

[0163] Furthermore, the system also includes:

[0164] The word library management module (9) is connected to the local personalized word library (6) and the personalized atlas library (7) respectively, and is used to:

[0165] Updating the personalized knowledge data in the local personalized vocabulary (6);

[0166] Building a personalized knowledge graph based on the personalized knowledge data, and updating the personalized knowledge graph according to the updated personalized knowledge data;

[0167] Optimize the prompt word engineering template of the financial and legal large language model based on the local personalized vocabulary (6).

[0168] Updating the personalized knowledge data in the local personalized word library (6) includes querying and retrieving the existing content in the local personalized word library, modifying and adjusting it, and adding new data.

[0169] The vocabulary management module (9) is mainly used for:

[0170] Actively manage and update personalized knowledge data in the local personalized vocabulary to reflect new developments and changes in the financial and legal fields;

[0171] Use the personalized knowledge data in the local personalized vocabulary to establish and optimize the prompt word engineering template of the financial law language model;

[0172] Use the personalized knowledge data in the local personalized vocabulary to build and update the personalized knowledge graph;

[0173] The entity recognition module (2), the entity relationship recognition module (4) and the human-computer loopback module (3) are respectively connected to record the iterative process of entity recognition and entity relationship recognition for reference in subsequent entity recognition and entity relationship recognition;

[0174] The personalized knowledge data in the local personalized vocabulary is used to train the local small model through a supervised machine learning algorithm.

[0175] The word library management module (9) adds the entities and entity relationships finally determined by the entity recognition module, entity relationship recognition module and human-machine loop module to the personalized knowledge data to enrich the local personalized word library. In other words, in the process of multi-round dialogue intelligent analysis of the human-machine loop, the local personalized word library is continuously enriched and improved. The content in the personalized word library can be used to construct the prompt engineering template of the optimized large language model and the personalized knowledge graph repository, and can also be used as a sample for supervised machine learning to train a local small model. Through this mechanism, the personalized applicability of extracting the involved entities and entity relationships from the financial case data can be improved, and the accuracy of identifying and mining the involved entities and entity relationships for different end-user applications can be improved.

[0176] See also Figure 5 ,Furthermore, personalized knowledge data includes financial related data extracted from ,financial cases that have been tried and decided by the courts;

[0177] The first entity identification unit (21) comprises:

[0178] A first search subunit (211) is used to use the financial law language model to search for financial related data of similar financial cases from a personalized word library according to pre-processed data, and obtain a first search result;

[0179] The entity recognition subunit (212) is connected to the first search subunit (211) and is used to: use the financial law large language model to obtain a first entity recognition result based on the preprocessed data and the first search result.

[0180] See also Figure 6 ,Furthermore, personalized knowledge data includes financial related data extracted from ,financial cases that have been tried and decided by the courts;

[0181] The first relationship identification unit (41) comprises:

[0182] The second search subunit (411) is used to use the financial law big language model to search for financial related data of similar financial cases from the personalized word library according to the final result of entity recognition and the pre-processed data, and obtain a second search result;

[0183] The relationship identification subunit (412) is connected to the second retrieval subunit (411) and is used to: use the financial law large language model to obtain the first entity relationship identification result according to the final result of entity identification, the preprocessed data and the second retrieval result.

[0184] The local personalized word library is a word library of personalized knowledge data stored locally, including traditional relational databases and vector databases. The local personalized word library acts on the financial legal big language model through database fuzzy retrieval, vector library retrieval, or RAG (Retrieval-Augmented Generation). The local personalized word library and the financial legal big language model can improve the recognition performance and interpretability of the financial legal big language model through RAG methods. On the one hand, because some financial related data of financial cases are added, the financial legal big language model can first retrieve the entity and entity relationship information of similar financial cases, such as financial cases of the same type, when performing entity and entity relationship recognition. Different types of financial cases may have differences in entity recognition and entity relationship recognition. For example, in a certain type of financial case, there is an organizational-level entity relationship in the entity relationship. Through the local personalized word library, the financial legal big language model uses the retrieved content as a reference to perform entity and entity relationship recognition of the financial case being processed, which can be more quickly and effectively recognized. Moreover, some people may have performed bad financial behavior more than once, and the degree of bad financial behavior may be different at different times. For example, the local personalized word library records the financial related data of a financial case X, which records that person X has bad financial behavior, and the organization relationship is the first level of serious bad financial behavior. In the financial case to be mined this time, person X also has bad financial behavior, but the bad behavior organization relationship is relatively low. If there is no local personalized word library, the financial law big language model is likely to identify entities and entity relationships based on the behavior of person X this time, and it may be considered that it is not an entity. However, because of the previous record of the local personalized word library, it is likely to identify person X as an entity this time, and is a person who has cumulatively performed bad financial behavior. Based on the local personalized word library, the financial law big language model outputs more accurate content with certain reference basis.

[0185] In addition, the word library management module is also used to record each iteration process, and the iteration process can be recorded in the local personalized word library, for example, the retrieval results of the entity recognition module and the entity relationship recognition module from the local personalized word library during iteration, as well as the entity recognition results of the entity recognition module, the entity relationship recognition results of the entity relationship recognition module, and the intelligent analysis results and manual operation process of the human-computer loop module are recorded in the local personalized word library, so that the local personalized word library has a long-term memory function, which is convenient for reference in subsequent iterations and helps to improve recognition performance. However, the large language model in the prior art usually has only a short-term context memory function and can only associate part of the previous information. The present invention has a long-term memory function through recording. In the next iteration, the word library management module guides the financial law large language model to refer to the previous retrieval results, recognition results and other iterative data through the prompt word project, so as to quickly identify entities and entity relationships. Moreover, it can also provide help for the next financial case to be mined. Moreover, each time the iteration process is recorded, these recorded iteration processes can be regarded as sample data, which is used for the subsequent use of the financial law large language model to quickly establish the prompt word project for entity recognition and entity relationship recognition. It can be seen that the accuracy of entity recognition and mining of large language model technology at this stage needs to be further improved due to the lack of understanding of past personalized knowledge, long-term memory loss, and machine hallucination and wrong answers. The present invention can assist the financial law large language model in memorizing and understanding specific words in the local personalized vocabulary, and better perform entity recognition, entity relationship recognition and mining on the current text. In particular, the local personalized vocabulary can more accurately identify the phenomenon of duplicate names among names and nicknames, and the non-standard brief descriptions in the same financial case file through multiple rounds of iterative recognition in the local personalized vocabulary that can assist memory and people in the loop.

[0186] Specifically, the result output display module (5) displays the identified entities and their relationship results to the user in the form of visual charts, etc., and supports multiple export functions.

[0187] Specifically, the result output display module (5) outputs and displays the final result of entity recognition, the final result of entity relationship recognition, the preprocessed data obtained by preprocessing, and the original input financial case file.

[0188] The result output display module (5) can export the entities and entity relationships finally identified in a variety of ways, including but not limited to Resource Description Framework (RDF), Extensible Markup Language (XML), JavaScript Object Notation (JSON), Character Separated Values ​​(CSV), graph database, relational database, etc.

[0189] See also Figure 7 The present invention provides a financial case information mining method based on a large language model, using the aforementioned financial case information mining system based on a large language model, comprising:

[0190] Step A1, inputting the financial case files to be mined, and using the fine-tuned financial legal language model to pre-process the financial case files to be mined, to obtain pre-processed data;

[0191] Step A2, using the financial law big language model and the constructed financial law knowledge graph to perform entity recognition on the preprocessed data to obtain entity recognition results;

[0192] Step A3, using a human-machine loopback module to intelligently analyze the entity relationship recognition results, and manually determine the final results of the entity relationship recognition;

[0193] Step A4, using the financial law big language model and the financial law knowledge graph, according to the final result of entity recognition, perform entity relationship recognition on the preprocessed data to obtain the entity relationship recognition result;

[0194] Step A5, using a human-machine loopback module to intelligently analyze the entity relationship recognition results, and manually determine the final results of the entity relationship recognition;

[0195] Step A6, jointly displaying the final results of entity recognition and entity relationship recognition.

[0196] Large Language Model (LLM) is an artificial intelligence technology based on deep learning, mainly used for natural language processing (NLP) tasks. Although large language models have made significant progress in the field of natural language processing, the technology generally uses black box models and does not provide complete intermediate operation steps for the output results. Therefore, due to training data bias, logical errors, fictional plots, etc., it will further produce problems such as machine hallucination, which also affects the use of large language model technology in application scenarios such as law, finance, and cases that require high accuracy of results.

[0197] Knowledge Graph (KG) is a structured knowledge representation method that uses concepts from graph theory to represent entities and their relationships. In a knowledge graph, entities are represented as nodes, and relationships between entities are represented as edges. Each node and edge can carry attribute information to provide a richer description. Knowledge graphs are widely used in artificial intelligence, big data analysis, information retrieval and other fields. Entities are the basic elements in knowledge graphs, usually representing objects or concepts in the real world, such as people, places, events, items, etc. Relationships represent the connection between entities and describe the interaction or association between entities. Although knowledge graphs have shown great potential and advantages in many fields, they also face a series of problems and limitations, such as polysemous words or polysemous relationships may lead to mismatching of entities or relationships; information between different data sources may contradict each other, resulting in inconsistent information in knowledge graphs, etc.

[0198] Therefore, in order to more accurately identify entities and entity relationships in financial cases, the present invention:

[0199] First, the conventional big language model is fine-tuned with laws and regulations related to finance to obtain a fine-tuned financial law big language model, which can better identify the content of financial cases.

[0200] Secondly, the present invention can reduce some mismatches in financial cases caused by polysemous words or polysemous relationships by constructing a financial law knowledge graph using financial-related laws and regulations. By utilizing the advantages of the knowledge graph, the financial law knowledge graph further verifies the entities and entity relationships identified by the financial law large language model.

[0201] Secondly, use human-in-the-loop technology. Human-in-the-loop (HITL) is a method of integrating human supervision and intervention in automated systems. In this model, humans and machines work together, with machines performing some tasks and humans supervising, verifying, and correcting the machine's decisions. Adjusting the entities and entity relationships identified by large language models and knowledge graphs through human-in-the-loop technology helps improve the accuracy, reliability, flexibility, and interpretability of the system.

[0202] Secondly, use local personalized vocabulary. Generally speaking, for example, local laws and regulations vary in different regions. Therefore, adding the names of some local financial companies, the names of people involved in finance, and the entities and entity relationships of some financial cases judged and executed locally can further help the large language model to identify entities and entity relationships.

[0203] Secondly, using personalized knowledge graphs, which are built based on local personalized vocabulary, personalized knowledge graphs are more helpful for the financial legal knowledge graph to verify entities and entity relationships.

[0204] The present invention combines the financial law big language model with the local personalized word library and the financial law knowledge graph technology, and uses the multi-round dialogue intention recognition mode of "human-machine loop" to reduce the "machine hallucination" of the big language model, and is ultimately used to improve the accuracy and efficiency of mining the entities and associations involved in the case. The present invention is suitable for institutions and units that handle financial case-related issues, helping them to quickly lock in key entity information and the associations between entities in massive financial case text data, thereby improving work efficiency and accuracy.

[0205] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. A financial case information mining system based on a large language model, characterized in that: include: An input and preprocessing module, used to input the financial case files to be mined, and preprocess the financial case files to be mined using the fine-tuned financial law language model to obtain preprocessed data; An entity recognition module, connected to the input and preprocessing modules, for performing entity recognition on the preprocessed data using the financial law large language model and the constructed financial law knowledge graph to obtain an entity recognition result; A human-machine loop module, connected to the entity recognition module, is used to: use the trained local small model to perform intelligent analysis on the entity recognition result, and manually determine the final result of entity recognition; An entity relationship recognition module, connected to the human-machine loopback module, is used to: use the financial law large language model and the financial law knowledge graph, and according to the final result of the entity recognition, perform entity relationship recognition on the preprocessed data to obtain an entity relationship recognition result; The human-machine loop module is also used to: use the local small model to perform intelligent analysis on the entity relationship recognition result, and manually determine the final result of the entity relationship recognition; A result output display module is connected to the human-computer loop module and is used to jointly display the final result of the entity recognition and the final result of the entity relationship recognition.

2. A financial case information mining system based on a large language model as claimed in claim 1, characterized in that: The system further comprises: A local personalized word library, used to store collected personalized knowledge data, wherein the personalized knowledge data is financial related data within a predetermined range; A personalized graph library, used to store a personalized knowledge graph constructed based on the personalized knowledge data; A financial law graph library, used to store the financial law knowledge graph; The entity recognition module is connected to the local personalized vocabulary, the personalized graph library and the financial legal graph library respectively, and includes: A first entity recognition unit, configured to jointly use the financial law large language model and the personalized knowledge data in the local personalized vocabulary to perform entity recognition on the preprocessed data to obtain a first entity recognition result; A second entity recognition unit, connected to the first entity recognition unit, is used to: jointly use the financial legal knowledge graph and the personalized knowledge graph to verify and match the first entity recognition result to obtain a second entity recognition result; The entity recognition result includes the first entity recognition result and the second entity recognition result.

3. A financial case information mining system based on a large language model as claimed in claim 1, characterized in that: The system further comprises: A local personalized word library, used to store collected personalized knowledge data, wherein the personalized knowledge data is financial related data within a predetermined range; A personalized graph library, used to store a personalized knowledge graph constructed based on the personalized knowledge data; A financial law graph library, used to store the financial law knowledge graph; The entity relationship recognition module is connected to the local personalized vocabulary, the personalized graph library and the financial legal graph library respectively, and includes: A first relationship recognition unit is used to: jointly use the financial law language model and the personalized knowledge data in the local personalized vocabulary, perform entity relationship recognition on the preprocessed data according to the final result of the entity recognition, and obtain a first entity relationship recognition result; A second relationship identification unit, connected to the first relationship identification unit, is used to: jointly use the financial legal knowledge graph and the personalized knowledge graph to verify and match the first entity relationship identification result to obtain a second entity relationship identification result; The entity relationship recognition result includes the first entity relationship recognition result and the second entity relationship recognition result.

4. A financial case information mining system based on a large language model as claimed in claim 2, characterized in that: The human-machine loopback module comprises: An intelligent processing unit deploys the local small model, and is used to use the local small model to analyze the first entity recognition result and the second entity recognition result to obtain a comprehensive entity recognition result; A human-computer interaction unit, connected to the intelligent processing unit, is used to: The entity comprehensive recognition result is displayed, and the entity comprehensive recognition result is manually adjusted to obtain an entity recognition adjustment result; When it is manually considered that there is no need to adjust the entity comprehensive recognition result, confirming the entity comprehensive recognition result as the final entity recognition result; When it is manually considered that the entity recognition adjustment result does not need to be iterated any further, confirming the entity recognition adjustment result as the final entity recognition result; When it is manually considered that the entity recognition adjustment result needs to be iterated again, the entity recognition adjustment result is added to the data as the entity recognition iteration; The intelligent processing unit is also used for: Passing the final result of the entity recognition to the entity relationship recognition module; Passing the entity recognition iteration adding data to the entity recognition module, so that the entity recognition module performs an iterative process; The entity recognition module is also used to: use the financial law large language model and the constructed financial law knowledge graph, combine the entity recognition to iteratively add data, perform entity recognition on the pre-processed data again, and obtain the entity recognition result.

5. A financial case information mining system based on a large language model as claimed in claim 2, characterized in that: The system further comprises a local personalized word library, wherein the local personalized word library is used to store the collected personalized knowledge data, wherein the personalized knowledge data is financial related data within a predetermined range; The human-machine loopback module is connected to the local personalized word library and includes: The intelligent processing unit deploys the local small model, and is used to: use the local small model and the local personalized vocabulary to analyze the first entity recognition result to obtain a third entity recognition result, and provide an entity comprehensive recognition result based on the first entity recognition result, the second entity recognition result and the third entity recognition result; The human-computer interaction unit is connected to the intelligent processing unit and is used to: display the entity comprehensive recognition result, and manually determine the entity recognition final result based on the entity comprehensive recognition result.

6. A financial case information mining system based on a large language model as claimed in claim 3, characterized in that: The human-machine loopback module comprises: An intelligent processing unit deploys the local small model, and is used to use the local small model to analyze the first entity relationship recognition result and the second entity relationship recognition result to obtain a comprehensive entity relationship recognition result; A human-computer interaction unit, connected to the intelligent processing unit, is used to: Displaying the entity relationship comprehensive recognition result, and manually adjusting the entity relationship comprehensive recognition result to obtain an entity relationship recognition adjustment result; When it is considered that there is no need to manually adjust the entity relationship comprehensive recognition result, confirming the entity relationship comprehensive recognition result as the entity relationship recognition final result; When it is manually considered that the entity relationship recognition adjustment result does not need to be iterated any further, confirming the entity relationship recognition adjustment result as the entity relationship recognition final result; When it is manually considered that the entity relationship recognition adjustment result needs to be iterated again, the entity relationship recognition adjustment result is added as entity relationship recognition iteration data; The intelligent processing unit is also used for: Passing the final result of entity relationship recognition to the result output display module; Passing the entity relationship recognition iteration adding data to the entity relationship recognition module, so that the entity relationship recognition module performs an iterative process; The entity relationship recognition module is also used to: use the financial law large language model and the constructed financial law knowledge graph, combine the entity relationship recognition iterative addition data and the final result of the entity recognition, perform entity relationship recognition on the pre-processed data, and obtain the entity relationship recognition result.

7. A financial case information mining system based on a large language model as claimed in claim 3, characterized in that: The system further comprises a local personalized word library, wherein the local personalized word library is used to store the collected personalized knowledge data, wherein the personalized knowledge data is financial related data within a predetermined range; The human-machine loopback module is connected to the local personalized word library and includes: The intelligent processing unit deploys the local small model, and is used to: use the local small model and the local personalized vocabulary to analyze the first entity relationship recognition result to obtain a third entity relationship recognition result, and give an entity relationship comprehensive recognition result based on the first entity relationship recognition result, the second entity relationship recognition result and the third entity relationship recognition result; The human-computer interaction unit is connected to the intelligent processing unit and is used to: display the entity relationship comprehensive recognition result, and manually determine the entity relationship recognition final result based on the entity relationship comprehensive recognition result.

8. A financial case information mining system based on a large language model as claimed in claim 3, characterized in that: The system further comprises: The word library management module is connected to the local personalized word library and the personalized atlas library respectively, and is used to: Updating the personalized knowledge data in the local personalized vocabulary; Constructing the personalized knowledge graph based on the personalized knowledge data, and updating the personalized knowledge graph according to the updated personalized knowledge data; The prompt word engineering template of the financial law large language model is optimized based on the local personalized word library.

9. A financial case information mining system based on a large language model as claimed in claim 8, characterized in that: The personalized knowledge data includes financial related data extracted from financial cases that have been tried and judged by the court; The first relationship identification unit includes: A second search subunit is used to use the financial law large language model to search for financial related data of similar financial cases from the personalized vocabulary according to the final result of entity recognition and the pre-processed data, so as to obtain a second search result; The relationship identification subunit is connected to the second retrieval subunit and is used to: use the financial legal large language model to obtain the first entity relationship identification result according to the final result of the entity identification, the preprocessed data and the second retrieval result.

10. A financial case information mining method based on a large language model, characterized in that: A financial case information mining system based on a large language model as described in any one of claims 1 to 9, comprising: Step A1, inputting the financial case files to be mined, and preprocessing the financial case files to be mined using the fine-tuned financial law language model to obtain preprocessed data; Step A2, using the financial law big language model and the constructed financial law knowledge graph to perform entity recognition on the pre-processed data to obtain an entity recognition result; Step A3, using a human-machine loopback module to intelligently analyze the entity relationship recognition result, and manually determine the final result of the entity relationship recognition; Step A4, using the financial law large language model and the financial law knowledge graph, and according to the final result of the entity recognition, performing entity relationship recognition on the preprocessed data to obtain an entity relationship recognition result; Step A5, using the human-machine loopback module to intelligently analyze the entity relationship recognition result, and manually determine the final result of the entity relationship recognition; Step A6, jointly displaying the final result of the entity recognition and the final result of the entity relationship recognition.

Images