Automatic analysis method of foreign-related rule of law information text based on retrieval-enhanced thinking chain

Abstract

The present invention discloses an automatic analysis method for foreign-related rule of law information text based on retrieval-enhanced thought chain, innovatively combines retrieval-enhanced generation (RAG) with large language model thought chain (COT), performs multiple LLM model calls according to the number of evaluation elements, and repeatedly executes the retrieval and thought chain iterative correction process until the foreign-related rule of law risk subjects, objects, behaviors, and events related to the final revised evaluation element em* are extracted; executes LLM calls, and extracts the foreign-related rule of law risk subjects, objects, behaviors, and events related to the final revised evaluation element em* from all revised evaluation elements [e1*, e2*,..., e m *] Re-extract the relevant foreign-related rule of law risk subjects, objects, behaviors, and event information for synthesis, and obtain the importance classification evaluation of the foreign-related rule of law information according to the importance classification rules of foreign-related rule of law. Compared with the prior art, the present invention improves the reasoning ability of large-scale language models in dealing with complex legal risk control issues.

Description

Technical Field

[0001] The present invention relates to the field of search-enhanced thinking chain and computer analysis technology, and in particular to an automatic analysis method of foreign-related laws based on search-enhanced thinking chain. Background Art

[0002] At present, the classification of the policy importance of foreign-related rule of law mainly relies on experts and manual work, and the commonly used method is factor analysis. Factor analysis uses the experience and in-depth domain knowledge of experts to identify and evaluate common factors that may lead to legal problems. The method first involves collecting information related to corporate operations, business activities and the legal environment, and then creating a list of potential factors of concern and assigning weights to these factors through discussion, questionnaires or analysis. The team will then analyze the company's specific business activities, identify related risks, and quantify them according to their probability of occurrence and their weights.

[0003] The advantages of risk factor analysis methods that rely on manual work include its simplicity and intuitiveness and its convenience in quickly identifying major risks. However, the subjective judgment of experts plays a key role in the evaluation of factor weights, which also implies the influence of human factors and the heavy reliance on expert knowledge. At the same time, this method may ignore unknown risk factors, resulting in certain limitations. Traditional assessment of the importance of rule of law information usually relies on the empirical analysis of legal experts, but this method is time-consuming and easily interfered by subjective factors, and the accuracy and real-time nature of its results are often questioned.

[0004] Explanation of relevant terms:

[0005] [Retrieval-Augmented Generation] Retrieval-Augmented Generation (RAG) appears in the field of text-to-text generation. The model first retrieves relevant information from a large corpus and then uses this information to guide the text generation process. This method can improve the quality and accuracy of generated text, especially when dealing with tasks that require specific domain knowledge. A typical RAG process is as follows: given an input retrieval, the retriever locates and finds relevant data sources, and interacts the retrieval results with the generator to enhance the entire generation process.

[0006] [Chain-of-thought Tips] Chain-of-thought (CoT) is a technology that allows large models to explicitly output intermediate steps during the reasoning process to improve their reasoning ability. This concept refers to the way humans solve complex problems, breaking down the problem into a series of reasoning steps in the form of natural language until the final conclusion is reached. Specifically, the goal of the chain of thought is to allow large models to explicitly output the intermediate step-by-step reasoning steps before outputting the final answer, thereby enhancing their conceptual understanding and reasoning ability. This is of great significance for solving tasks that require precise reasoning, such as mathematical problems, common sense reasoning, etc. Through the chain of thought, large models can get closer to the way humans think when solving problems, thereby improving their intelligence level.

[0007] [Generative Language Model] A Generative Language Model (GLM) is a deep learning model that uses a large amount of unlabeled text or labeled text for training to learn natural language processing tasks. These models learn general representations of language in a pre-training phase, which allows them to be used for a wide range of natural language processing tasks without requiring too much labeled data. GPT3 and ChatGPT are representative works of generative language models, which aim to simulate human natural language processing capabilities by using neural network models for text generation and processing. By using these models, users can have more natural and realistic conversations with computers and implement natural language processing tasks such as speech recognition, text summarization, machine translation, text classification, computer code generation, movie script creation, etc.

[0008] [Vector database] A vector database is a database used to store retrieval and analysis vectors. It is different from traditional databases. Traditional database retrieval can usually be attributed to point search and range search. Both point search and range search are precise searches, that is, the results obtained by retrieval either meet the conditions or do not meet the conditions. The vector search of a vector database is usually an approximate search, that is, searching for results that are close to the search conditions, that is, the results obtained by retrieval are the most similar to the input conditions. Summary of the invention

[0009] In view of the defects of the prior art, the present invention aims to propose an automatic analysis method of foreign-related rule of law information text based on retrieval enhanced thinking chain, innovatively combining retrieval enhanced generation (RAG) with large language model thinking chain (CoT), thereby realizing an intelligent analysis method of foreign-related legal text information features based on retrieval enhanced thinking chain, and using large language models to handle complex legal risk control issues.

[0010] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0011] A method for automatically analyzing foreign-related rule of law information text based on retrieval-enhanced thinking chain, comprising the following steps:

[0012] Perform a search and iterative revision process that includes:

[0013] The large language model is called for the first time to generate the initial thought chain T1, which is used as the zero-sample thought chain of the importance of foreign-related rule of law information. The foreign-related rule of law information element database is searched with the preliminary reasoning chain T1 as the retrieval condition, and the matching evaluation elements [e1, e2, ..., e m ]’s search results [r1,r2,...,rx];

[0014] Call the LLM model for the second time, and modify and enhance the initial thinking chain T1 based on the search result [r1, r2, ..., rx] to obtain a revised thinking chain T1*;

[0015] The LLM model is called for the third time to extract the subjects, objects, behaviors and events of foreign-related rule of law risks related to the e1* assessment factor; the vector database is searched to obtain the text segments with similar semantics to the next revised assessment factor e2* as the relevant text segments of the next revised assessment factor e2*;

[0016] The fourth call of the LLM model is made to extract the subjects, objects, behaviors and events of foreign-related rule of law risks corresponding to the e2* factor;

[0017] And so on, the retrieval and iterative correction process is performed repeatedly until the LLM model is called for the m+2th time, where m represents the total number of evaluation factors. The final revised evaluation factor em* related to the foreign-related rule of law risk subject, object, behavior, event is extracted;

[0018] Execute the m+3th LLM call, and select all the corrected evaluation elements [e1*,e2*,...,e m *] Re-extract the relevant foreign-related rule of law risk subjects, objects, behaviors, and event information for synthesis, and obtain the importance grading assessment of the foreign-related rule of law information according to the importance grading rules of the foreign-related rule of law: Represents the question-based thinking chain in the large language model CoT cot And all the corrected thought chains Generate retrieval to enhance thinking chain answer A2

[0019] Compared with the prior art, the beneficial effects and advantages of the present invention are:

[0020] (1) The retrieval-augmented chain of thoughts method was applied to the importance classification of foreign-related rule of law information. The Chain of Thoughts (CoT) prompt and the Retrieval Augmented Generation (RAG) method were combined to improve the reasoning ability of large language models in dealing with complex issues of legal risk control.

[0021] (2) By iteratively leveraging the retrieved external knowledge to revise each thinking step, the accuracy and reliability of the generated results are improved compared to revising the entire thinking chain at one time.

[0022] (3) The adopted zero-shot cueing strategy does not need to be customized for a specific task, demonstrating its generalization ability in open-domain tasks;

[0023] (4) It shows great potential in reducing false information (known as “hallucination”) in the model generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the method for automatic analysis of foreign-related rule of law information text based on retrieval-enhanced thinking chain of the present invention. DETAILED DESCRIPTION

[0025] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0026] The automatic analysis method of foreign-related rule of law information features based on retrieval-enhanced thinking chain of the present invention includes the following key points:

[0027] (1) We designed a set of retrieval-enhanced thinking chains for automatic analysis of multi-dimensional foreign-related rule of law information features, and realized the automatic analysis process of "associate and revise the list of focus factors -> recall relevant texts for each focus point -> extract key information elements for a single focus point -> summarize the results of multiple evaluation points for comprehensive analysis".

[0028] (2) A foreign-related rule of law importance grading system based on information elements was constructed, which enabled the analysis of foreign-related rule of law information characteristics to be factorized and quantified. The system also innovatively combined thought chain prompts and search enhancement generation technology to achieve a grading and classification of the importance of foreign-related rule of law information.

[0029] (3) The introduction of a zero-sample rule of law information importance assessment thinking chain: Generating a thinking chain of the importance of rule of law information without a reference sample helps to process legal information in different fields and makes the model's analytical ability more generalized.

[0030] (4) The generated results of the large language model were subjected to multiple rounds of retrieval and iterative correction, thereby improving the accuracy of extracting key information from long texts of foreign-related rule of law information and the accuracy of text information classification.

[0031] Example:

[0032] The present invention innovatively combines retrieval enhanced generation (RAG) with large language model thinking chain (COT) to form a foreign-related rule of law information feature analysis method based on retrieval enhanced thinking chain. The following is a formal definition of this process.

[0033] First, the retrieval process and generation process in the retrieval-augmented generation technology (RAG) are combined to generate an enhanced language model for the automatic analysis task of foreign-related rule of law. The performance of the large language model for the automatic analysis task of foreign-related rule of law information is increased through the generation process. The formal definition of this step is as follows:

[0034] For a given original search q, firstly, the relevant foreign-related rule of law electronic document sequences {D1, D2, ..., D K} to search, where D K is any set of foreign-related rule of law electronic record documents; then, any foreign-related rule of law electronic record document D of the retrieved K foreign-related rule of law electronic record documents K Together with the original search q, it is used as the input of the large language enhancement model for the automatic analysis of foreign-related rule of law information features. The search results are interacted with the generator to obtain the generation process answer a gen Therefore, the generation process of RAG is expressed by the probability P1(A1|q) of generating the retrieval-enhanced answer A1 based on the original retrieval q, as shown below:

[0035]

[0036] Among them, P ret (D K |q) is based on the original search q to retrieve document D K The probability, P gen (a gen |q,D K ) is based on the original search q and any foreign-related rule of law electronic record document D K Get the generation process answer a gen probability;

[0037] Based on the Chain of Thought (CoT) technology, a chain of thought is generated, and according to a series of logics of the chain of thought, the problem of automatic analysis of foreign-related rule of law information characteristics is gradually solved; the formal definition of this step is as follows:

[0038] For the given chain of thought question cot , based on the chain of thought (CoT) technology, a chain of thought {t1, t2, ..., t n}, t n It is any logical step in the n logical steps of the problem-solving process, generating a thinking chain answer a CoT Therefore, we use the question based on the chain of thought. cot The probability P of generating the answer A2 of the entire chain of thought cot (a|q) describes the CoT generation process as shown in the following formula:

[0039] P cot (A2|question cot )=P cot (t1, t2, ..., t n , a CoT |question cot )

[0040] Among them, P cot (t1, t2, ..., t n , a CoT |question cot ) is based on the question of thought chain cot Generate a thought chain {t1, t2, ..., t n} and the corresponding thinking chain answers a CoT probability;

[0041] In each logical step of CoT n Add to RAG search results based on the current question RACT and the previous thought chain {t1, t2, ..., t n-1}The retrieved document set D K Correct any thought process step n The output of the new algorithm combining retrieval enhancement (RAG) and chain of thought (CoT) is named Retrieval-Augmented Chain of Thought (RACT); the formal definition of this step is as follows:

[0042] Corresponding to each thinking step tn given retrieval enhancement thinking chain question question RACT First, based on the question RACT And the previous chain of thought {t1, t2, ..., t n-1} Perform a retrieval operation to retrieve the document set D that is most relevant to the current state of mind from the K knowledge bases i(i∈K), the specific process is: each document is vectorized in advance and stored in the vector database, the Euclidean distance between the current question and the previous thought chain vector is calculated, and the document set D with the closest Euclidean distance is selected. i ;

[0043] Then, using the most relevant document set D retrieved i To enhance thinking steps i Generation of question chaining with retrieval-based enhancement RACT The probability P of generating the answer A2 of the retrieval-enhanced thinking chain RACT (A2|question RACT ) describes the RACT process as shown below:

[0044]

[0045] Among them, D i Indicates the i-th question based on the current question retrieval to enhance the thinking chain question RACT And the previous chain of thought {t1, t2, ..., t n-1 The most relevant document set retrieved, P ret (D i |question RACT , t1, t2, ..., t n-1 ) indicates that in a given retrieval enhancement thinking chain question question RACT And the previous chain of thought {t1, t2, ..., t n-1} to retrieve the most relevant document set D i The probability, P cot (t i |question RACT , t1, t2, ..., t n-1 , D i ) is to search for enhanced thinking chain questions in a given question question RACT , the previous thought chain {t1, t2, ..., t n-1} and the most relevant document set D retrieved i Generate the current thinking step t i The probability, finally, P cot (a|question RACT , t1, t2, ..., t n ) is given in the question RACT and the complete chain of thoughts {t1, t2, ..., t n}, the probability of generating the retrieval enhanced thinking chain answer A2 (that is, the final answer) in the case of}. Through this design, the RACT algorithm can make full use of the most relevant information to the current problem-solving status in each step of the thinking process, thereby improving the ability and efficiency of solving complex problems.

[0046] like Figure 1 As shown, it is an algorithm architecture diagram of the automatic analysis method of foreign-related rule of law information features based on retrieval enhanced thinking chain of the present invention. The basic data involved in the algorithm include the foreign-related rule of law information in the form of long text, which is obtained by dividing the article into blocks and short text into segments and then performing text vectorization processing, and the foreign-related rule of law element library. The specific description of the algorithm architecture is as follows:

[0047] Step 1, execute the P1 instruction, make the first call (LLM call 1) of the large language model (LLM model) to generate the initial thinking chain T1, use the initial thinking chain T1 as the zero-sample thinking chain of the importance of foreign-related rule of law information for preliminary reasoning, where T1 contains multiple evaluation elements [e1, e2, ..., em], use the preliminary reasoning chain T1 as the search condition to search the foreign-related rule of law information element database and obtain the matching evaluation elements [e1, e2, ..., e m ] of the search results [r1, r2, ..., rx]; the generation basis of the initial thought chain T1 is P cot (t1, t2, ..., t n ,aCo T |question cot ), which is equivalent to the preliminary thinking chain in RACT; specifically, the foreign-related rule of law information includes but is not limited to foreign laws, international conventions, treaties, legislative drafts, think tank research reports and other information; the rule of law elements include but are not limited to the subjects, objects, behavior patterns, rights, responsibilities of laws, regulations, conventions, treaties, as well as core concepts in related fields and their relationships.

[0048] Step 2, execute the P2 instruction, perform the second call of the LLM model (LLM call 2), modify and enhance the initial thought chain T1 generated by the LLM (large language model) based on the search results [r1, r2, ..., rx], and obtain the revised thought chain T1*, wherein T1* contains the revised evaluation elements [e1*, e2*, ..., em*], and retrieve the text segments with semantic similarity to the revised evaluation element e1* from the vector database as the e1* related text segments;

[0049] Step 3: Execute the P3 instruction to call the LLM model for the third time (LLM call 3), extract the subjects, objects, behaviors and events of foreign-related rule of law risks related to the e1* factor; search the vector database to obtain the text segment with similar semantics to the next revised assessment factor e2* as the relevant text segment of the next revised assessment factor e2*;

[0050] Step 4, execute the P4 instruction, make the fourth call of the LLM model (LLM call 4), and extract the foreign-related rule of law risk subjects, objects, behaviors, and events corresponding to the e2* element;

[0051] Step 5, and so on, repeatedly perform the retrieval and iterative correction process of steps 2 to 4 until the Pm+2 instruction is executed, and the LLM model is called for the m+2th time (LLM calls m+2) to extract the foreign-related rule of law risk subjects, objects, behaviors, and events related to the final revised assessment factor em*; for the foreign-related rule of law information of the long text, segment it into n short texts, embed it into vectorization, and store it in the vector database;

[0052] For each inference step t n Perform a search operation to retrieve the document set D that is most relevant to the current state of mind from the foreign-related rule of law element database (or text database) i = {r1, r2, ..., rx}, then, use the retrieved information elements to enhance the thinking step t n The generation of a revised chain of thought The formula is as follows:

[0053]

[0054] in, Representation retrieval enhances thinking chain based on question RACT Generate a revised chain of thoughts The probability, P ret (D i |question RACT ,t1,t2,...,t n-1 ) indicates that the retrieval process is based on the question question RACT and all thought steps t1,t2,...,t n-1 Generate the most relevant document set D i probability; Represents the question-based thinking chain in the large language model CoT RACT , all thinking steps t1, t2, ..., t n-1 and the most relevant document set D i Generate a revised chain of thoughts probability;

[0055] Step 6. Finally, execute the Pm+3 instruction to perform the m+3th LLM call (LLM call m+3) and select all the corrected evaluation factors [e1*, e2*, ..., e m *] Re-extract the relevant foreign-related rule of law risk subjects, objects, behaviors, and event information for synthesis, and obtain the importance grading assessment of the foreign-related rule of law information according to the importance grading rules of the foreign-related rule of law: Representing the problem-based queestion in the thinking chain of the large language model CoT cot And all the corrected thought chains Generate retrieval-enhanced thinking chain answer A2.

[0056] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions of the technical solution of the present invention by ordinary technicians in this field do not deviate from the essence and scope of the technical solution of the present invention.

Claims

1. A method for automatic analysis of foreign-related rule of law information text based on retrieval-enhanced thinking chain, characterized in that: The following steps are involved: Perform a search and iterative revision process that includes: The large language model is called for the first time to generate the initial thought chain T1, which is used as the zero-sample thought chain of the importance of foreign-related rule of law information. The foreign-related rule of law information element database is searched using the initial thought chain T1 as the search condition to obtain the matching evaluation elements [e1, e2, ..., e m ]’s search results [r1,r2,...,rx]; Call the LLM model for the second time, and modify and enhance the initial thinking chain T1 based on the search results [r1, r2, ..., rx] to obtain a revised thinking chain T1*, wherein T1* contains revised evaluation elements [e1*, e2*, ..., em*]; The LLM model is called for the third time to extract the subjects, objects, behaviors and events of foreign-related rule of law risks related to the assessment factor e1*; the vector database is searched to obtain the text segments with similar semantics to the next revised assessment factor e2* as the relevant text segments of the next revised assessment factor e2*; The fourth call of the LLM model is made to extract the subjects, objects, behaviors and events of foreign-related rule of law risks related to the assessment factor e2*; In this way, the above retrieval and iterative correction process is repeatedly performed until the LLM model is called for the m+2th time, where m represents the total number of evaluation factors, and the last revised evaluation factor em* related to the foreign-related rule of law risk subject, object, behavior and event is extracted; Execute the m+3th LLM call, and select all the corrected evaluation elements [e1*,e2*,...,e m *] extract relevant foreign-related rule of law risk subjects, objects, behaviors and event information for integration, and obtain the importance grading assessment of the foreign-related rule of law information according to the importance grading rules of foreign-related rule of law: Represents the question-based thinking chain in the large language model CoT cot And all the corrected thought chains Generate retrieval-enhanced thinking chain answer A2.

2. According to claim 1, a method for automatic analysis of foreign-related legal information text based on retrieval-enhanced thinking chain, characterized in that: The revised thought chain T1* contains revised evaluation elements [e1*, e2*, ..., em*], and the search vector database is used to obtain text segments with similar semantics to the revised evaluation element e1* as relevant text segments of the evaluation element e1*.

3. According to claim 1, the method for automatic analysis of foreign-related legal information text based on retrieval-enhanced thinking chain is characterized in that: The method further comprises: Use the retrieved information elements to enhance the thinking step n The generation of a revised chain of thought The formula is as follows: in, Representation retrieval enhances thinking chain based on question RACT Generate a revised chain of thoughts The probability, P ret (D i |question RACT ,t1,t2,...,t n-1 ) indicates that the retrieval process is based on the question question RACT and all thought steps t1,t2,...,t n-1 Generate the most relevant document set D i probability; Represents the questionR in the large language model thinking chain CoT ACT , all thinking steps t1, t2, ..., t n-1 and the most relevant document set D i Generate a revised chain of thoughts probability.

4. According to claim 3, the method for automatic analysis of foreign-related rule of law information text based on retrieval-enhanced thought chain is characterized in that: Get the most relevant document set D i The specific process is as follows: each document is vectorized in advance and stored in the vector database, the Euclidean distance between the current question and the previous thought chain vector is calculated, and the document set D with the closest Euclidean distance is selected. i .

Images