Large model RAG retrieval method and system oriented to judicial field

Abstract

The invention relates to a large model RAG retrieval method and system oriented to the judicial field, belongs to the technical field of natural language processing, deep learning, large language model and retrieval enhancement generation in artificial intelligence, and is used for enhancing the ability of a large model to retrieve accurate legal texts. According to the method, a large language model is finely adjusted by using instruction adjustment data generated without supervision, and aiming at the specific requirement of a retrieval enhancement generation model (RAG) in the legal field, the attention capability of the RAG on documents with high correlation in legal document retrieval and content generation is enhanced by optimizing the processing step of the model, so that the legal document retrieval and content generation efficiency is improved. Therefore, the performance and the accuracy of the model in the professional field of laws are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a large-model RAG retrieval method and system for the judicial field, belonging to the technical field of natural language processing, deep learning, large-scale language models and retrieval enhancement generation in artificial intelligence. Background Art

[0002] With the increasing contradiction between too many cases and too few judges, how to use technology to complete legal tasks and improve the work efficiency and quality of judicial personnel has attracted widespread attention from scholars in the fields of computer science and law. As a knowledge-intensive professional field, judicial tasks place high demands on models to understand and apply legal knowledge. The complex knowledge reasoning involved in legal case analysis poses a challenge to existing artificial intelligence models. In artificial intelligence systems applied in the judicial field, natural language processing (NLP) technology is a core component, which enables computers to understand and analyze complex language structures in legal documents. Early technologies focused on characterizing and learning case facts, representing cases as legal element vectors, N-grams vectors, etc. Such methods are difficult to capture the deep semantics in legal documents. With the rapid development of neural network algorithms, using deep learning methods to learn distributed semantic representations of cases has gradually become a hot topic and basic paradigm in legal intelligence research. However, these methods often focus on single tasks and charges, and have poor generalization and universality.

[0003] Since the advent of ChatGPT, large language models (abbreviated as large models) have become a new paradigm for natural language processing. Inspired by open domain research, many scholars have begun to pay attention to the training and application of judicial pre-trained language models. Early work mainly attempted to migrate general domain pre-trained models to the judicial field for direct application. Later, in response to the lack of legal knowledge in large models, a large amount of corpus such as judicial documents, laws and regulations, and legal books was used to conduct secondary pre-training and fine-tuning of large models, thereby constructing legal large models such as ChatLaw and Lawyer-LLaMA. However, such models have problems such as uncontrollable output. In order to better serve judicial practice, retrieval augmented generation (RAG) technology was introduced into large language models. RAG combines the advantages of information retrieval technology and large models, and can retrieve relevant information from external knowledge sources before generating answers to ensure that the output content is accurate and reliable. This method is highly applicable in the legal field. Through the retrieved specific legal provisions and similar cases, the RAG model can provide more accurate legal services by integrating such information. However, the highly specialized legal industry requires accuracy and relevance in text processing, as well as the unique properties of legal texts, such as the formality of terms, the complexity of structures, and the abstractness of concepts, which increase the difficulty of retrieving the required text. Therefore, developing a technical method that can effectively improve the capabilities of large-scale RAG models is of great significance to improving the efficiency of text automation processing and information retrieval in the legal field. Summary of the invention

[0004] In the current judicial field big model RAG method, due to the highly specialized legal industry's requirements for accuracy and relevance of text processing, as well as the unique attributes of legal texts, it is very difficult for the big model to retrieve the required legal texts. Therefore, the present invention proposes a big model RAG retrieval method for the judicial field, aiming to enhance the ability of the big model to retrieve reliable and relevant legal texts.

[0005] The technical solution of the present invention is as follows:

[0006] This paper proposes a relevance-focused finetuning (ReFT) method to enhance the ability of large models to retrieve accurate legal texts. This method uses unsupervised generated instruction adjustment data to fine-tune large language models. It optimizes the processing steps of the model to enhance its ability to focus on highly relevant documents in legal document retrieval and content generation, targeting the specific needs of the retrieval-augmented generation model (RAG) in the legal field, thereby significantly improving the performance and accuracy of the model in the legal professional field.

[0007] The overall structure of this program:

[0008] The present invention proposes a technical solution for generating a golden answer using legal knowledge, synthesizing instruction tuning data based on the golden answer, and training a large model based on this batch of data to further improve its performance in the RAG scenario.

[0009] A large-model RAG retrieval method for the judicial field includes the following steps:

[0010] (1) Golden answer generation

[0011] In the process of generating the golden answer, the present invention proposes a mathematical model to calculate the conditional probability output between a given instruction and a specific legal document. This model is based on deep learning technology and is defined by the following expression:

[0012] L(x|d)=softmax(Model(x,d))(1)

[0013] Among them, x is an instruction in a real scenario. Through contextual prompt words, x combines the legal consultation questions raised by the user and the pre-annotated legal documents related to the user's questions. Relevant legal documents refer to relevant documents that may be helpful in answering user questions. The difference from golden label documents is that they have a large amount of information and most documents have little reference value for answering user questions; d refers to a small number of golden label documents carefully selected and annotated by professionals. Golden label documents include all the knowledge based on which the large model processes the user's specific instructions, that is, the large model can correctly answer user questions based on golden label documents; golden label documents are screened and annotated by experts or experienced annotators to ensure that they contain key information and knowledge to solve specific instructions. Preferably, in the present invention, relevant legal documents and golden label documents are selected and annotated from legal theories, case judgments, legal provisions, legal interpretations and judicial documents. Golden label documents provide an authoritative and accurate source of information for the model.

[0014] The function Model(x, d) is a deep learning model. The present invention uses a large language model based on the Transformer decoder architecture to describe the process of how to generate answers based on the input x and the document d. The softmax function can be used to convert the predicted value output by the model into a probability distribution, so that each possible output answer corresponds to a probability value.

[0015]

[0016] Where y i For the model Model(x i,d) Prediction value, i is the input number, and finally, the model outputs the answer with the highest probability as the prediction result; that is, the input of the model Model is comprehensive information including user instructions and golden label documents. This design enables the model to accurately understand and answer questions in a specific legal context; the model output is the golden label answer, which is a verified high-quality answer that can be directly applied to subsequent legal analysis or decision support. By integrating a huge legal knowledge base, including but not limited to regulations, laws and their interpretations, this technology can ensure the accuracy of the answer while ensuring that the answer is in-depth and in line with legal logic.

[0017] (2) Instruction fine-tuning data synthesis

[0018] Synthesizing the current instruction data requires integrating three parts, namely: 1) Initial instructions, which are specific questions raised by the user, and the scope of the question is manually marked. The scope of the question includes the scope of the laws involved, including two major categories: criminal and civil. Civil issues are divided into eight categories: "personality rights disputes", "wedding banquet family, inheritance disputes", "property rights disputes", "contracts, quasi-contract disputes", "intellectual property and competition disputes", "labor disputes and personnel disputes", "civil disputes related to companies, securities, insurance, bills, etc." and "tort liability disputes"; 2) Golden label documents and Top N search documents. The search documents are the top N relevant legal documents retrieved by the BM25 search algorithm according to the user's initial instructions. The golden label documents are the best reference documents manually selected by experts based on the user's questions and markings; 3) Generated golden label answers, which are the answers generated by the large model in step (1) based on the user's instructions and golden label documents;

[0019] In the instruction fine-tuning data synthesis, the information is divided into two parts, namely input and output; the input part includes the initial instruction, the Top N retrieved documents obtained by the retrieval model based on this instruction, and the gold-label document. The three are combined through a prompt word. The prompt word is "Please refer to the following document question user answer" followed by the initial instruction, followed by the prompt word "Related documents:", followed by the retrieved document and the gold-label document. The gold-label document is randomly inserted into the Top N retrieved document sequence; the output is the gold-label answer generated by the model in step (1); the purpose of this step is to let the large model learn how to generate high-quality answers based on the gold-label document from the Top N documents returned by the retrieval model.

[0020] After the model is trained, in actual applications, the present invention no longer manually selects gold-label documents, but generates more accurate answers based on the characteristics of the Top N documents that are learned through training and pay more attention to gold-label documents. The purpose of training the large model in the present invention is to enable the large model to select documents with higher reference value from the numerous retrieved documents, and generate more accurate answers with the help of these documents. This method not only improves the flexibility and adaptability of the model in actual applications, but also greatly enhances the reliability and accuracy of the answers.

[0021] (3) Fine-tuning of large models

[0022] In the instruction adjustment training based on retrieval enhancement, the goal is to train the large model to generate high-quality answers from the combination of the top N retrieval documents returned by the retrieval model and the manually annotated gold-label documents; the training goal is expressed by the following mathematical formula:

[0023] minmize L(θ)=-log p (y|x,C(x);θ) (3)

[0024] Among them, y is the gold standard answer, x is the initial instruction input by the user, C(x) is the combination of the retrieved Top N search documents and the gold standard documents, θ is the model parameter, and L(θ) is the loss function; the optimization goal is to improve the accuracy of the model in extracting relevant information and generating answers in actual situations by minimizing the negative log-likelihood of the predicted output under given input and context conditions. In practice, a parameter-efficient fine-tuning method is used to train the large model. Specifically, the LoRA method is used. This method is a common method for fine-tuning large models. Additional parameters are introduced in each layer of the Transformer large model. Only the additional parameters are updated during training to achieve the fine-tuning purpose, and the initial parameters of the large model are not changed.

[0025] Step (1) records the model output as the golden answer; Step (2) aims to construct the paired data required for large model training, with the input being the user's initial instructions, the retrieved top documents and the golden document, and the output being the golden answer; Step (3) trains the large model on the data in step (2), with the aim of making the large model highly sensitive to the golden document and enhancing the large model's ability to answer based on the most relevant document from multiple retrieved documents.

[0026] (4) After steps (1) to (3) are completed, a trained RAG enhanced large model is obtained. When answering user questions, the BM25 retrieval algorithm can be used to automatically retrieve the most matching document from multiple retrieved related documents for the large model to provide answers, effectively improving the accuracy of the answers.

[0027] A large-model RAG retrieval system for the judicial field includes an answer generation module, a data synthesis module, and a model fine-tuning module. The answer generation module calculates the conditional probability output between a given instruction and a specific legal document through a mathematical model, and the answer with the highest model output probability is used as the prediction result, which is the golden answer; the data synthesis module integrates and synthesizes the initial instruction, the golden document, the Top N search documents, and the generated golden answer by using prompt word combination; the model fine-tuning module adjusts the training based on the retrieval-enhanced instruction, trains the large model to generate high-quality answers from the combination of the Top N search documents returned by the retrieval model and the manually annotated golden documents, and adopts an efficient parameter fine-tuning method to train the large model to obtain a trained RAG-enhanced large model, which is used to answer user questions.

[0028] The purpose of the present invention is to improve the performance of large-model RAG. In-depth RAG methods do not only use RAG technology; the method of the present invention to improve the performance of large-model RAG is to construct a pair of gold-standard documents and gold-standard answer training data, and add multiple related documents as noise, so as to prompt the large model to learn and pay attention to the gold-standard documents. The method proposed by the present invention is aimed at the problem of incompatibility between the retrieval algorithm and the large-model RAG method, which can be effectively alleviated by this method.

[0029] The beneficial effects of the present invention are:

[0030] The present invention can effectively improve the relevance and accuracy of legal texts generated by large models and improve the efficiency of legal information retrieval systems. By inserting gold-label documents to construct a training data set, the large model can effectively improve the sensitivity to key information in the retrieved documents and accurately answer user questions based on these key information. In addition, by extracting and utilizing information in complex legal texts, the method reduces dependence on professionals, reduces time costs and the possibility of operational errors. By improving the model's depth of understanding of documents, the method makes the generated legal text not only correct in form, but also highly consistent with the retrieved legal knowledge or instructions in content. The method is applicable to a variety of large model application scenarios, such as similar case retrieval, law prediction, judgment prediction and contract review. In practical applications, the method of the present invention effectively improves the application effect of large model RAG; there is no need to mark gold-label documents, and the trained model can automatically focus on the most suitable key documents in multiple retrieved documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but is not limited thereto.

[0033] Embodiment 1:

[0034] A large-model RAG retrieval method for the judicial field includes the following steps:

[0035] (1) Golden answer generation

[0036] Collect consultation questions from users in real scenarios. Data can be collected through legal consultation websites, user surveys, lawyers, courts, etc. The collection target is 1,500 pieces of data, which are the instructions of users in real scenarios.

[0037] For the 1,500 user instructions, relevant legal documents and golden label documents are pre-labeled. Golden label documents are most relevant to user instructions, including laws and regulations, legal interpretations, and case judgments. One instruction can correspond to multiple golden label documents, and multiple golden label answer candidates can be obtained by adjusting different combinations of golden label documents and instructions. In this implementation case, it is simplified to one golden label document for each user instruction.

[0038] In the process of generating the golden answer, the present invention proposes a mathematical model to calculate the conditional probability output between a given instruction and a specific legal document. This model is based on deep learning technology and is defined by the following expression:

[0039] L(x|d)=softmax(Model(x,d))(1)

[0040] Among them, x is an instruction in a real scenario. Through contextual prompt words, x combines the legal consultation questions raised by the user and the pre-annotated legal documents related to the user's questions. Relevant legal documents refer to relevant documents that may be helpful in answering user questions. The difference from golden label documents is that they have a large amount of information and most documents have little reference value for answering user questions; d refers to a small number of golden label documents carefully selected and annotated by professionals. Golden label documents include all the knowledge based on which the large model processes the user's specific instructions, that is, the large model can correctly answer user questions based on golden label documents; golden label documents are screened and annotated by experts or experienced annotators to ensure that they contain key information and knowledge to solve specific instructions. Preferably, in the present invention, relevant legal documents and golden label documents are selected and annotated from legal theories, case judgments, legal provisions, legal interpretations and judicial documents. Golden label documents provide an authoritative and accurate source of information for the model.

[0041] Select the large model (model selection Baichuan2-13B-chat) as Model (x, d), and use the zero-shot method to generate the golden answer, that is, combine the user instruction and the golden document into a large model prompt, and input it to the large model. The prompt format is:

[0042] "To answer user questions based on the documentation:

[0043] Document: ##Gold Label Document Content

[0044] User questions: ##User instructions

[0045] answer:"

[0046] Get the big model's answer and record it.

[0047] The function Model(x, d) is a deep learning model. The present invention uses a large language model based on the Transformer decoder architecture to describe the process of how to generate answers based on the input x and the document d. The softmax function can be used to convert the predicted value output by the model into a probability distribution, so that each possible output answer corresponds to a probability value.

[0048]

[0049] Where y i For the model Model(x i ,d) Prediction value, i is the input number, and finally, the model outputs the answer with the highest probability as the prediction result; that is, the input of the model Model is comprehensive information including user instructions and golden label documents. This design enables the model to accurately understand and answer questions in a specific legal context; the model output is the golden label answer, which is a verified high-quality answer that can be directly applied to subsequent legal analysis or decision support. By integrating a huge legal knowledge base, including but not limited to regulations, laws and their interpretations, this technology can ensure the accuracy of the answer while ensuring that the answer is in-depth and in line with legal logic.

[0050] (2) Instruction fine-tuning data synthesis

[0051] Synthesizing the current instruction data requires integrating three parts, namely: 1) Initial instruction, which is the specific question raised by the user, and the scope of the question is manually marked. The scope of the question includes the scope of the laws involved, including two major categories: criminal and civil. Civil issues are divided into eight categories: "personality rights disputes", "wedding family and inheritance disputes", "property rights disputes", "contract and quasi-contract disputes", "intellectual property and competition disputes", "labor disputes and personnel disputes", "civil disputes related to companies, securities, insurance, bills, etc." and "tort liability disputes"; 2) Golden label documents and Top N search documents. The search documents are the top N relevant legal documents retrieved by the BM25 search algorithm according to the user's initial instructions. The golden label documents are the best reference documents selected by experts based on the user's questions and markings; the search document dataset comes from the collection of Chinese laws and regulations. This implementation case refers to the STARD dataset, and the retrieval model uses the BM25 model; 3) Generated golden label answers, which are the answers generated by the large model in step (1) according to the user's instructions and golden label documents;

[0052] In the instruction fine-tuning data synthesis, the information is divided into two parts, namely input and output; the input part includes the initial instruction, the Top N retrieved documents obtained by the retrieval model based on this instruction, and the gold-label document. The three are combined through a prompt word. The prompt word is "Please refer to the following document question user answer" followed by the initial instruction, followed by the prompt word "Related documents:", followed by the retrieved document and the gold-label document. The gold-label document is randomly inserted into the Top N retrieved document sequence; the output is the gold-label answer generated by the model in step (1); the purpose of this step is to let the large model learn how to generate high-quality answers based on the gold-label document from the Top N documents returned by the retrieval model.

[0053] After the model is trained, in actual applications, the present invention no longer manually selects gold-label documents, but generates more accurate answers based on the characteristics of the Top N documents that are learned through training and pay more attention to gold-label documents. The purpose of training the large model in the present invention is to enable the large model to select documents with higher reference value from the numerous retrieved documents, and generate more accurate answers with the help of these documents. This method not only improves the flexibility and adaptability of the model in actual applications, but also greatly enhances the reliability and accuracy of the answers.

[0054] (3) Fine-tuning of large models

[0055] The large model uses Baichuan2-13B-base. In the instruction adjustment training based on retrieval enhancement, the goal is to train the large model to generate high-quality answers from the combination of the top N retrieval documents returned by the retrieval model and the manually annotated gold-label documents. The training goal is expressed by the following mathematical formula:

[0056] minmize L(θ)=-log p (y|x,C(x);θ) (3)

[0057] Among them, y is the gold standard answer, x is the initial instruction input by the user, C(x) is the combination of the retrieved Top N search documents and the gold standard documents, θ is the model parameter, and L(θ) is the loss function; the optimization goal is to improve the accuracy of the model in extracting relevant information and generating answers in actual situations by minimizing the negative log-likelihood of the predicted output under given input and context conditions. In practice, a parameter-efficient fine-tuning method is used to train the large model. Specifically, the LoRA method is used. This method is a common method for fine-tuning large models. Additional parameters are introduced in each layer of the Transformer large model. Only additional parameters are updated during training to achieve the purpose of fine-tuning, while the initial parameters of the large model are not changed. The training time is 10 epochs, the batch size is set to 16, and other parameters are set to the default settings of the fine-tuning framework. In this implementation example, LLaMA Factory is used as the large model fine-tuning framework, and LoRA is used as the fine-tuning method.

[0058] Step (1) records the model output as the golden answer; Step (2) aims to construct the paired data required for large model training, with the input being the user's initial instructions, the retrieved top documents and the golden document, and the output being the golden answer; Step (3) trains the large model on the data in step (2), with the aim of making the large model highly sensitive to the golden document and enhancing the large model's ability to answer based on the most relevant document from multiple retrieved documents.

[0059] (4) After steps (1) to (3) are completed, a trained RAG enhanced large model is obtained. When answering user questions, the BM25 retrieval algorithm can be used to automatically retrieve the most matching document from multiple retrieved related documents for the large model to provide answers, effectively improving the accuracy of the answers.

[0060] The trained model can be used in practical applications. In practical applications, since there is no golden document, the model will spontaneously judge and select the most relevant document in the Top N as the golden document for answering.

[0061] Example 2

[0062] A large-model RAG retrieval system for the judicial field includes an answer generation module, a data synthesis module, and a model fine-tuning module. The answer generation module calculates the conditional probability output between a given instruction and a specific legal document through a mathematical model, and the answer with the highest model output probability is used as the prediction result, which is the golden answer; the data synthesis module integrates and synthesizes the initial instruction, the golden document, the Top N search documents, and the generated golden answer by using prompt word combination; the model fine-tuning module adjusts the training based on the retrieval-enhanced instruction, trains the large model to generate high-quality answers from the combination of the Top N search documents returned by the retrieval model and the manually annotated golden documents, and adopts an efficient parameter fine-tuning method to train the large model to obtain a trained RAG-enhanced large model, which is used to answer user questions.

Claims

1. A large-model RAG retrieval method for the judicial field, characterized in that: The steps include: (1) Golden answer generation In the process of generating the golden answer, a mathematical model is proposed to calculate the conditional probability output between a given instruction and a specific legal document. This model is based on deep learning technology and is defined by the following expression: L(x|d)=softmax(Model(x,d))(1) Among them, x is the instruction in the real scenario. Through the context prompt word method, x combines the legal consultation question raised by the user and the pre-annotated legal documents related to the user's question; d refers to the manually selected and annotated golden label document, which includes all the knowledge based on which the big model processes the user's specific instructions, that is, the big model correctly answers the user's question based on the golden label document; The function Model(x, d) is a deep learning model that uses a large language model based on the Transformer decoder architecture to describe the process of generating answers based on input x and document d. The softmax function can convert the predicted value of the model output into a probability distribution, so that each possible output answer corresponds to a probability value. Where y i For the model Model(x i ,d) Prediction value, i is the input number, and finally, the model outputs the answer with the highest probability as the prediction result; (2) Instruction fine-tuning data synthesis Synthesizing the current instruction data requires integrating three parts, namely: 1) Initial instruction, which is the specific question raised by the user, and the scope of the question is manually marked. The scope of the question includes the scope of the laws involved; 2) Golden label documents and Top N search documents. The search documents are the top N relevant legal documents retrieved by the BM25 search algorithm according to the user's initial instruction. The golden label document is the best reference document selected manually based on the user's question and mark; 3) Generated golden label answer, which is the answer generated by the large model in step (1) based on the user's instruction and the golden label document; In the instruction fine-tuning data synthesis, the information is divided into two parts, namely input and output; the input part includes the initial instruction, the top N search documents obtained by the retrieval model based on this instruction, and the gold-label documents. The three are combined through prompt words. The prompt word is "Please refer to the following document question user answer" followed by the initial instruction, and then the prompt word is "Related documents" followed by the search documents and the gold-label documents. The gold-label documents are randomly inserted into the top N search document sequence; the output is the gold-label answer generated by the model in step (1); (3) Fine-tuning of large models In the instruction adjustment training based on retrieval enhancement, the goal is to train the large model to generate high-quality answers from the combination of the top N retrieval documents returned by the retrieval model and the manually annotated gold-label documents; the training goal is expressed by the following mathematical formula: minimize L(θ)=-log p (y|x,C(x);θ) (3) Where y is the gold standard answer, x is the initial instruction input by the user, C(x) is the combination of the retrieved Top N search documents and the gold standard documents, θ is the model parameter, and L(θ) is the loss function. The large model is trained using an efficient parameter fine-tuning method. Specifically, the LoRA method is used to introduce additional parameters to each layer of the Transformer large model. During training, only the additional parameters are updated to achieve the fine-tuning purpose, and the initial parameters of the large model are not changed. (4) After steps (1) to (3) are completed, a trained RAG enhanced large model is obtained. When answering user questions, the model is used to automatically retrieve the most matching document from multiple retrieved relevant documents through the BM25 retrieval algorithm to provide the large model with answers.

2. The large-model RAG retrieval method for the judicial field according to claim 1 is characterized in that: In step (1), relevant legal documents and gold label documents are selected and annotated from legal theories, case judgments, legal provisions, legal interpretations and judicial documents.

3. The large-model RAG retrieval method for the judicial field according to claim 1 is characterized in that: In step (2) of step 1), the scope of the problem includes the scope of laws involved, including criminal and civil categories. Civil issues are divided into eight categories: personal rights disputes, wedding family and inheritance disputes, property rights disputes, contract and quasi-contract disputes, intellectual property and competition disputes, labor disputes and personnel disputes, civil disputes related to companies, securities, insurance, bills, etc., and tort liability disputes.

4. A large-model RAG retrieval system for the judicial field, characterized in that: It includes an answer generation module, a data synthesis module, and a model fine-tuning module. The answer generation module uses a mathematical model to calculate the conditional probability output between a given instruction and a specific legal document. The model outputs the answer with the highest probability as the prediction result, which is the golden answer. The data synthesis module combines the initial instruction, the golden document, the Top N search documents, and the generated golden answer using prompt words. The model fine-tuning module adjusts training based on retrieval enhancement instructions. The training large model generates high-quality answers from the combination of the Top N retrieval documents returned by the retrieval model and the manually annotated gold label documents. The large model is trained using an efficient parameter fine-tuning method to obtain a trained RAG enhanced large model, which is used to answer user questions.

Images