A method and system for dynamic retrieval and enhanced generation of border inspection legal questions and answers
By adopting dynamic search enhancement generation method in the border inspection legal question and answer system, combining ChatGLM2-6B model, QLoRA and Prompt tuning, the problem of low answer quality in the existing technology is solved, and the accurate identification and retrieval of legal knowledge is achieved, and the improvement of legal knowledge is improved. The quality of the generated answers.
Patent Information
- Application Number
- CN202410829490.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-25
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-06-25
AI Technical Summary
In a border inspection legal question-and-answer system based on large language models, the existing technology has hallucinations, the generated answers do not match the facts, and the existing search-reading methods cannot ensure the quality of legal knowledge.
The dynamic search and enhance generation method of border inspection legal Q&A is adopted. By extracting and confidence calculation of the legal elements in the generated answers, dynamic search is performed when the confidence is lower than the threshold, and the parameters are fine-tuned using ChatGLM2-6B model, QLoRA and Prompt tuning. , combining AMR link prediction and intensive vector search model to ensure the quality of legal knowledge of the answers.
Dynamic search with low confidence is achieved, the quality of answers in the legal question and answer system is improved, and the accuracy and relevance of generated content is ensured.
Smart Images

Figure CN118520093B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of applying large language model technology to the technical field of legal question answering, and in particular, relates to a method and system for dynamic retrieval enhancement generation of legal question answering for border inspection. Background Art
[0002] In border inspection legal question-answering based on large language models, large language models generally have hallucination problems when generating answers, that is, the generated content does not match the facts, and sometimes even contains information that is completely irrelevant to the question.
[0003] Therefore, in specific fields, especially in question-answering systems for the legal field, a two-stage "retrieval-reading" approach is usually adopted for implementation. However, the "retrieval-reading" method used in existing legal question-answering systems has structural defects. The entire process must be retrieved once, and the existing method cannot ensure the quality of the legal knowledge in the generated answers. Summary of the invention
[0004] The purpose of this application is to provide a border inspection legal question and answer dynamic retrieval enhancement generation method and system for the border inspection legal field, so as to improve the quality of legal knowledge in the generated answers.
[0005] In order to achieve the above-mentioned purpose, one aspect of the present application provides a method for dynamic retrieval and enhanced generation of legal questions and answers for border inspections, including the following steps: S1: generate answers based on questions; S2: extract legal elements in the generated answers and calculate their confidence; if the confidence is less than a threshold, execute S3, and if the confidence is greater than or equal to the threshold, output the answer; S3: retrieve legal content from a preset knowledge base based on the question; S4: use the retrieved legal content as input information for S1, and execute S1 again to achieve dynamic retrieval and enhanced generation.
[0006] Preferably, S1 further includes: directly generating an answer and using border inspection legal data to generate an answer, and the answer generation formulas are as follows:
[0007] ;
[0008] in, Indicates the generated answer, q is the question, t is the number of times it is generated, t is initially 1, and each time it is generated, t+1. represents the generating function, represents the retrieval function, D is the external legal knowledge, , contains n question-answer pairs.
[0009] Preferably, S1 also includes: using the ChatGLM2-6B model, and training the ChatGLM2-6B model by effectively fine-tuning QLoRA and Prompttuning parameters.
[0010] Preferably, the parameter effective fine-tuning QLoRA method formula is as follows:
[0011] ;
[0012] in, are the frozen pre-trained weights, is the incremental weight.
[0013] Preferably, Prompt tuning provides domain-specific knowledge as input messages to the ChatGLM2-6B model.
[0014] Preferably, S2 also includes: predicting the answer A collection of legal elements , w is the total number of feature categories, and the formula is as follows:
[0015] ;
[0016] in, Represents the border inspection legal element identification function.
[0017] Preferably, S2 further includes: using an AMR-based link prediction method to identify legal elements, the steps of which are as follows:
[0018] S201: Use AMR Parser to convert the answer text into AMR Graph;
[0019] S202: Using AMR-Text Aligner to ensure that the nodes in the AMR graph correspond to the words in the answer text;
[0020] S203: Get answers using the pre-trained RoBERTa model The context representation vector of each word in
[0021] ;
[0022] S204: Calculating Answers in Span ProPosal The contextual representation of a specific word range Span in ,
[0023] The calculation formula is as follows:
[0024] ;
[0025] in, Represents a span in the text. Indicates the span starting vector, Indicates the span end vector, , , is a weight matrix used to combine the contextual representations of different words;
[0026] S205: Use a feedforward neural network (FFNN) to calculate the logit score for each span. The calculation formula is as follows:
[0027] ;
[0028] S206: retain the top 50 spans with the highest scores and represent the context of the retained spans Embedded into AMRgraph to form Span AMR Graph vector, Represents a point in the AMR graph. The formula is as follows:
[0029] ;
[0030] in Refers to the node type lookup table, which includes four types: trigger, surround trigger, candidate span, and others;
[0031] S207: Predict the legal elements that each candidate span plays The formula is as follows:
[0032] ;
[0033] S208: Legal elements Use FFNN to generate logit scores again, the formula is as follows:
[0034] ;
[0035] Get all legal elements set The logit score ;
[0036] S209: Use Minimum to calculate all logit scores as their confidence scores. The formula is as follows:
[0037] .
[0038] Preferably, S3 further includes:
[0039] S301: using a text segmenter to segment an external knowledge document into document pieces in a preset format;
[0040] S302: storing the document piece in a vector database Vector DB in an embedded form;
[0041] S303: The dense vector retrieval model of the encoder architecture is used to retrieve from the Vector DB, and the relevance score is calculated using the dot product. The formula is as follows:
[0042] ;
[0043] in, is the problem vector, is the border inspection legal corpus vector.
[0044] Preferably, when training the retrieval model, cross-batch negative sampling is used to train all paragraphs as negative samples of the current question so that the model learns dense representations of questions and paragraphs. The loss function formula during training is as follows:
[0045] ;
[0046] in, For the problem, is a positive sample of the problem, is the negative sample of m questions;
[0047] On the other hand, the present application provides a border inspection legal question and answer dynamic retrieval enhancement generation system, using the above generation method, including:
[0048] A generation module configured to generate answers to border inspection legal questions using border inspection legal data;
[0049] A border inspection legal element recognition module is configured to extract the legal elements in the generated answer and calculate the confidence level thereof; and
[0050] The retrieval module is configured to retrieve legal content in a preset knowledge base according to the question when the confidence level is lower than a threshold.
[0051] The technical solution provided by this application can achieve the following beneficial effects:
[0052] This application recognizes legal elements and calculates confidence levels, and only enters retrieval at low confidence levels, thereby achieving dynamic retrieval enhancement generation. This solves the structural defects of the "retrieval-reading" method used in existing legal question-and-answer sessions, and can ensure the quality of legal knowledge in the generated answers. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0054] Figure 1 It is a flow chart of the border inspection legal question and answer dynamic retrieval enhancement generation method provided by the embodiment of the present application;
[0055] Figure 2 is a flow chart of a confidence calculation method provided in an embodiment of the present application;
[0056] Figure 3 is a flow chart of the retrieval method provided in the embodiment of the present application;
[0057] Figure 4 It is a schematic diagram of the structure of the border inspection legal question and answer dynamic retrieval enhancement generation system provided by the embodiment of the present application;
[0058] Among them: 101-generation module, 102-border inspection legal element identification module, 103-retrieval module. DETAILED DESCRIPTION
[0059] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0060] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0061] On the one hand, the present application provides a border inspection legal question and answer dynamic retrieval enhancement generation method, such as Figure 1 As shown, the following steps are included:
[0062] S1: Generate answers based on questions.
[0063] Specifically, we generate answers directly and use accurate legal data to generate answers, generating a suitable answer for the border inspection legal question q. , the answer generation formulas are as follows:
[0064] ;
[0065] in, Indicates the generated answer, q is the question, t is the number of times it is generated, t is initially 1, and each time it is generated, t+1. represents the generating function, represents the retrieval function, D is the external legal knowledge, , contains n question-answer pairs.
[0066] S1 also includes: using the ChatGLM2-6B model, using parameters to effectively fine-tune QLoRA and Prompt tuning to train the ChatGLM2-6B model.
[0067] The formula for effectively fine-tuning the parameters of the QLoRA method is as follows:
[0068] ;
[0069] in, are the frozen pre-trained weights, is the incremental weight.
[0070] Specifically, the model is first preliminarily quantized to 4 bits, all parameters are frozen for training, and then low-rank adapter (LoRA) weights are injected into each linear layer of the Transformer. , using the matrix and matrix To approximate , ,in For order, Initialized with Gaussian. , , so that the fine-tuning parameter amount changes from Reduce to , while not changing the dimension of the output data.
[0071] Given a training sample ,in For the problem, A collection of relevant legal texts. Target reply, in order to optimize the adapter parameters , using the maximum target reply The negative log-likelihood is calculated, that is:
[0072] ;
[0073] In this process, the source problem was taken into consideration and relevant legal provisions , to ensure that the generated answer not only conforms to the semantics of the question but also meets the constraints of legal norms.
[0074] For prompt tuning, prompt tuning provides domain-specific knowledge as input messages to the model, similar to short-term memory, so that the model can better understand human instructions. Prompt tuning based on external data is used to fine-tune the pre-trained large language model. Given a problem and several supporting legal documents , use the following prompt form to provide prompts:
[0075] Instructions:...
[0076] Context:...
[0077] Question:...
[0078] Answer:...
[0079] This method guides the model in the form of instructions, questions and legal document D as evidence context information input into the model, which not only improves the adaptability of the model but also better lays the foundation for facts and the latest information.
[0080] S2: Extract the legal elements in the generated answer and calculate its confidence; if the confidence is less than the threshold, execute S3; if the confidence is greater than or equal to the threshold, output the answer.
[0081] Specifically, the answer generated in S1 Automatically extract important legal content and predict the answer A collection of legal elements , w is the total number of feature categories, and the formula is as follows:
[0082] ;
[0083] in, Represents the border inspection legal element identification function.
[0084] Furthermore, the AMR-based link prediction method is used to identify legal elements, such as Figure 2 As shown, the steps are as follows:
[0085] S201: Use AMR Parser to convert the answer text into AMR Graph; S202: Use AMR-TextAligner to ensure that the nodes in the AMR graph correspond to the words in the answer text; S203: Use the pre-trained RoBERTa model to obtain the answer The context representation vector of each word in ; S204: Calculate the answer in Span ProPosal The contextual representation of a specific word range Span in , the calculation formula is as follows:
[0086] ;
[0087] in, Represents a span in the text. Indicates the span starting vector, Indicates the span end vector, , , is a weight matrix used to combine the contextual representations of different words; S205: Use a feedforward neural network (FFNN) to calculate the logit score for each span. The calculation formula is as follows:
[0088] ;
[0089] S206: retain the top 50 spans with the highest scores and represent the context of the retained spans Embedded into AMRgraph to form Span AMR Graph vector, Represents a point in the AMR graph. The formula is as follows:
[0090] ;
[0091] in Refers to the node type lookup table, including trigger, surround trigger, candidate span, and others; S207: predict the legal elements played by each candidate span The formula is as follows:
[0092] ;
[0093] S208: Legal elements Use FFNN to generate logit scores again, the formula is as follows:
[0094] ;
[0095] Get all legal elements set The logit score ; S209: Use Minimum to calculate all logit scores as their confidence scores. The formula is as follows:
[0096] .
[0097] S3: Retrieve legal content from the preset knowledge base based on the question.
[0098] Specifically, only when the confidence score S is lower than the threshold When searching, such as Figure 3 As shown, S301: Use a text segmenter to divide the external knowledge document into document pieces of a preset form, that is, semantically meaningful document pieces of a specified size; S302: Store the document pieces in a vector database Vector DB in an embedded form; S303: Use a dense vector retrieval model with an encoder architecture to retrieve from Vector DB, and use dot product to calculate the relevance score, the formula is as follows:
[0099] ;
[0100] in, is the problem vector, is the border inspection legal corpus vector.
[0101] When training the retrieval model, we use cross-batch negative sampling to train all paragraphs as negative samples of the current question so that the model can learn dense representations of questions and paragraphs. The loss function formula during training is as follows:
[0102] ;
[0103] in, For the problem, is the positive sample of the problem, are negative samples of m questions.
[0104] S4: Use the retrieved legal content as the input information of S1, and execute S1 again to achieve dynamic retrieval enhanced generation.
[0105] On the other hand, the present application provides a border inspection legal question and answer dynamic retrieval enhanced generation system, such as Figure 4 As shown, it includes a generation module, a border inspection legal element recognition module and a retrieval module.
[0106] Specifically, the generation module can be vividly described as a text generation task. The backbone architecture of the generation module adopts the ChatGLM2-6B model with 6.2 billion parameters. QLoRA and Prompt tuning are used to effectively fine-tune the ChatGLM2-6B parameters to construct the generation module. The border inspection legal element recognition module is configured to extract the legal elements in the generated answers and calculate their confidence. The retrieval module is configured to retrieve legal content in the preset knowledge base according to the question when the confidence is lower than the threshold.
[0107] It is understandable that this example provides an experimental environment with the following configuration: operating system: Ubuntu 16.04, CPU: Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz, GPU: Tesla V100S 32GB*6, Python: 3.9, Pytorch: 1.12.1, memory: 500GB.
[0108] The parameter configuration is: for the generation module, for QLoRA, lora rank = 16, r = 8, dropout = 0.1, maximum sequence length = 512, batch size = 16, learning rate = 1e-5, step size = 4; for Prompt tuning, maximum sequence length = 512, batch size = 8, learning rate = 1e-5, step size = 4; for the border inspection legal element recognition module, the batch size is 16, the number of training epochs is 3, the learning rate is 1e-4, and the sequence length is 256; for the retrieval module, the batch size = 64, the learning rate = 3e-5, the step size = 2, the maximum length of the question and answer is 32 and 512 respectively, and k is set to 5.
[0109] The experimental data is as follows: The BorderLaw-QA dataset consists of 1,329 question-answer pairs, as shown in Table 1 below. BorderLaw-QA contains 51 types of questions, and each question contains an average of 26 question-answer pairs. Among them, the question type about making interrogation records contains the most question-answer pairs, which is 114. The average length of each question-answer pair is 287. Each question type contains at least 20 question-answer pairs, and there are 41 such question types.
[0110] Table 1
[0111]
[0112] The JEC-QA dataset is a judicial examination dataset proposed by Tsinghua University. The data comes from the multiple-choice questions of the National Unified Legal Professional Qualification Examination, including 26,365 multiple-choice questions.
[0113] In order to comprehensively evaluate the effectiveness of the enhanced generation method for dynamic retrieval of border inspection laws, evaluation indicators are set. The evaluation indicators at least include generation quality, legal element recognition accuracy and retrieval performance.
[0114] Among them, METEOR is used as the measurement indicator when measuring the generation quality. METEOR combines accuracy, recall and sentence fluency to accurately calculate the similarity between the candidate text and the reference text, and then effectively evaluate the quality of the text generation task; for the evaluation of the accuracy of legal element recognition, F1 value is used as the measurement indicator to ensure the accurate recognition and judgment of legal elements; for retrieval performance, MRR is selected as the measurement indicator. MRR focuses on the position of user demand items in the model recommendation list. The higher the position, the better the retrieval performance.
[0115] Specifically, METEOR is an indicator for evaluating text generation tasks. METEOR calculates the similarity between the candidate text and the reference text based on the word-level precision P and recall R, as well as the penalty Penalty for word order. It considers the precision and recall of the entire corpus, as well as sentence fluency. The larger the indicator, the better. The calculation formula is as follows:
[0116] ;
[0117] ;
[0118] ;
[0119] ;
[0120] ;
[0121] in, is the number of unigrams in the candidate that exist in the reference text, is the number of unigrams in the candidate, is the number of unigrams in the reference translation.
[0122] F1 is an indicator to measure the accuracy of the classification model. It is the harmonic mean of precision and recall. The larger the F1 value, the better. The calculation formula of F1 is as follows:
[0123] ;
[0124] MRR is the average reciprocal ranking, which emphasizes the position of the user's demand item in the model recommendation list. The larger the index, the better. The calculation formula of MRR is as follows:
[0125] ;
[0126] Among them, S is the sample set, |S| is the number of samples, is the predicted ranking of the i-th sample.
[0127] This example provides a generative model baseline experiment. Three generative models are experimented on the experimental data BorderLaw-QA. As shown in Table 2 below, although the number of parameters of ChatGLM2-6B is smaller than that of Baichuan-7B and LLaMA-7B, it has the highest METEOR score. Therefore, ChatGLM2-6B is selected as the generative model.
[0128] Table 2
[0129]
[0130] Threshold Experiments,In the DRAG-BILQA framework, retrieval is performed only at a low threshold. , set multiple on BorderLegal-QA The test shows that the best result is achieved only when the METEOR is relatively high and the search ratio is low. In Table 3 below, decreases, METEOR also decreases, the retrieval ratio decreases, and When the value is 0.70, the METEOR score and retrieval ratio are relatively good, so It is set to 0.70 for subsequent experiments.
[0131] Table 3
[0132]
[0133] This example provides an experiment on the impact of border inspection legal element recognition on generation quality. DRAG-BILQA includes three modules, among which LFRModule - Indicates that there is no border inspection legal element recognition module, and LFRModule - 1 Represents the test conducted on the BorderLegal-QA dataset, LFRModule- 2Representative tests conducted on the JEC-QA dataset. As shown in Table 4 below, DRAG-BILQA has higher F1 and METEOR scores than other models on both datasets, and the higher the F1, the higher the METEOR score. After in-depth observation of the table data, the METEOR score is generally low. The reason for this phenomenon is that legal content is usually long, and the METEOR indicator is mainly used to measure the similarity between the automatically generated text and the reference answer. When processing long texts, it is difficult to effectively capture and highlight key information, which in turn affects the final score.
[0134] Table 4
[0135]
[0136] This example provides an experiment on the impact of the retrieval module on the generation quality. DRAG-BILQA consists of three modules. Retrieval Module- means there is no retrieval module. - 1 Indicates testing on BorderLegal-QA, Retrieval Module - 2 Indicates that the test was conducted on JEC-QA. As shown in Table 5 below, the METEOR score is generally low. By comparison, we found that our model has the highest METEOR score and the better the retrieval performance, the higher the METEOR score of the model. By comparison, we found that as the number of retrieval steps increases, the retrieval effect of the model is also constantly improving, which is very important for complex legal query tasks because it means that the model can gradually find more accurate answers.
[0137] Table 5
[0138]
[0139] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A dynamic retrieval enhancement generation method for border inspection legal questions and answers, characterized in that: The following steps are involved: S1: Generate answers based on questions; Including: directly generating answers and using border inspection legal data to generate answers. The answer generation formulas are as follows: ; in, Indicates the generated answer, q is the question, t is the number of times it is generated, t is initially 1, and each time it is generated, t+1. represents the generating function, represents the retrieval function, D is the external legal knowledge, , contains n question-answer pairs; The ChatGLM2-6B model is used to train the ChatGLM2-6B model using parameter effective fine-tuning QLoRA and Prompt tuning; the parameter effective fine-tuning QLoRA method formula is as follows: ; in, are the frozen pre-trained weights, is the incremental weight; Prompt tuning provides domain-specific knowledge as input messages to the ChatGLM2-6B model; S2: Extract the legal elements in the generated answer and calculate its confidence; If the confidence is less than the threshold, S3 is executed; if the confidence is greater than or equal to the threshold, the answer is output; Predict the answer A collection of legal elements , w is the total number of feature categories, and the formula is as follows: ; in, represents the border inspection legal element identification function; The AMR-based link prediction method is used to identify legal elements. The steps are as follows: S201: Use AMR Parser to convert the answer text into AMR Graph; S202: Using AMR-Text Aligner to ensure that the nodes in the AMR graph correspond to the words in the answer text; S203: Get answers using the pre-trained RoBERTa model The context representation vector of each word in ; S204: Calculating Answers in Span ProPosal The contextual representation of a specific word range Span in , the calculation formula is as follows: ; in, Represents a span in the text. Indicates the span starting vector, Indicates the span end vector, , , is a weight matrix used to combine the contextual representations of different words; S205: Use a feedforward neural network (FFNN) to calculate the logit score for each span. The calculation formula is as follows: ; S206: retain the top 50 spans with the highest scores and represent the context of the retained spans Embedded into the AMR graph to form the Span AMR Graph vector, Represents a point in the AMR graph, The formula is as follows: ; in Refers to the node type lookup table, including trigger, surround trigger, candidate span, and others. S207: Predict the legal elements that each candidate span plays The formula is as follows: ; S208: Legal elements Use FFNN to generate logit scores again, the formula is as follows: ; Get all legal elements set The logit score ; S209: Use Minimum to calculate all logit scores as their confidence scores. The formula is as follows: S3: Retrieve legal content from the preset knowledge base based on the question; S4: Use the retrieved legal content as the input information of S1, and execute S1 again to achieve dynamic retrieval enhanced generation.
2. The border inspection legal question and answer dynamic retrieval enhancement generation method according to claim 1 is characterized in that: S3 also includes: S301: using a text segmenter to segment an external knowledge document into document pieces of a preset format; S302: storing the document piece in a vector database Vector DB in an embedded form; S303: The dense vector retrieval model of the encoder architecture is used to retrieve from the Vector DB, and the relevance score is calculated using the dot product. The formula is as follows: ; in, is the problem vector, is the border inspection legal corpus vector.
3. The border inspection legal question and answer dynamic retrieval enhancement generation method according to claim 1 is characterized in that: Also includes: When training the retrieval model, we use cross-batch negative sampling to train all paragraphs as negative samples of the current question so that the model can learn dense representations of questions and paragraphs. The loss function formula during training is as follows: ; in, For the problem, is the positive sample of the problem, are negative samples of m questions.
4. A border inspection legal question and answer dynamic retrieval enhancement generation system, using the generation method according to any one of claims 1 to 3, characterized in that: include: A generation module configured to generate answers to border inspection legal questions using border inspection legal data; A border inspection legal element recognition module is configured to extract legal elements from the generated answer and calculate the confidence level thereof; and The retrieval module is configured to retrieve legal content in a preset knowledge base according to the question when the confidence level is lower than a threshold.
