Case question answering method based on large language model, medium and equipment

Abstract

The invention relates to the technical field of intelligent question answering methods, in particular to a case question answering method based on a large language model, a medium and equipment. The method specifically comprises the steps of obtaining graphic and text information and related questions of a case; and inputting the intelligent question and answer model to obtain an optimal answer. The intelligent question and answer model is obtained through the following steps: training a pre-training model through a constructed multi-modal corpus based on transfer learning to obtain a base model; based on LoRA and P-Tuning, performing fine tuning on the parameters of the base model to obtain a fine-tuned large language model; introducing a CLIP module to perform feature fusion on the graphic and text information to obtain a preliminary question and answer model; introducing a thinking chain to obtain a generation model capable of outputting a plurality of candidate answers with reasoning chains; and training an existing scoring model to obtain an evaluation model capable of outputting an optimal candidate answer. According to the method, the performance of the question answering technology in the aspect of generating accurate and reliable answers can be improved, and error information or illusion is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent question answering methods, and in particular to a case question answering method, medium and device based on a large language model. Background Art

[0002] With the rise of artificial intelligence (LegalAI), various industries and fields are undergoing major transformations. It benefits all kinds of groups by automating tasks and answering questions. It improves people's work efficiency by reducing the heavy burden of paper work and simplifies the process of obtaining daily life services, consulting services and remote case consultation for ordinary people.

[0003] However, the application of these technologies also faces challenges such as answer hallucinations and insufficient reasoning capabilities of the model. Therefore, solving the problem of hallucinations in intelligent judgment and improving the reasoning capabilities of the model have become current research hotspots.

[0004] Combining the research and collation of current question-answering technology and large model technology in designated application fields, it can be seen that there are few annotated corpora of existing cases, and due to the particularity of case question-answering tasks, some intelligent question-answering methods cannot be directly transferred to this field. Since large language models have achieved excellent results in various tasks, using large language models to solve case question-answering problems is currently a research hotspot, but the following problems still exist: 1) Although large language models have shown excellent results in many NLP tasks, the commonly used fine-tuning methods are still not ideal in case intelligent question answering tasks.

[0005] 2) Although the large language model has good results in intelligent question answering in general fields, due to the particularity of case question answering tasks, the logic and accuracy of intelligent question answering are still insufficient. Summary of the invention

[0006] The present invention aims to provide a case question-answering method, medium and device based on a large language model, which can improve the performance of question-answering technology in generating accurate and reliable answers and reduce the generation of erroneous information or hallucinations. The specific technical solution is as follows: The present invention provides a case question answering method based on a large language model, comprising the following steps: Obtain graphic and text information of the case and relevant issues of the case; The obtained case graphic information and case-related questions are input into the intelligent question-answering model to obtain the optimal answer. The intelligent question-answering model is obtained through the following steps: S1: Construct a multimodal corpus containing case-related knowledge, and train a pre-trained model through the multimodal corpus based on transfer learning to obtain a base model suitable for the case, wherein: the data format of the multimodal corpus includes text and pictures; S2: Fine-tune the parameter information of the base model based on LoRA and P-Tuning to obtain a fine-tuned large language model; S3: Based on the fine-tuned large language model, the CLIP module is introduced to perform multimodal feature fusion on the image and text information of the input case, and a preliminary question-answering model that can understand and fuse image and text information is obtained; S4: Introduce a multi-dimensional logically associated thinking chain into the preliminary question-answering model to obtain a generative model; the generative model outputs multiple candidate answers with reasoning chains based on the relevant questions of the input case; S5: Train the existing scoring model based on the contrastive learning method and the candidate answers generated by the generative model to obtain an evaluation model that can output the best answer among multiple candidate answers.

[0007] Optionally, the S1 includes: S1.1. Collect multi-source data in case-related fields, and pre-process the collected multi-source data in case-related fields to obtain a multimodal corpus containing case-related knowledge, wherein: the text data of case-related knowledge includes case-related regulations and case judgment information; the pre-processing includes data cleaning of the multi-source data in case-related fields, and labeling of unlabeled data in the multi-source data in case-related fields; S1.2. Select a language model pre-trained on a large-scale general corpus as a pre-trained model, and initialize the pre-trained model; S1.3. Extract annotated data sets in case-related fields from the multimodal corpus, and continue to train the pre-trained model with the annotated data sets in case-related fields to obtain a fine-tuned model. The specific formula is as follows: ; in, is the loss function on the target task; For pre-trained models; It is a labeled dataset in case-related fields. is the fine-tuned model; S1.4, adapting the fine-tuned model to a specific case question-answering task to obtain an adapted model, wherein the case question-answering task includes question understanding, answer generation, and information extraction; S1.5. Optimize the adapted model to obtain a base model suitable for the case.

[0008] Optionally, the S1.5 includes: Perform multi-task learning on the adapted model to obtain an optimized model that can perform multiple tasks simultaneously; Introduce regularization terms and learning rate scheduling strategies into the optimization model to obtain a regularized model; The knowledge graph or multimodal corpus is introduced into the regularized model to obtain a base model suitable for the case.

[0009] Optionally, the S2 includes: Based on the LoRA fine-tuning method, a low-rank matrix is ​​introduced to optimize the weight matrix of a specific layer in the base model to obtain the base model after the first fine-tuning. Based on the P-Tuning fine-tuning method, the prompt for the case question answering task at the input of the base model after the first fine-tuning is optimized to obtain the fine-tuned large language model.

[0010] Optionally, the S3 includes: The CLIP model is used to encode the image and text information of the input case respectively, generating the image embedding vector and the text embedding vector; Images and texts are embedded in a shared semantic space for comparative learning and fusion, resulting in a preliminary question-answering model that can understand and integrate image and text information.

[0011] Optionally, the S4 includes: S4.1. Construct a multi-dimensional logically related thinking chain based on syllogism and analogical reasoning; specifically: Input the relevant regulations as the main premise and the case's graphic and text information as the secondary premise into the preliminary question-answering model; Determine whether the input case graphic information complies with the relevant provisions under the main premise. If so, the preliminary question-answering model outputs a preliminary conclusion. If not, search for historical cases and judgments similar to the current case and deduce similar conclusions by analogy. The main premise, the secondary premise, and the inference conclusion based on the main premise and the secondary premise are the three elements of the syllogism. Combine the reasoning conclusions of syllogism and analogy to generate a complete chain of reasoning and answers to case consultations; S4.2. Introduce the multi-dimensional logically related thinking chain into the preliminary question-answering model to obtain the generative model.

[0012] Optionally, the S5 includes: S5.1. Construct contrastive learning loss function and ranking loss function; S5.2. Based on the contrastive learning loss function and the ranking loss function, the existing scoring model is trained to learn to distinguish high-quality answers from low-quality answers among multiple candidate answers and to rank them, thereby obtaining an evaluation model that can output the best answer among multiple candidate answers, where: high-quality answers are positive samples, and low-quality or irrelevant answers are negative samples; Specifically: S5.2.1. Input relevant questions and candidate answers of the case: Input relevant questions and multiple candidate answers into the existing scoring model; S5.2.2. Compute sentence embeddings: Generate sentence embeddings for questions and candidate answers using a pre-trained language model. S5.2.3. Construct a total loss function: Construct a total loss function consisting of a contrastive learning loss function and a ranking loss function; S5.2.4. Calculate loss: Evaluate the quality of each candidate answer by calculating the total loss function and sort the candidate answers by their relevance to the question; S5.2.5, Gradient update: Update the weights of the existing scoring model by optimizing the total loss function using the gradient descent algorithm through the back-propagation algorithm; S5.2.6. Iterative training: Repeat S5.2.4 to S5.2.5 until the total loss function of the existing scoring model converges, and an evaluation model that can output the best answer among multiple candidate answers is obtained.

[0013] Optionally, the contrastive learning loss function is a classification objective function or a triplet objective function, the classification objective function The specific formula is as follows: ; in: The time step for training the existing scoring model; is the trainable weight matrix, ; is the dimension of sentence embedding, is the number of label categories in the classification objective function; , For two different sentence embeddings, for , The distance between Triplet objective function The specific formula is as follows: ; in: For a given anchor sentence Sentence embeddings, For positive sentences Sentence embeddings, Negative sentence sentence embeddings; It is a boundary; Ranking loss function The specific formula is as follows: ; in: is the candidate set consisting of candidate answers, and They are Candidate answers and Candidate answers, , are the labels of the candidate answers, ; Indicates candidate answers With the problem similarity; Candidate answer With the problem The similarity of , are the problem numbers, .

[0014] The present invention also provides a readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implements the case question and answer method based on a large language model as described above.

[0015] The present invention also provides an electronic device, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory. When the computer program instructions are executed by the processor, the case question and answer method based on the large language model is performed.

[0016] The base model constructed based on transfer learning in the present invention allows the model to learn from a wide range of data sources and apply it to the professional field of the case, which can effectively enhance the base model's ability to understand case-specific terms and contexts; Through the fine-tuning method combining LoRA and P-Tuning, the knowledge of the large language model is transferred to the target task to the maximum extent, reducing the dependence on a large amount of domain-specific annotated data, and realizing the rapid adaptation and deployment of the model; and the parameter information of the base model is used as the initialization parameter of the new model to be trained on the target domain, and then some parameters of the model are updated through the annotated corpus of the target domain to obtain the fine-tuned large language model, which can speed up and optimize the learning efficiency of the fine-tuned large language model, without having to learn the model from scratch, which greatly reduces the training cost; Through training, the CLIP model can map images and texts in the multimodal corpus into a shared semantic space for multimodal feature fusion, and obtain a preliminary question-answering model, so that the model can understand and combine image and text information to provide more accurate and comprehensive answers when conducting intelligent question-answering. Furthermore, the present invention introduces a multi-dimensional logically associated thinking chain on the basis of the preliminary question-answering model to obtain a generative model, so that the generative model not only relies on the matching of surface facts, but can also make more reasonable judgments at multiple levels through reasoning ability; Finally, by training the existing scoring model, we obtain an evaluation model that can output the best candidate answer to reduce hallucinations during model question and answering.

[0017] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 Schematic diagram of the process from S2 to S3 in an embodiment of the present invention; Figure 2 It is a specific technical roadmap of S3 in the embodiment of the present invention; Figure 3 It is a specific technical roadmap of S4 in the embodiment of the present invention. DETAILED DESCRIPTION

[0019] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0020] In one embodiment, see Figure 1 to Figure 2 , a case question answering method based on a large language model includes the following steps: 1. Obtain graphic and text information of the case and relevant issues of the case; 2. Input the obtained case text and picture information and relevant questions of the case into the intelligent question-answering model to obtain the best answer; The intelligent question answering model is obtained through the following steps: S1: Construct a multimodal corpus containing case-related knowledge, and train a pre-trained model through the multimodal corpus based on transfer learning to obtain a base model suitable for the case, wherein: the data format of the multimodal corpus includes text and pictures; In some specific emerging fields, if you want to build a model to handle question-answering tasks, it is usually difficult to obtain enough labeled data to support model training. Therefore, it is imperative to build a reliable model that only needs a small amount of labeled training data. Transfer learning is an ideal solution to this problem. It uses existing knowledge to solve problems in the target domain that is different from the source domain but related to the source domain. For case question-answering tasks in case-related fields, the large language model that has been trained on a large-scale corpus is continued to be pre-trained, and some language features of the source domain are transferred to the target domain (such as case-related fields) to improve the performance of the model. The resulting pre-trained model is then adapted to the case question-answering model in the target domain, maximizing the reuse of existing knowledge without investing a lot of cost to re-collect the labeled corpus of the target domain, thereby achieving rapid migration and application of the model.

[0021] S1 includes: S1.1. Collect multi-source data in case-related fields, and pre-process the collected multi-source data in case-related fields to obtain a multimodal corpus containing case-related knowledge.

[0022] Among them: the text data of case-related knowledge includes case-related regulations and case judgment information; If labeled data is scarce, unlabeled data can be used first, which can help the model learn the language characteristics of case-related fields. These unlabeled data can be used to optimize the pre-training stage through self-supervised learning.

[0023] Preprocessing includes data cleaning of multi-source data in case-related fields and labeling of unlabeled data in multi-source data in case-related fields. Data cleaning is a key step to ensure the quality of model training, removing noise and irrelevant information to ensure data quality; the labeling process can be achieved through expert review or semi-automatic tools, especially accurate labeling of specific terms and case scenarios in case-related fields. In order to alleviate the problem of scarce labeled data, data enhancement techniques can be used, such as expanding the diversity of data sets through text generation models.

[0024] S1.2. Select a language model pre-trained on a large-scale general corpus as a pre-trained model, and initialize the pre-trained model; S1.3. Extract annotated data sets in case-related fields from the multimodal corpus, and continue to train the pre-trained model with the annotated data sets in case-related fields to obtain a fine-tuned model. The specific formula is as follows: ; in, is the loss function on the target task; For pre-trained models; It is a labeled dataset in case-related fields. is the fine-tuned model; Transfer learning extracts language knowledge from a large-scale multimodal corpus, and then further trains the model with annotated data specific to the case-related field, so that the model has the ability to understand case-specific terminology and context. Specifically, by fine-tuning the data on the case-related field, the model can master the terminology, relevant regulations, crime patterns, and case reasoning process in the field. The fine-tuning stage is the core step of transfer learning. In this stage, the pre-trained language model will be further trained on specific corpora in the target field (such as cases) so that the model can adjust the language features of the source field and learn knowledge such as professional terminology, case types, and behavior patterns specific to the case-related field. The main task of the fine-tuning stage is to adjust the parameters of the model so that it can adapt to the specific needs of the target domain. Since the knowledge in the case-related field is highly specialized, the model needs to optimize the parameters through gradient descent on limited annotated data to maximize its performance in this field.

[0025] S1.4, adapting the fine-tuned model to a specific case question-answering task to obtain an adapted model, wherein the case question-answering task includes question understanding, answer generation, and information extraction; The target task usually includes classification, sequence labeling and other task forms. The choice of loss function depends on the task type. In this embodiment, the target of case analysis is classification task, and the loss function Using cross entropy loss To define: ; in, is a sample in the labeled dataset of the case-related field Tags, is a sample output of the model The class probability of is the number of samples in the labeled dataset in the case-related field. This loss function is optimized during the training process so that the class probability output by the model is as close to the true label as possible.

[0026] A label refers to a category associated with each sentence or sentence pair. In a question-answering system, it may be a question category or a classification of an answer. In this embodiment, when doing a "question-answer matching" task, the label may be "match" or "mismatch". In other types of tasks, labels may represent different categories, such as case category, question type, etc.

[0027] The labels are used to classify or distinguish different types of questions or answers in the model through contrastive learning tasks. In another example of this embodiment, the task of the model is to classify the description of the crime case according to the similarity between the question pair, then the labels may correspond to different types of cases or case categories.

[0028] In summary, labels in this context are related to the semantic matching of questions and answers, which determines whether a sentence pair belongs to the same category (e.g., similar case descriptions and question types).

[0029] In classification tasks, the true label refers to the category or label that the sample actually belongs to in the training data. In the context of case analysis, the true label refers to the category that each training sample (i.e., the case description or related data) actually belongs to. In this embodiment, cases may be classified according to different types. For each training sample, the true label indicates the actual type of the case.

[0030] Questions and candidate answers involve the analysis of the case. The relationship between the true label and the question and candidate answer may be reflected in the following aspects: Question: The question can be some form of case analysis task, such as "What type of case is this?" or "What is the nature of this case?".

[0031] Candidate answers: Candidate answers are some category labels.

[0032] True label: For each training sample (i.e., each case), the true label is the actual type of the case (i.e., a category in the corresponding candidate answer). During training, the model will output the probability of each candidate answer (such as the probability of each type) based on the case description and the question. The goal of the cross entropy loss function is to make the probability output by the model as close to the true label as possible, that is, to make the model correctly classify the case.

[0033] For sequence labeling tasks (such as case summary generation or entity recognition), a sequence labeling loss function can be used. In this embodiment, a conditional random field (CRF) loss function is used. , the specific formula is as follows: ; in: is the time step The eigenvectors on ; is the time step Output labels, Yes Tags Parameters; is the total time step; are all possible labels, For inversion.

[0034] After the model is fine-tuned, verification and evaluation become key links. Traditional evaluation indicators, such as accuracy and recall, have certain reference value, but cannot fully reflect the actual performance of the model when handling cases. Therefore, the evaluation process needs to combine the requirements of domain-specific tasks and design more diverse evaluation criteria, covering the model's reasoning ability, interpretability, generation ability and other aspects. In addition, the evaluation of domain experts is particularly important in this process. Through the feedback loop with experts, the model can be continuously optimized to ensure that it can provide solutions that meet actual needs.

[0035] S1.5, optimizing the adapted model to obtain a base model suitable for the case; In this embodiment, before performing multi-task learning on the adapted model, the method further includes: model optimization; The model optimization process is performed by minimizing the loss function, usually using gradient descent and its variants (such as Adam optimizer) to optimize the model parameters. The model parameters in this embodiment are set to , the optimization process can be expressed by the following formula: ; in: Iteration time step for optimizing the adapted model; The time step is The model parameters when The time step is Model parameters when ; is the learning rate; is the loss function About the Model parameter The gradient of the CRF loss function is: It is the specific annotation position when processing the sequence.

[0036] S1.5 includes: Perform multi-task learning on the adapted model to obtain an optimized model that can perform multiple tasks simultaneously; Introduce regularization terms and learning rate scheduling strategies into the optimization model to obtain a regularized model; The specific formula of the regularization term is as follows: ; Loss function after adding regularization term for: ; in, and for different regularization hyperparameters; are the parameters of the model, is the number of the model parameter. This can constrain the model to avoid overfitting during training.

[0037] The knowledge graph or multimodal corpus is introduced into the regularized model to obtain a base model suitable for the case.

[0038] In the model optimization stage, the model can be enhanced through knowledge graphs or multimodal corpora. For example, by introducing case-related multimodal corpora or relevant regulations , the model can find relevant information during reasoning, thereby improving the model's reasoning accuracy. Assuming that the probability of the multimodal corpus and the model output There is a certain relationship between them, and the loss function of the model can be adjusted as , the specific formula is as follows: ; in, For multimodal corpora with labels The relevant data number; Represents the multimodal corpus with labels By introducing the information of the multimodal corpus, the model can more accurately identify the specific features of the case during the reasoning process.

[0039] The focus of the model optimization phase is to introduce domain knowledge into the model to further enhance its reasoning and decision-making capabilities. The background knowledge of the model can be enriched by introducing a multimodal corpus, thereby improving its understanding and reasoning capabilities for complex cases. At the same time, as more annotated data is collected, the model should be periodically updated and iteratively trained to adapt to new threats and challenges in case-related fields.

[0040] Finally, during the model deployment and application phase, customized interfaces and APIs should be designed according to specific application scenarios (such as case trial support, network security monitoring, criminal case analysis, etc.) to embed the model into the actual workflow. At the same time, considering the ever-changing nature of cases, the model monitoring and updating mechanism should also be carried out continuously to ensure its long-term effectiveness and adaptability. In this way, it can be ensured that the constructed base model can continue to provide effective support in dynamic and complex case-related fields.

[0041] S2: Fine-tune the parameter information of the base model based on LoRA and P-Tuning to obtain a fine-tuned large language model; In case-based intelligent question and answering, the application of LoRA fine-tuning can significantly improve the performance of the model in dealing with complex problems. For those questions that require multi-step logical reasoning or deep understanding of context, LoRA fine-tuning can help the model more accurately parse the key elements of the problem and effectively extract relevant information from a large amount of data. This is crucial to improving the accuracy and adaptability of intelligent question and answering. In addition, the role of P-Tuning fine-tuning in intelligent question and answering is reflected in generating more accurate and relevant answers to questions. Through continuous updating and optimization of prompts, the model can better understand the specific requirements of the question and accurately quote and integrate relevant information in the answer. This method is particularly suitable for question and answer tasks that require understanding of complex contexts and deep information processing. Through fine-tuning, the knowledge of the large language model is transferred to the target task to the maximum extent, reducing the dependence on a large amount of domain-specific annotated data, and realizing rapid adaptation and deployment of the model. The parameter information of the base model is used as the initialization parameter of the new model to be trained on the target domain, and some parameters of the model are subsequently updated through the annotated corpus of the target domain. This method can speed up and optimize the learning efficiency of the model without having to learn the model from scratch, which greatly reduces the training cost.

[0042] In intelligent question-answering tasks in case-related fields, the combination of LoRA (Low-Rank Adaptation) and P-Tuning can make up for the shortcomings of a single fine-tuning method and jointly improve the processing capabilities of the model. They optimize the model from different angles, and when combined, they can significantly improve the performance of the model in complex reasoning tasks, especially in case reasoning and the processing of technical details.

[0043] S2 includes: Based on the LoRA fine-tuning method, a low-rank matrix is ​​introduced to optimize the weight matrix of a specific layer in the base model to obtain the base model after the first fine-tuning. In this embodiment, the weight matrix of a certain layer of the model is , after LoRA fine-tuning, the weight matrix of this layer It is decomposed into the product of low-rank matrices. The mathematical formula for LoRA fine-tuning is as follows: ; in: is the original weight matrix. is the increment of the low-rank matrix, and are low-rank matrices to reduce computation and storage overhead.

[0044] Through LoRA fine-tuning, the model can effectively adjust the weights of these layers to enhance the model's performance in case tasks, such as enhancing the ability to understand technical details and the accuracy of case reasoning.

[0045] Based on the P-Tuning fine-tuning method, the prompt for the case question answering task at the input of the base model after the first fine-tuning is optimized to obtain the fine-tuned large language model.

[0046] In this embodiment, the core of P-Tuning fine-tuning is to optimize the task prompt so that the model can better understand the context of a specific task. Assume that the input prompt is , P-Tuning through the optimization prompt To adjust the output of the model. Assume that the original prompt input is After P-Tuning, the optimized prompt is obtained. : ; in: is the original prompt. It is the prompt adjustment obtained by gradient descent or other optimization methods during P-Tuning.

[0047] By optimizing prompts, P-Tuning can guide the model to be more accurate in handling specific tasks (such as case reasoning and case fact matching), thereby improving the performance of the question-answering system.

[0048] In summary, the reasoning process after combining LoRA and P-Tuning fine-tuning can be expressed as: ; in: Represents the model weight matrix based on LoRA fine-tuning optimization Regarding relevant regulations and the facts of the case The process of reasoning. Indicates the optimized task prompt The process of reasoning.

[0049] During fine-tuning, a loss function is usually defined To measure the gap between the model prediction and the true label, the goal is to minimize the loss function. With the combination of LoRA and P-Tuning fine-tuning, the loss function can be expressed as: ; in: It is a loss function fine-tuned and optimized by LoRA, which measures the reasoning error based on relevant regulations and case facts. It is the loss function after fine-tuning and optimization through P-Tuning, which measures the question-answering error based on prompt optimization.

[0050] Finally, the minimization of the loss function is used to update the low-rank matrix increment in LoRA fine-tuning through back-propagation and , and the prompts in P-Tuning fine-tuning .

[0051] After the model is fine-tuned with LoRA and P-Tuning, the final inference formula can be expressed as: ; Here, the softmax function is used to normalize the model output and generate the final answer. This output represents the model's reasoning result on the matching of relevant provisions in the case with the case facts.

[0052] The key advantages of combining LoRA with P-Tuning are as follows: (1) Complementarity: LoRA optimizes the internal part of the model, and P-Tuning optimizes the input and output.

[0053] (2) LoRA improves the model’s reasoning ability when dealing with specific tasks (such as technical issues of cases and case reasoning) by optimizing the model’s internal structure and weights, especially in tasks with deep reasoning levels and high background knowledge requirements. It helps the model better understand the relationship between case facts and relevant regulations.

[0054] (3) P-Tuning focuses on optimizing the prompts at the input end, so that the model can better understand and parse the background information of complex questions and generate accurate answers. By optimizing the prompts, P-Tuning can guide the model to focus on specific case details and case information, thereby improving the accuracy of the question-answering system.

[0055] (4) Enhanced reasoning capabilities: LoRA optimizes multi-level reasoning, and P-Tuning optimizes task context.

[0056] (5) In a case, the reasoning task may involve multiple levels of information processing, including extracting information from case facts, matching with relevant regulations, and performing logical reasoning. LoRA fine-tuning optimizes the model's reasoning ability at these levels, allowing the model to efficiently capture key information of the case and perform effective reasoning.

[0057] (6) At the same time, P-Tuning optimizes prompts to enable the model to accurately grasp the core of the problem at each level. For example, when dealing with the question of "whether it constitutes a cyber attack", P-Tuning will help the model extract the most relevant technical details from the facts of the case and guide the model to use the correct relevant regulations for reasoning.

[0058] (7) Reduce the number of parameters: LoRA maintains lightweight, and P-Tuning improves efficiency.

[0059] (8) The low-rank matrix structure of LoRA will not significantly increase the number of model parameters, making the fine-tuning process more efficient while ensuring the deployability of the model in resource-constrained environments.

[0060] (9) P-Tuning reduces the computational overhead by optimizing task prompts rather than model parameters, making the fine-tuning process more efficient and scalable.

[0061] S3: Based on the fine-tuned large language model, the CLIP module is introduced to perform multimodal feature fusion on the image and text information of the input case, and a preliminary question-answering model that can understand and fuse image and text information is obtained; S3 includes: The CLIP model is used to encode the image and text information of the input case respectively, generating the image embedding vector and the text embedding vector; Images and texts are embedded in a shared semantic space for comparative learning and fusion, resulting in a preliminary question-answering model that can understand and integrate image and text information.

[0062] Specifically: Case analysis usually involves a large amount of multimodal data, such as text, images, log files, etc. Traditional text analysis methods cannot effectively process image information, while the multimodal feature fusion method based on the CLIP (Contrastive Language-Image Pre-training) model can combine image and text information to achieve more accurate case analysis and intelligent question answering. This solution aims to develop an intelligent question answering system that can understand and integrate image and text information by combining the CLIP model and a large language model to improve the automated processing capabilities of cases.

[0063] This study will use a method that combines the CLIP model and a large language model to perform feature fusion and multimodal data processing. The CLIP model encodes the image and maps the image content to a semantic space, so that it can be compared, learned and fused with text features. The large language model is responsible for processing and generating text-based answers or reasoning.

[0064] The CLIP model was proposed by OpenAI. It uses contrastive learning to embed images and text into a shared semantic space, and can process and understand the relationship between images and text. The CLIP model consists of two parts: Image encoder: Encode the image through a convolutional neural network (CNN) or Vision Transformer (ViT) to generate an embedding vector of the image ; Text encoder: Encode the text through the pre-trained Transformer model to generate the text embedding vector ; The core of the CLIP model is to optimize the similarity between images and texts through contrastive learning, that is, by maximizing the similarity between image and text vectors with the same semantics and minimizing the similarity between image and text vectors with different semantics.

[0065] The core idea of ​​the CLIP model is to optimize the common representation of images and texts through contrastive learning, so that images and texts with the same semantics are as close as possible in the shared semantic space, while images and texts with different semantics should be as far away as possible. This is achieved by designing a specific loss function.

[0066] The goal of contrastive learning is to maximize the similarity between positive samples (i.e., semantically consistent between image and text pairs) and minimize the similarity between negative samples (i.e., semantically inconsistent between image and text pairs). To achieve this, the CLIP model optimizes a contrastive loss function to measure the similarity between image and text vectors.

[0067] The loss function used by CLIP is based on contrastive loss, often referred to as information maximization loss. This loss function calculates the similarity between image and text vectors in a shared semantic space and encourages the model to keep image and text vectors with similar semantics close to each other, and image and text vectors with different semantics as far away as possible.

[0068] For each pair of image and text, the loss function of the CLIP model is It can be expressed as follows: ; in: is the number of negative samples; is the embedding vector of the image, representing the image semantic features. is the embedding vector of the text, representing the text semantic features. and are the embedding vectors of other images and texts respectively. It is usually a hyperparameter less than 1, which controls the compactness of the embedding space and affects the scale of similarity calculation. Calculate the similarity between the image and text vectors. The denominator contains the sum of the similarities of all images and all texts with the current image or text, ensuring that the impact of negative samples is effectively minimized.

[0069] In case analysis, CLIP can effectively compare case-related images (such as screenshots, on-site photos, network traffic diagrams, etc.) with descriptive text (such as case overviews, human descriptions, event logs, etc.). Through training, the CLIP model can map images and text into a shared semantic space, so that the preliminary question-answering model can understand and combine image and text information to provide more accurate and comprehensive answers when conducting intelligent question-answering.

[0070] S4: Introduce a multi-dimensional logically associated thinking chain into the preliminary question-answering model to obtain a generative model; the generative model outputs multiple candidate answers with reasoning chains based on the relevant questions of the input case; See also Figure 3 , S4 includes: S4.1. Construct a multi-dimensional logically related thinking chain based on syllogism and analogical reasoning; specifically: ①. Input the relevant regulations as the main premise and the graphic information of the case as the secondary premise into the preliminary question-answering model; Traditional case reasoning relies on deductive reasoning, which combines relevant provisions (primary premise) with case facts (secondary premise) to draw conclusions about specific cases. However, in cases, relevant provisions are often relatively general, and case facts are complex and changeable, and the relationship between the two is not always linear or direct. Therefore, relying solely on traditional syllogistic reasoning may lead to insufficient reasoning or even erroneous reasoning results.

[0071] ②. Determine whether the input case graphic information complies with the relevant provisions under the main premise. If so, the preliminary question-answering model outputs a preliminary conclusion. If not, search for historical cases and judgments similar to the current case and deduce similar conclusions by analogy. The main premise, the secondary premise, and the inference conclusion based on the main premise and the secondary premise are the three elements of the syllogism. Specifically: Combine the reasoning conclusions of syllogism and analogy to generate a complete chain of reasoning and answers to case consultations; In this example, according to the reasoning structure of the syllogism, if the facts of the case meet the applicable relevant provisions, we can conclude that , that is, the judgment or relevant provisions of the case apply: if and If true, then we conclude , and Respectively indicate the relevant provisions and case facts used in the current step; Right now: ; in, is the conclusion of the case. and If all are true, then the inference result is correct. is true.

[0072] Analogical reasoning helps the model reason by identifying precedent cases that are similar to the current case. Suppose we have a set of precedent cases , where each case has its facts described The core idea of ​​analogical reasoning is that if a case With the current case If the two cases are similar in some key features, we can infer the verdict of the current case. To measure the case Precedent Cases Similarity: ; in It is the current case The factual characteristics of It is a precedent case The factual characteristics of It is a similarity measurement function, which can usually be calculated by cosine similarity, Euclidean distance, etc.

[0073] Once the similarity measure of the cases is calculated, the process of analogical reasoning can be expressed as: ; in, is the predicted conclusion of the current case, It is a precedent case Through weighted similarity measurement, the model will predict the most likely decision result.

[0074] According to syllogism and analogical reasoning, two results are obtained: and , then the final inference result It can be expressed as: ; in, is a weight coefficient that controls the influence of the two reasoning methods. Through this weighting mechanism, the model can flexibly combine deductive reasoning and analogical reasoning, thereby improving the accuracy and adaptability of reasoning. For different cases, the weight It can be adjusted according to the characteristics of the case to better adapt to the relationship between different types of case facts and relevant provisions.

[0075] In some complex cases, reasoning may require multiple iterations or recursion. In this case, we can regard the reasoning process as a recursive model. Assume that the reasoning chain contains multiple steps, and the reasoning result of each step depends on the conclusion of the previous step. We can express it as a recursive formula: ; in, It is The result of step reasoning, It is the result of the previous round of reasoning.

[0076] Through recursive reasoning, the model can gradually optimize and improve the reasoning process, and gradually enhance its reasoning ability for complex cases.

[0077] S4.2. Introduce the multi-dimensional logically related thinking chain into the preliminary question-answering model to obtain the generative model.

[0078] In this embodiment, the introduction of thought chain has the following advantages: (1) Enhanced reasoning ability ZK-CoT not only relies on the strict rules of syllogism, but also introduces the flexibility of analogical reasoning. By enhancing the model's ability to handle potential mismatches between relevant regulations and case facts, the model can provide more reasonable and accurate reasoning results when faced with complex or ambiguous cases.

[0079] (2) Multi-level reasoning structure ZK-CoT combines deductive reasoning and analogical reasoning through a multi-level reasoning structure, ensuring that the model has flexible adaptability when dealing with different types of case problems. The syllogism provides a rigorous logical framework, while analogical reasoning provides the model with greater flexibility and adaptability, enabling it to make more realistic reasoning in complex and ambiguous situations.

[0080] (3) Enhanced explanatory power Analogical reasoning can help the model understand and apply the logic behind relevant regulations. Analogical reasoning can help improve the model's explanatory power by drawing on the application of relevant regulations and judgment logic in similar cases.

[0081] (4) Supporting reasoning in complex cases In cases, the relationship between case facts and relevant regulations is often complex and indirect. ZK-CoT can effectively handle case reasoning tasks in complex cases, especially when the case facts involve new types of criminal behavior or technical issues, analogical reasoning can provide more appropriate answers.

[0082] S5: Train the existing scoring model based on the contrastive learning method and the candidate answers generated by the generative model to obtain an evaluation model that can output the best answer among multiple candidate answers.

[0083] S5 includes: S5.1. Construct contrastive learning loss function and ranking loss function; S5.2. Based on the contrastive learning loss function and the ranking loss function, the existing scoring model is trained to learn to distinguish high-quality answers from low-quality answers among multiple candidate answers and to rank them, thereby obtaining an evaluation model that can output the best answer among multiple candidate answers, where: high-quality answers are positive samples, and low-quality or irrelevant answers are negative samples; Specifically: S5.2.1. Input case-related questions and candidate answers: and multiple candidate answers Input into existing scoring model; S5.2.2. Compute sentence embeddings: Generate sentence embeddings for questions and candidate answers using a pre-trained language model. S5.2.3. Construct a total loss function: Construct a total loss function consisting of a contrastive learning loss function and a ranking loss function; The contrastive learning loss function is a classification objective function or a triplet objective function. The classification objective function The specific formula is as follows: ; in: The time step for training the existing scoring model; is the trainable weight matrix, , is the set of real numbers, is the dimension of sentence embedding, is the number of label categories in the classification objective function; , Embedding for two different sentences; The classification objective function can help the model distinguish different candidate answers, improve the classification accuracy of the model in different types of questions, and ensure that the generated answers have high semantic relevance.

[0084] Triplet objective function The specific formula is as follows: ; in: For a given anchor sentence Sentence embeddings, For positive sentences Sentence embeddings, Negative sentence sentence embeddings; It is a boundary; Given an anchor sentence , a positive sentence and a negative sentence , the triplet loss function adjusts the network so that and The distance between and The distance between.

[0085] By comparing positive and negative samples, the model can distinguish correct from incorrect answers in complex reasoning tasks, especially in case reasoning and case analysis.

[0086] Ranking loss function The specific formula is as follows: ; is a candidate set consisting of candidate answers, that is, from all possible labels in the current task The candidate set selected from and They are Candidate answers and Candidate answers, , are the labels of the candidate answers, ; Indicates candidate answers With the problem similarity; Candidate answer With the problem The similarity of , are the problem numbers, .

[0087] In this embodiment, It is obtained by calculating the cosine similarity. The specific formula is as follows: ; in: and The candidate answers are and Problems These sentence embedding vectors are generated by pre-trained models such as RoBERTa. Represents the dot product of vectors, that is, calculates the inner product between two vectors. and They are vectors and The L2 norm of , that is, their respective modulus lengths.

[0088] The core idea of ​​the ranking loss function is to minimize the target loss so that the model can correctly rank multiple candidate answers according to their relevance to the question. Specifically, the ranking learned by the model should be: , answers with stronger relevance should be ranked first; this improves the model's ability to calculate the similarity between questions and answers, and is suitable for question matching and similar answer selection.

[0089] The total loss function for evaluating the model is the weighted sum of contrastive learning loss and ranking loss: ; in: and is a hyperparameter used to adjust the contrastive learning loss and ranking loss The weight in the total loss function. By jointly optimizing these two loss functions, the evaluation model can not only distinguish between good and bad answers, but also effectively rank multiple candidate answers, thereby improving the model's ability to generate high-quality answers.

[0090] S5.2.4. Calculate loss: Evaluate the quality of each candidate answer by calculating the total loss function and sort the candidate answers by their relevance to the question; S5.2.5, Gradient update: Update the weights of the existing scoring model by optimizing the total loss function using the gradient descent algorithm through the back-propagation algorithm; S5.2.6, iterative training: repeat S5.2.4 to S5.2.5 until the total loss function of the existing scoring model converges, and obtains an evaluation model that can output the best answer among multiple candidate answers. In this embodiment, the training is considered to have converged when the loss reduction between two rounds of iterative training is less than 0.001.

[0091] Under this training mechanism, the evaluation model will mimic the real evaluation indicators to rank candidate answers. This means that even in the absence of standard answers, the model can effectively rank answers based on the context of the original document and the question. A key advantage of this approach is that it can improve the model's ability to generate high-quality answers when there is no direct reference answer, especially when dealing with complex questions that require comprehensive understanding and reasoning.

[0092] This embodiment also provides a readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implements the case question and answer method based on a large language model as described above.

[0093] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the electronic device.

[0094] This embodiment also includes an electronic device, including: at least one processor, at least one memory, and computer program instructions stored in the memory. When the computer program instructions are executed by the processor, the case question and answer method based on the large language model as described above is performed.

[0095] The electronic device may be a computing device such as a mobile phone, a desktop computer, a notebook, a PDA, a cloud server, etc. The electronic device may include, but is not limited to, a processor and a memory. For example, the electronic device may also include an input / output device, a network access device, a bus, etc.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A case question answering method based on a large language model, characterized in that: The steps include: Obtain graphic and text information of the case and relevant issues of the case; The obtained case graphic information and case-related questions are input into the intelligent question-answering model to obtain the optimal answer. The intelligent question-answering model is obtained through the following steps: S1: Construct a multimodal corpus containing case-related knowledge, and train a pre-trained model through the multimodal corpus based on transfer learning to obtain a base model suitable for the case, wherein: the data format of the multimodal corpus includes text and pictures; S2: Fine-tune the parameter information of the base model based on LoRA and P-Tuning to obtain a fine-tuned large language model; S3: Based on the fine-tuned large language model, the CLIP module is introduced to perform multimodal feature fusion on the image and text information of the input case, and a preliminary question-answering model that can understand and fuse image and text information is obtained; S4: Introduce a multi-dimensional logically associated thinking chain into the preliminary question-answering model to obtain a generative model; the generative model outputs multiple candidate answers with reasoning chains based on the relevant questions of the input case; S5: Train the existing scoring model based on the contrastive learning method and the candidate answers generated by the generative model to obtain an evaluation model that can output the best answer among multiple candidate answers.

2. The case question answering method based on a large language model according to claim 1, characterized in that: The S1 includes: S1.

1. Collect multi-source data in case-related fields, and pre-process the collected multi-source data in case-related fields to obtain a multimodal corpus containing case-related knowledge, wherein: the text data of case-related knowledge includes case-related regulations and case judgment information; the pre-processing includes data cleaning of the multi-source data in case-related fields, and labeling of unlabeled data in the multi-source data in case-related fields; S1.

2. Select a language model pre-trained on a large-scale general corpus as a pre-trained model, and initialize the pre-trained model; S1.

3. Extract annotated data sets in case-related fields from the multimodal corpus, and continue to train the pre-trained model with the annotated data sets in case-related fields to obtain a fine-tuned model. The specific formula is as follows: ； in, is the loss function on the target task; For pre-trained models; It is a labeled dataset in case-related fields. is the fine-tuned model; S1.4, adapting the fine-tuned model to a specific case question-answering task to obtain an adapted model, wherein the case question-answering task includes question understanding, answer generation, and information extraction; S1.

5. Optimize the adapted model to obtain a base model suitable for the case.

3. The case question answering method based on a large language model according to claim 2 is characterized in that: The S1.5 includes: Perform multi-task learning on the adapted model to obtain an optimized model that can perform multiple tasks simultaneously; Introduce regularization terms and learning rate scheduling strategies into the optimization model to obtain a regularized model; The knowledge graph or multimodal corpus is introduced into the regularized model to obtain a base model suitable for the case.

4. The case question answering method based on a large language model according to claim 3 is characterized in that: The S2 includes: Based on the LoRA fine-tuning method, a low-rank matrix is ​​introduced to optimize the weight matrix of a specific layer in the base model to obtain the base model after the first fine-tuning. Based on the P-Tuning fine-tuning method, the prompt for the case question answering task at the input of the base model after the first fine-tuning is optimized to obtain the fine-tuned large language model.

5. The case question answering method based on a large language model according to claim 4 is characterized in that: The S3 includes: The CLIP model is used to encode the image and text information of the input case respectively, generating the image embedding vector and the text embedding vector; Images and texts are embedded in a shared semantic space for comparative learning and fusion, resulting in a preliminary question-answering model that can understand and integrate image and text information.

6. The case question answering method based on a large language model according to claim 5 is characterized in that: The S4 includes: S4.

1. Construct a multi-dimensional logically related thinking chain based on syllogism and analogical reasoning; specifically: Input the relevant regulations as the main premise and the case's graphic and text information as the secondary premise into the preliminary question-answering model; Determine whether the input case graphic information complies with the relevant provisions under the main premise. If so, the preliminary question-answering model outputs a preliminary conclusion. If not, search for historical cases and judgments similar to the current case and deduce similar conclusions by analogy. The main premise, the secondary premise, and the inference conclusion based on the main premise and the secondary premise are the three elements of the syllogism. Combine the reasoning conclusions of syllogism and analogy to generate a complete chain of reasoning and answers to case consultations; S4.

2. Introduce the multi-dimensional logically related thinking chain into the preliminary question-answering model to obtain the generative model.

7. The case question answering method based on a large language model according to claim 6 is characterized in that: The S5 includes: S5.

1. Construct contrastive learning loss function and ranking loss function; S5.

2. Based on the contrastive learning loss function and the ranking loss function, the existing scoring model is trained to learn to distinguish high-quality answers from low-quality answers among multiple candidate answers and to rank them, thereby obtaining an evaluation model that can output the best answer among multiple candidate answers, where: high-quality answers are positive samples, and low-quality or irrelevant answers are negative samples; Specifically: S5.2.

1. Input relevant questions and candidate answers of the case: Input relevant questions and multiple candidate answers into the existing scoring model; S5.2.

2. Compute sentence embeddings: Generate sentence embeddings for questions and candidate answers using a pre-trained language model. S5.2.

3. Construct a total loss function: Construct a total loss function consisting of a contrastive learning loss function and a ranking loss function; S5.2.

4. Calculate loss: Evaluate the quality of each candidate answer by calculating the total loss function and sort the candidate answers by their relevance to the question; S5.2.5, Gradient update: Update the weights of the existing scoring model by optimizing the total loss function using the gradient descent algorithm through the back-propagation algorithm; S5.2.

6. Iterative training: Repeat S5.2.4 to S5.2.5 until the total loss function of the existing scoring model converges, and an evaluation model that can output the best answer among multiple candidate answers is obtained.

8. The case question answering method based on a large language model according to claim 7 is characterized in that: The contrastive learning loss function is a classification objective function or a triplet objective function. The classification objective function The specific formula is as follows: ； in: The time step for training the existing scoring model; is the trainable weight matrix, ; is the dimension of sentence embedding, is the number of label categories in the classification objective function; , For two different sentence embeddings, for , The distance between Triplet objective function The specific formula is as follows: ； in: For a given anchor sentence Sentence embeddings, For positive sentences Sentence embeddings, Negative sentence sentence embeddings; It is a boundary; Ranking loss function The specific formula is as follows: ； in: is the candidate set consisting of candidate answers, and They are Candidate answers and Candidate answers, , are the labels of the candidate answers, ; Indicates candidate answers With the problem similarity; Candidate answer With the problem The similarity of , are the problem numbers, .

9. A readable storage medium, characterized in that: Computer program instructions are stored thereon, and when the computer program instructions are executed by a processor, a case question answering method based on a large language model as described in any one of claims 1 to 8 is implemented.

10. An electronic device, characterized in that: include: At least one processor, at least one memory, and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, the case question answering method based on a large language model as described in any one of claims 1 to 8.

Images