Legal field instruction data construction method and system based on large language model

Abstract

The invention discloses a legal field instruction data construction method and system based on a large language model, and is applied to the field of artificial intelligence and natural language processing. The initial data set comprises public data, legal question and answer data, manual annotation data, legal three-segment theory data, multi-round dialogue data and cause data; performing preprocessing and data integration on the initial data set to generate a legal field instruction data set; inputting the law field instruction data set into a large language model for training to obtain a fine-tuned large language model; and according to a legal consultation question input by a user, outputting an answer by the fine-tuned large language model to complete a legal consultation service.

Description

A method and system for constructing legal domain instruction data based on a large language model Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a method and system for constructing legal domain instruction data based on a large language model. Background Technology

[0002] In recent years, Large Language Models (LLMs) have achieved remarkable success in the field of natural language processing. These models, trained on large-scale corpora, are able to understand complex language structures and demonstrate outstanding performance in various natural language processing tasks. First, LLMs have become conversational assistants in our daily lives: general-purpose large models can understand and respond to various topics in everyday conversations, providing users with services such as weather inquiries, news broadcasts, and lifestyle suggestions. Second, LLMs enable cross-domain knowledge retrieval: in multiple fields such as education, entertainment, and tourism, general-purpose large models can provide cross-domain information retrieval and knowledge answers, helping users obtain the information they need. Furthermore, LLMs serve as bridges for multilingual communication: general-purpose large models supporting multiple languages ​​can act as communication bridges between different languages, promoting international exchange and cooperation.

[0003] With the rise of general-purpose large-scale models, large-scale models in Chinese vertical fields have also entered a period of rapid development. Medical large-scale models can provide doctors with diagnostic assistance and treatment suggestions by analyzing medical records, medical images, and other information. In the financial sector, vertical large-scale models can help financial institutions conduct credit assessments, market forecasts, and other tasks, reducing business risks. In the education sector, vertical large-scale models can provide personalized learning materials and tutoring suggestions based on students' learning progress and interests.

[0004] Traditional legal intelligence systems aim to retrieve legal documents, regulations, precedents, and other legal information, but due to a lack of semantic understanding of specific issues, they fail to provide satisfactory answers. Large Language Models (LLMs) have achieved excellent results in various natural language processing tasks, inspiring us to train LLMs in the legal field to aid legal retrieval. However, in the Chinese legal field, due to the complexity of legal issues and the rigor of legal provisions, there is currently no large legal model with satisfactory practical applications.

[0005] To overcome these shortcomings, this application proposes a method and system for constructing legal domain instruction data based on a large language model. Summary of the Invention

[0006] The purpose of this application is to provide a method and system for constructing legal domain instruction data based on a large language model, aiming to solve the above-mentioned problems.

[0007] To achieve the above objectives, this application provides the following technical solution:

[0008] This application provides a method for constructing legal domain instruction data based on a large language model, including:

[0009] Obtain an initial dataset, which includes: public data, legal Q&A data, manually annotated data, legal syllogism data, multi-turn dialogue data, and case data;

[0010] The initial dataset is preprocessed and integrated to generate a legal domain instruction dataset;

[0011] The legal domain instruction dataset is input into the large language model for training, resulting in a fine-tuned large language model.

[0012] Based on the legal consultation questions input by the user, the finely tuned large language model outputs the answers, thus completing the legal consultation service.

[0013] Furthermore, the initial dataset includes:

[0014] The public data includes, but is not limited to: Chinese and English parallel corpora, rich task Chinese data, GPT4 translated into Chinese data, Chinese and English bilingual human-computer question and answer Chinese data, and self-awareness data;

[0015] The legal syllogism data is a data structure with legal provisions as the major premise, case facts as the minor premise, and judgment as the conclusion. A solution is developed based on the data structure.

[0016] The multi-turn dialogue data is generated using GPT-4 by simulating a guided legal consultation scenario through prompts.

[0017] The case data is obtained by using the legal provision interface to retrieve legal information corresponding to the legal consultation question, and the most relevant legal provisions are used as reference information when the large language model outputs the answer.

[0018] Furthermore, the step of inputting the legal domain instruction dataset into the large language model for training to obtain a fine-tuned large language model specifically includes the following steps:

[0019] Based on the pre-training phase of the large language model, unsupervised learning is performed on the legal domain instruction dataset to enable the large language model to learn syntax and semantics.

[0020] Based on the fine-tuning phase of the large language model, supervised learning is performed on preset task data;

[0021] Set the training parameters of the large language model, including but not limited to: batch size, learning rate, and weight initialization;

[0022] After training is completed, the large language model is validated and evaluated using a validation set to obtain a fine-tuned large language model.

[0023] Furthermore, the process of providing legal consultation services by outputting answers from a finely tuned large language model based on user-inputted legal questions includes the following steps:

[0024] The legal consultation question is judged by a pre-built classification model. If it is judged to be a legal consultation, the legal consultation question is input into the pre-built retrieval model, and the most relevant legal provisions are output.

[0025] The legal consultation questions and the obtained legal provisions are input into a finely tuned large language model for analysis, and the final answer is output.

[0026] Furthermore, the legal consultation questions input by the user are subjected to both in-depth and in-depth evolution;

[0027] The deep evolution includes: adding constraints, deepening, specifying, adding reasoning steps, and increasing complexity;

[0028] The legal consultation question is input into gpt-3.5-turbo for expansion using pre-selected deep evolution instructions;

[0029] The evolved legal consultation questions will be cleaned up, removing characters including but not limited to #givenprompt and double quotes.

[0030] Furthermore, the breadth evolution is based on the legal consultation questions after the depth evolution, and calls the interface to retrieve relevant laws and regulations to obtain the corresponding legal provisions;

[0031] Input the corresponding rules into gpt-3.5-turbo to get the problem after breadth-first evolution.

[0032] This application provides a legal domain instruction data construction system based on a large language model, including:

[0033] Acquisition Module: Acquires the initial dataset, which includes: public data, legal Q&A data, manually annotated data, legal syllogism data, multi-turn dialogue data, and case data;

[0034] Processing module: preprocesses and integrates the initial dataset to generate a legal domain instruction dataset;

[0035] Training module: Input the legal domain instruction dataset into the large language model for training to obtain a fine-tuned large language model;

[0036] Output module: Based on the legal consultation questions input by the user, the fine-tuned large language model outputs the answers to complete the legal consultation service.

[0037] This application provides an apparatus comprising a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a method for constructing legal domain instruction data based on a large language model; the processor is used to execute the program instructions stored in the memory to implement a method for constructing legal domain instruction data based on a large language model.

[0038] This application provides a storage medium storing processor-executable program instructions for executing a method for constructing legal domain instruction data based on a large language model.

[0039] This application provides a method and system for constructing legal domain instruction data based on a large language model, which has the following beneficial effects:

[0040] (1) By acquiring public data, legal Q&A data, manually annotated data, legal syllogism data, multi-turn dialogue data, and case data, a high-quality and diverse initial dataset is formed, enabling the large language model to learn richer language representations and domain knowledge during the training process, and to better understand and handle legal issues.

[0041] (2) This application further improves the performance of the model in the legal field by pre-training and fine-tuning the large language model, thereby enhancing the model's generalization ability and accuracy;

[0042] (3) The method and system proposed in this application can provide users with efficient and accurate legal consultation services. By obtaining the legal consultation questions input by users, the system can automatically determine the intent of the questions and generate answers through a legal retrieval model and a fine-tuned large language model, thereby improving the efficiency and accuracy of legal consultation. Attached Figure Description

[0043] Figure 1 is a flowchart illustrating a method for constructing legal domain instruction data based on a large language model according to Embodiment 1 of this application;

[0044] Figure 2 is a technical roadmap diagram of a method for constructing legal domain instruction data based on a large language model according to Embodiment 1 of this application;

[0045] Figure 3 is a schematic diagram of the deep evolution process in Embodiment 1 of this application;

[0046] Figure 4 is a schematic flowchart of the breadth evolution in Embodiment 1 of this application;

[0047] Figure 5 is a schematic diagram of the structure of a legal domain instruction data construction system based on a large language model according to Embodiment 2 of this application;

[0048] Figure 6 is a schematic diagram showing the experimental comparison between Embodiment 2 of this application and four other models;

[0049] Figure 7 is a schematic diagram of the device structure in Embodiment 3 of this application;

[0050] Figure 8 is a schematic diagram of the storage medium structure of Embodiment 4 of this application. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0053] Example 1

[0054] Please refer to Figure 1, which is a flowchart illustrating a method for constructing legal domain instruction data based on a large language model according to Embodiment 1 of this application; the steps include:

[0055] S1: Obtain the initial dataset, which includes: public data, legal Q&A data, manually annotated data, legal syllogism data, multi-turn dialogue data, and case data.

[0056] In this embodiment, to ensure that the large language model retains its original general capabilities while enhancing its Chinese dialogue capabilities during training, public data was acquired for training the large model. These datasets cover parallel Chinese and English corpora, such as lima and airoboros-v1.4.1; rich task-based Chinese datasets that ensure instruction diversity, such as baai_ol_data; GPT4 translation data into Chinese, such as alpaca_data; bilingual Chinese-English human-computer question-and-answer datasets, such as sharegpt_data; and self-cognition data, such as self_cognition_data. By comprehensively utilizing this series of datasets, the large language model achieves both general capabilities and strong performance in Chinese dialogue during training.

[0057] Legal Q&A Data: A large amount of legal Q&A data was obtained from legal websites and cleaned. During the cleaning process, short responses were excluded, and this data was used for reconstruction.

[0058] Legal Syllogism Data: Because large-scale language models often lack sufficient legal knowledge when dealing with long questions, they may misunderstand the problem, resulting in inadequate solutions for users. Therefore, this application uses legal syllogism data for processing. The typical structure of a legal syllogism is referenced, where the legal provision serves as the major premise, the facts of the case as the minor premise, and the judgment as the conclusion. Solutions are written following this structure to ensure high quality and logical clarity in the generated answers; the conclusion, the cited legal provisions, and the facts of the case mentioned in the question form the major premise, while the minor premise is the minor premise. This structure helps improve the quality and accuracy of the answers, making them more consistent with the standards of legal reasoning.

[0059] Multi-round dialogue data: In daily legal consultation practice, the communication process between clients and lawyers plays a crucial role. Clients often raise their concerns briefly, which typically lack in-depth legal analysis and key information. In such cases, lawyers need to do more than just answer surface-level questions; they must skillfully guide the conversation to elicit more detailed and specific case information from the client. This guided dialogue allows for a deeper exploration of the legal details that may be involved in the case. This guided communication helps lawyers better understand the complexity of the case, identify potential legal issues, and provide clients with more comprehensive and accurate legal advice.

[0060] This application simulates a real-life legal consultation scenario where a lawyer guides a client by designing the roles of lawyer and client in the prompt framework. It utilizes GPT-4 to generate high-quality multi-turn dialogue data within the context of legal consultation. The specific algorithm is as follows:

[0061] Algorithm 1: Guided Dialogue Construction

[0062] Input: dataQ i ,InitializeH=[],turn=0,max_turns=4

[0063] C i =Q i ,

[0064] Repeat:

[0065] R=get_lawyer_response(history),L i =R,

[0066] turn+=1

[0067] R=get_client_response(history),C i =R,

[0068] Until:turn = max_turns

[0069] Output: H

[0070] Q i For the customer question in the i-th data entry, H represents the historical dialogue record between the customer and the lawyer. This represents the record in the i-th data entry concerning a customer's question. L represents the record in the i-th data item concerning the lawyer's answers to questions. i This represents the lawyer's response in the i-th data entry (each time the GPT-4 interface L is called). i (Updated once), C i This represents the query from the customer role in the i-th data entry (each time the GPT-4 interface C is called). i (Updated once), max_turns represents the maximum number of dialogue turns set, and turn is used to record the current number of dialogue turns.

[0071] Case data: To enhance the model's knowledge in the legal field, a series of strategies were adopted to enrich data diversity. Questions containing both criminal and administrative case causes were acquired to ensure broad data coverage, encompassing multiple aspects of the legal field.

[0072] S2: Preprocess and integrate the initial dataset to generate a legal domain instruction dataset.

[0073] S3: Input the legal domain instruction dataset into the large language model for training to obtain a fine-tuned large language model.

[0074] In this embodiment, based on the pre-training phase of the large language model, unsupervised learning is performed on the legal domain instruction dataset, enabling the large language model to learn syntax and semantics. Based on the fine-tuning phase of the large language model, supervised learning is performed on preset task data.

[0075] The training parameters of the large language model are set, including but not limited to: batch size, learning rate, and weight initialization. After training is completed, the large language model is validated and evaluated using a validation set to obtain a fine-tuned large language model.

[0076] S4: Based on the legal consultation question input by the user, the finely tuned large language model outputs the answer to complete the legal consultation service.

[0077] In this embodiment, the legal consultation question is judged by a pre-built classification model. If it is judged to be a legal consultation, the legal consultation question is input into a pre-built retrieval model, which outputs the most relevant legal provisions. The legal consultation question and the obtained legal provisions are input into a fine-tuned large language model for analysis, and the final answer is output.

[0078] The legal consultation question input by the user undergoes both deep and broad evolution. Deep evolution includes: adding constraints, deepening, specifying, adding reasoning steps, and increasing complexity; using pre-selected deep evolution instructions, the legal consultation question is input into gpt-3.5-turbo for expansion; the evolved legal consultation question is then cleaned, removing characters including but not limited to #given prompt and double quotes.

[0079] In summary, Embodiment 1 of this application constructs a high-quality legal domain instruction dataset by acquiring and processing public data and legal domain data, and by introducing manually labeled data, legal syllogism data, and guided multi-turn dialogue data. The legal domain instruction dataset is then used to train a large language model, improving the model's performance and generalization ability, and providing users with efficient and accurate legal consultation services.

[0080] Example 2

[0081] Please refer to Figure 5, which is a schematic diagram of the structure of a legal domain instruction data construction system based on a large language model according to Embodiment 2 of this application; the specific content includes:

[0082] Acquisition Module: Acquires the initial dataset, which includes: public data, legal Q&A data, manually annotated data, legal syllogism data, multi-turn dialogue data, and case data;

[0083] Processing module: preprocesses and integrates the initial dataset to generate a legal domain instruction dataset;

[0084] Training module: Input the legal domain instruction dataset into the large language model for training to obtain a fine-tuned large language model;

[0085] Output module: Based on the legal consultation questions input by the user, the fine-tuned large language model outputs the answers to complete the legal consultation service.

[0086] In this embodiment, the system was experimentally validated. One hundred questions were selected, and five baselines, including a fine-tuned large language model, were comprehensively evaluated. An in-depth analysis was conducted based on the following clearly defined scoring criteria.

[0087] 3: The answer is completely relevant to the search, the cited regulations and laws are highly relevant, and the logic is applied correctly.

[0088] 2: The answers are usually related to the search results, the legal citations are correct, but the laws are incorrect, while the logic is correct.

[0089] 1: The answer is not related to the search content, but the legal citation is incorrect, the law is incorrect, or the logic is incorrect.

[0090] 0: The answer is irrelevant to the search results, or the legal citation is incorrect, or the law is incorrect, or the logic is incorrect.

[0091] Please refer to Figure 6, which is a schematic diagram comparing the experimental results of Embodiment 2 of this application with four other models.

[0092] The comparison results of the finely tuned large language model proposed in this application with four other models in terms of human evaluation scoring are shown. It can be seen that compared with ChatGPT, Baichuan, and ChatGLM2, this application performs better in answering questions about the accurate citation of regulations and laws, and demonstrates logical applicability in a considerable number of cases. In the 30 evaluation questions, the finely tuned large language model achieved a higher answer rate than Tongyifarui. Overall, the finely tuned large language model outperforms other models in answering legal consultation questions.

[0093] Example 3

[0094] Please refer to Figure 7, which is a schematic diagram of the device structure of Embodiment 3 of this application. The device 50 includes a processor 51 and a memory 52 coupled to the processor 51.

[0095] The memory 52 stores program instructions for implementing the above-described method for constructing legal domain instruction data based on a large language model.

[0096] The processor 51 is used to execute program instructions stored in the memory 52 to implement a legal domain instruction data construction based on a large language model.

[0097] The processor 51 can also be referred to as a CPU (Central Processing Unit).

[0098] Processor 51 may be an integrated circuit chip with signal processing capabilities. Processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0099] Example 4

[0100] Please refer to Figure 8, which is a schematic diagram of the storage medium structure of Embodiment 4 of this application. The storage medium of this embodiment stores a program file 61 capable of implementing all the above methods. This program file 61 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or devices such as computers, servers, mobile phones, and tablets.

[0101] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0102] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0103] Although embodiments of this application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the appended claims and their equivalents.

[0104] Of course, the present invention may have many other embodiments. Based on this embodiment, other embodiments obtained by those skilled in the art without any creative effort are all within the scope of protection of the present invention.

Claims

1. A method for constructing legal domain instruction data based on a large language model, characterized in that, include: An initial dataset is obtained, comprising: public data, legal question-and-answer data, manually annotated data, legal syllogism data, multi-turn dialogue data, and case data. The initial dataset is preprocessed and integrated to generate a legal domain instruction dataset. This dataset is then input into a large language model for training, resulting in a fine-tuned large language model. Based on the user's input of a legal consultation question, the fine-tuned large language model outputs an answer, completing the legal consultation service.

2. The method for constructing legal domain instruction data based on a large language model according to claim 1, characterized in that, The initial dataset includes: the public data, including but not limited to: parallel Chinese and English corpora, rich task Chinese data, data translated into Chinese by GPT4, Chinese and English bilingual human-computer question-and-answer data, and self-awareness data; the legal syllogism data is a data structure with legal provisions as the major premise, case facts as the minor premise, and judgment as the conclusion, and a solution is developed based on the data structure; the multi-turn dialogue data: the multi-turn dialogue data is generated using GPT-4 by simulating guided legal consultation scenarios through prompts; the case data: the legal provisions information corresponding to the legal consultation questions is retrieved using the legal provision interface call method to obtain the most relevant legal provisions, which serve as reference information when the large language model outputs its answer.

3. The method for constructing legal domain instruction data based on a large language model according to claim 1, characterized in that, The step of inputting the legal domain instruction dataset into the large language model for training to obtain the fine-tuned large language model specifically includes the following steps: based on the pre-training stage of the large language model, unsupervised learning is performed on the legal domain instruction dataset to enable the large language model to learn syntax and semantics; based on the fine-tuning stage of the large language model, supervised learning is performed on preset task data. The training parameters of the large language model are set, including but not limited to: batch size, learning rate, and weight initialization. After training is completed, the large language model is validated and evaluated using a validation set to obtain a fine-tuned large language model.

4. The method for constructing legal domain instruction data based on a large language model according to claim 1, characterized in that, The process of providing legal consultation services by outputting answers from a finely tuned large language model based on user-inputted legal consultation questions includes the following steps: First, the legal consultation question is judged using a pre-built classification model. If it is determined to be a type of legal consultation, the question is input into a pre-built retrieval model, which outputs the most relevant legal provisions. Second, the legal consultation question and the obtained legal provisions are input into the finely tuned large language model for analysis, and the final answer is output.

5. The method for constructing legal domain instruction data based on a large language model according to claim 1, characterized in that, The legal consultation question input by the user is subjected to both deep and broad evolution. The deep evolution includes adding constraints, deepening, specifying, adding reasoning steps, and increasing complexity. The legal consultation question is input into gpt-3.5-turbo for expansion using pre-selected deep evolution instructions. The evolved legal consultation question is then cleaned to remove characters including but not limited to #givenprompt and double quotes.

6. The method for constructing legal domain instruction data based on a large language model according to claim 5, characterized in that, The breadth-based evolution is based on the legal consultation question after deep evolution. It calls the interface for retrieving relevant laws and regulations to obtain the corresponding legal provisions. The corresponding legal provisions are then input into gpt-3.5-turbo to obtain the breadth-based evolved question.

7. A system for constructing legal domain instruction data based on a large language model according to any one of claims 1-6, characterized in that, The system includes: an acquisition module for acquiring an initial dataset, which includes public data, legal question-and-answer data, manually annotated data, legal syllogism data, multi-turn dialogue data, and case data; a processing module for preprocessing and integrating the initial dataset to generate a legal domain instruction dataset; a training module for inputting the legal domain instruction dataset into a large language model for training to obtain a fine-tuned large language model; and an output module for outputting answers based on user-inputted legal consultation questions, thus completing the legal consultation service.

8. A device, characterized in that, The device includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a method for constructing legal domain instruction data based on a large language model as described in any one of claims 1-6; the processor is used to execute the program instructions stored in the memory to implement a method for constructing legal domain instruction data based on a large language model.

9. A storage medium, characterized in that, The system stores processor-executable program instructions for executing a legal domain instruction data construction method based on a large language model as described in any one of claims 1-6.

Images