Legal knowledge question-answering system constructed based on large language model and method thereof

Abstract

The invention discloses a legal knowledge question-answering system constructed based on a large language model and a method thereof, and relates to the technical field of large language models and the field of legal application. The system comprises a law and regulation knowledge base construction module, an RAG retrieval module, an LLM enhancement module and a user interaction interface. Based on an RAG retrieval enhancement generation related technology, the invention aims to perform efficient knowledge management on massive laws and regulations and recall related legal knowledge through accurate knowledge retrieval so as to enhance the effect and accuracy of an LLM large-scale language model in providing legal consultation service; by combining the RAG technology and the LLM technology, efficient knowledge management and accurate retrieval of massive laws and regulations are achieved, and an innovative and efficient solution is provided for the field of legal consultation services; according to the technology, developers do not need to re-train the whole LLM for each specific task, and only need to provide additional input for the LLM by connecting a related knowledge base, so that the accuracy of answers is improved.

Description

Technical Field

[0001] The present invention belongs to the technical fields of large language models and legal applications, and particularly relates to a legal knowledge Q&A system and method based on large language models. Background Art

[0002] In the public security field, business operators are responsible for receiving alarm information, preliminarily judging the police situation, and giving corresponding guidance.

[0003] During the actual connection process, the alarm caller often asks legal questions such as "Which laws and regulations have my actions violated and what impacts will they cause?" In the face of these questions, the operator needs to quickly query the system to give an answer.

[0004] The main problems existing in the prior art in legal knowledge Q&A include:

[0005] (1) Existing legal knowledge base systems often cannot be updated in a timely manner, resulting in a lack of accurate and reliable basis for operators when answering questions related to new legal provisions.

[0006] (2) Low query efficiency: Due to the large number of provisions and cases in the legal knowledge base and the lack of effective indexing and classification mechanisms, operators often need to spend a lot of time and effort in querying, seriously affecting the efficiency of receiving police reports.

[0007] (3) Inaccurate answers: Due to the complex relevance and applicability among legal provisions, existing systems often cannot automatically judge and give the most suitable answer for the current situation. This may lead to misunderstandings or omissions when operators answer the questions of alarm callers, thus affecting the subsequent handling of police situations.

[0008] (4) Lack of intelligent support: Most existing legal knowledge base systems adopt simple keyword matching technologies and lack intelligent analysis and reasoning capabilities. This makes the system often unable to provide satisfactory answers when dealing with complex and changeable legal questions.

[0009] In summary, there are many problems in the prior art in the legal knowledge Q&A of operators, and there is an urgent need for a more efficient, accurate, and intelligent solution. In this context, we propose an innovative solution: relying on advanced large language model technology, deeply learning and mining various existing legal documents to build a comprehensive, accurate, and real-time updated legal regulations knowledge base. This knowledge base can not only cover the core legal provisions such as the Constitution, the Criminal Law, and the Civil Code, but also integrate multi-level legal information such as administrative regulations, local regulations, and departmental rules to form a complete and systematic legal system.

[0010] Through this knowledge base, we can provide fast and accurate intelligent question-answering services for operators. When the alarm caller asks relevant legal questions, the system can quickly retrieve and analyze relevant legal provisions, and give clear answers and explanations. This not only greatly improves the connection efficiency, but also ensures the accuracy and authority of legal consultations. In addition, our large language model technology also has the ability to continuously learn and optimize. As the legal system is constantly updated and improved, our knowledge base can also be updated in real time to ensure that the legal services provided always conform to the latest laws and regulations. This further enhances the practicality and reliability of the system. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to provide a legal knowledge question-answering system and method based on a large language model in view of the deficiencies of the background technology; based on the RAG retrieval-enhanced generation related technology, it aims to efficiently manage the knowledge of a large number of laws and regulations, and recall relevant legal knowledge through accurate knowledge retrieval to enhance the effect and accuracy of the LLM large language model when providing legal consultation services.

[0012] The present invention adopts the following technical solutions to solve the above technical problems:

[0013] A legal knowledge question-answering system based on a large language model, comprising a legal regulations knowledge base construction module, a RAG retrieval module, an LLM enhancement module and a user interface;

[0014] Among them, the legal regulations knowledge base construction module is used to manage the knowledge of a large number of legal regulations from different sources through a knowledge management framework; knowledge management includes the parsing, cleaning, sharding, data vectorization and vector storage of data from different data sources and formats;

[0015] The RAG retrieval module is used to efficiently retrieve information about the user's question in the legal regulations knowledge base;

[0016] The LLM enhancement module is used to generate natural and fluent answers to legal consultation services based on large language model technology; receive the relevant legal knowledge returned by the RAG retrieval module, and combine the specific context of the user's question to generate personalized and accurate legal advice or answers;

[0017] The user interface is used to provide a friendly interaction experience for users, allowing users to input legal questions or query keywords, and display the answer results generated by the LLM enhancement module.

[0018] As a further preferred solution of the legal knowledge question-answering system based on a large language model of the present invention, the legal regulations knowledge base construction module specifically includes the following;

[0019] (1) Data collection: Collect laws and regulations data from multiple sources, including but not limited to national laws and regulations, local regulations, judicial interpretations, and administrative rules; data sources include government official websites, legal databases, professional legal institutions, and also include proprietary data provided by customers;

[0020] (2) Text extraction: Convert the collected data in PDF, HTML, and Word formats into a standardized plain text format; The excellent OCR and layout recognition technologies are used to effectively ensure the integrity of the text;

[0021] (3) Paragraph block division: Blocking refers to splitting the input long text into concise and meaningful units; Instead of using fixed-length splitting, semantic splitting is used to avoid the loss of context semantic information; Semantic splitting uses the discourse analysis tool of NLP to extract the main relationships between paragraphs, and combines all paragraphs containing the master-slave relationship into one paragraph; After the article is segmented, it is ensured that each text block is talking about the same thing; By selecting different data parsing and blocking methods, the optimal blocking result can be obtained for data from different sources to ensure the atomicity and integrity of the clauses. This process is similar to tuning parameters, and the blocking result can be visually displayed through the system page;

[0022] (4) Text embedding: Text embedding refers to a process of converting text data into a vector matrix. The BGE model is selected for text embedding. This model supports multiple versions of Chinese and English, and also supports fine-tuning and local deployment;

[0023] (5) Vector database storage: The process of building an index after data vectorization and writing it into the database is summarized as the data storage process. ES is selected as the vector database for the laws and regulations knowledge base.

[0024] As a further preferred solution of the present invention for constructing a legal knowledge Q&A system based on a large language model, the RAG retrieval module specifically includes the following;

[0025] (1) User question embedding: Preprocess the user question and then perform embedding; Perform natural language processing on the user input question, including word segmentation, part-of-speech tagging, and semantic understanding, to accurately capture the intention and key information of the user question; The embedded vector model selects the BGE model of the knowledge base construction module to avoid the deviation between different models;

[0026] (2) Similarity matching: Select a hybrid retrieval method to retrieve vector knowledge. By intelligently hybridizing keyword-based search, semantic search, and vector search technologies, the advantages of each method are utilized, so that the system can adapt to different query types and information needs, and ensure that it always retrieves the most relevant and context-rich information.

[0027] As a further preferred solution of the legal knowledge Q&A system constructed based on the large language model in the present invention, the LLM enhancement module specifically includes the following:

[0028] (1) Template filling: Receive the relevant legal knowledge returned by the RAG retrieval module, and generate personalized and accurate legal advice or answers in combination with the specific context of the user's question; the user's query and the retrieved additional context are filled into a prompt template;

[0029] (2) Retrieval enhancement: Integrate advanced retrieval enhancement technologies such as hyde and self-rag, accurately evaluate the relevance and support of the retrieved information, thereby significantly improving the quality of the LLM's answers and effectively reducing misjudgment or hallucination phenomena;

[0030] (3) Intelligent refusal to answer: Have the function of intelligent refusal to answer in the scenario of factual conflict to ensure the accuracy of the output.

[0031] A method for constructing a legal knowledge Q&A system based on a large language model specifically includes the following steps:

[0032] Step 1, User input question: The user inputs a legal question or query keyword through the user interaction interface;

[0033] Step 2, Question understanding and preprocessing: The system performs natural language processing on the user's input question, including steps such as word segmentation, part-of-speech tagging, and semantic understanding, to accurately capture the intention and key information of the user's question;

[0034] Step 3, RAG retrieval: The RAG retrieval module performs efficient information retrieval in the legal regulations knowledge base according to the preprocessed user question; adopt advanced retrieval algorithms, including BM25 retrieval algorithm and TF-IDF retrieval algorithm, and combine semantic similarity calculation to recall the most relevant legal regulations articles from the knowledge base;

[0035] Step 4, Legal knowledge integration: Integrate the relevant legal knowledge returned by the RAG retrieval module, extract key information, and construct it into a format suitable for LLM processing;

[0036] Step 5, LLM generate answer: The LLM enhancement module receives the integrated legal knowledge, and generates personalized and accurate legal advice or answers in combination with the specific context of the user's question; based on industry-leading open-source large models, including ChatGLM, Qwen, InternLM, etc., ensure that the generated answers are natural, fluent and conform to legal logic;

[0037] Step 6, Result display: Display the answer result generated by the LLM enhancement module to the user through the user interaction interface, providing clear and definite legal consultation services.

[0038] Compared with the prior art, the present invention adopting the above technical solutions has the following technical effects:

[0039] A legal knowledge Q&A system and method based on large language models according to the present invention are related to RAG (Retrieval-Augmented Generation) technology, aiming to efficiently manage knowledge of a vast amount of laws and regulations, and recall relevant legal knowledge through accurate knowledge retrieval to enhance the effectiveness and accuracy of large language models (LLMs) when providing legal consultation services. By combining RAG technology and LLM technology, efficient knowledge management and accurate retrieval of a vast amount of laws and regulations are achieved, providing an innovative and efficient solution for the field of legal consultation services. This technology enables developers to improve the accuracy of answers by simply connecting relevant knowledge bases and providing additional inputs to the LLM without retraining the entire LLM for each specific task:

[0040] 1. By introducing RAG technology, the present invention realizes knowledge retrieval and recall for relevant questions, can quickly locate the most relevant and accurate legal knowledge from a vast amount of laws and regulations, thus significantly improving the accuracy and efficiency of consultation services;

[0041] 2. By combining RAG technology and LLM (Large Language Model), the present invention realizes intelligent understanding and analysis of user questions; can dynamically retrieve and integrate relevant legal knowledge according to the specific content and background of user questions to generate personalized and accurate answers, thus enhancing the flexibility and adaptability of the system;

[0042] 3. By introducing RAG technology, the present invention realizes dynamic management and update of laws and regulations knowledge; can automatically detect and integrate newly promulgated or revised legal provisions to ensure that the provided legal services are always consistent with the latest laws and regulations, thus enhancing the practicality and reliability of the system;

[0043] 4. This patent enhances the interpretability of legal answers through a visual reasoning chain to ensure that the reasoning process is clear and traceable. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flowchart of the legal regulations knowledge Q&A system of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] The technical solutions of the present invention will be further described in detail below with reference to the drawings:

[0046] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention. The present invention will be described in detail below according to the accompanying drawings and preferred embodiments, and the purpose and effect of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0047] The present invention creatively addresses the key problems existing in the current legal consulting service field and proposes an innovative solution. This solution realizes efficient knowledge management of a vast amount of laws and regulations based on RAG (Retrieval-Augmented Generation) related technologies and combines LLM (Large Language Model) to achieve intelligent understanding and analysis of user questions. Specifically, the present invention aims to solve the following technical problems:

[0048] The problem of ensuring the timeliness and accuracy of legal data: Facing the huge and complex legal system, existing legal consulting service systems often have difficulty accurately retrieving and recalling relevant legal knowledge in a short time. This not only reduces the efficiency of consulting services but also may lead to misleading legal advice for users due to inaccurate information retrieval. The present invention realizes knowledge retrieval and recall of related problems by introducing RAG technology, and can quickly locate the most relevant and accurate legal knowledge from a vast amount of laws and regulations, thus significantly improving the accuracy and efficiency of consulting services.

[0049] The flexibility and adaptability of the system: When dealing with user questions, existing legal consulting service systems often rely on preset templates or rules to answer, lacking flexibility and personalization. This results in the system being difficult to give satisfactory answers when facing complex and changing questions. The present invention realizes intelligent understanding and analysis of user questions by combining RAG technology and LLM (Large Language Model). The system can dynamically retrieve and integrate relevant legal knowledge according to the specific content and background of user questions, generating personalized and accurate answers, thereby enhancing the flexibility and adaptability of the system.

[0050] Practicality and reliability issues of the system: Existing legal consultation service systems face significant challenges in terms of updating and maintenance. As the legal system continues to develop and improve, new laws and judicial interpretations are constantly introduced, and the system needs to be continuously updated to maintain its accuracy and practicality. However, existing systems often lack an efficient update mechanism, resulting in the system being unable to make timely adjustments when faced with new laws. This invention achieves dynamic management and update of legal regulations knowledge by introducing RAG technology. The system can automatically detect and integrate newly promulgated or revised laws, ensuring that the legal services provided are always consistent with the latest laws and regulations, thereby enhancing the practicality and reliability of the system.

[0051] Interpretability and transparency issues of legal reasoning: In the legal field, the legal reasoning process often involves complex interpretation and derivation of provisions. Existing LLMs lack interpretability of the reasoning process and may produce the "hallucination" phenomenon, that is, the generated answers lack logic or are irrelevant to the question. Legal answers not only need to be accurate but also need to have sufficient transparency to facilitate users' understanding of the reasoning process. This patent enhances the interpretability of legal answers by visualizing the reasoning chain, ensuring that the reasoning process is clear and traceable.

[0052] In summary, this invention aims to efficiently manage knowledge of a vast amount of laws and regulations by introducing RAG-related technologies, and combine LLM technology to achieve intelligent question-answering services, so as to solve key problems such as inaccurate knowledge retrieval, lack of flexibility and personalization in the existing technology, and difficulties in updating and maintenance. This innovative solution will bring a revolutionary change to the field of legal consultation services, improving service quality and user experience.

[0053] This invention is based on RAG (Retrieval-Augmented Generation) related technologies, aiming to efficiently manage knowledge of a vast amount of laws and regulations, and recall relevant legal knowledge through accurate knowledge retrieval to enhance the effectiveness and accuracy of large language models (LLMs) when providing legal consultation services. The specific structure, detailed steps, and key parameters of this invention will be elaborated in detail below.

[0054] The specific structure includes:

[0055] (1) Legal and Regulatory Knowledge Base Construction Module: This patent conducts knowledge management on a vast amount of laws and regulations from different sources through a self-developed knowledge management framework. Knowledge management includes data parsing, cleaning, sharding, data vectorization, and vector storage for data from different sources and formats. This knowledge base stores a vast amount of legal and regulatory texts, including but not limited to national laws and regulations, local regulations, judicial interpretations, administrative rules and regulations, etc. The knowledge base adopts an efficient indexing and storage mechanism to ensure fast and accurate retrieval. The index construction process is an important initial step executed offline. First, the original data needs to be cleaned and extracted, and various file formats such as PDF, HTML, and Word are converted into standardized plain text. To adapt to the context limitations of the language model, these texts are divided into smaller and more manageable chunks, and this process is called chunking. Then, an embedding model is used to convert these chunks into vector representations. Finally, an index is created to store these text chunks and their vector embeddings in the form of key-value pairs, thus achieving an efficient and scalable search function.

[0056] (2) RAG Retrieval Module: This module conducts efficient information retrieval on user questions in the legal and regulatory knowledge base. It adopts advanced natural language processing technologies, can accurately understand the intention of user questions, and accordingly retrieve the most relevant legal and regulatory articles in the knowledge base. The similarity retrieval process involves vector similarity and text similarity, etc., and the user query is used to retrieve relevant contexts from external knowledge sources. For this purpose, the user query is processed by an encoding model to generate semantically relevant embeddings. Then, a similarity search is conducted in the vector database to retrieve the k closest data objects. Semantic search can improve the quality of RAG and is suitable for organizations that want to add a large number of external knowledge sources to their LLM applications. In this process, context retrieval is challenging in terms of scale, so it will reduce the quality of the generated output.

[0057] (3) LLM Enhancement Module: This module is based on large language model technology and can generate natural and fluent answers to legal consultation services. It receives the relevant legal knowledge returned by the RAG retrieval module and combines it with the specific context of the user question to generate personalized and accurate legal advice or answers. The user query and the retrieved additional context are filled into a prompt template. Finally, the enhanced prompt in the retrieval step is input into the LLM.

[0058] (4) User Interface: This interface provides a friendly interaction experience for users, allowing users to input legal questions or query keywords and display the answer results generated by the LLM enhancement module.

[0059] The detailed steps are as follows:

[0060] (1) User Input Question: The user inputs a legal question or query keyword through the user interface.

[0061] (2) Problem understanding and preprocessing: The system performs natural language processing on the user's input question, including steps such as word segmentation, part-of-speech tagging, and semantic understanding, to accurately capture the intent and key information of the user's question.

[0062] (3) RAG retrieval: The RAG retrieval module performs efficient information retrieval in the knowledge base of laws and regulations based on the preprocessed user question. It uses advanced retrieval algorithms such as BM25, TF-IDF, etc., combined with semantic similarity calculation, to recall the most relevant laws and regulations articles from the knowledge base.

[0063] (4) Legal knowledge integration: The system integrates the relevant legal knowledge returned by the RAG retrieval module, extracts key information, and constructs a format suitable for LLM processing.

[0064] (5) LLM generates answers: The LLM enhancement module receives the integrated legal knowledge and generates personalized and accurate legal advice or answers in combination with the specific context of the user's question. This module is based on leading open-source large models in the industry, such as ChatGLM, Qwen, InternLM, etc., to ensure that the generated answers are natural, fluent, and conform to legal logic.

[0065] (6) Result display: The system displays the answer results generated by the LLM enhancement module to the user through the user interface, providing clear and explicit legal consultation services.

[0066] The key parameters are as follows:

[0067] (1) Construction of the knowledge base of laws and regulations: The effectiveness of RAG highly depends on the quality of semantic search. If irrelevant or low-quality documents are retrieved, the response quality of the LLM will not be good. Good document quality is the basis for retrieval-augmented generation RAG.

[0068] This module provides the basic components and tools required for constructing the RAG knowledge base. Therefore, in practical applications, certain customized development can be carried out according to specific requirements. For example, this module configures different methods such as word, excel, pdf, etc. for data reading, different methods such as book, full text, layout recognition, abstract retrieval, etc. for data parsing, different slicing methods such as fixed-length segmentation, semantic segmentation, etc. for data chunking; different encoding methods such as ERNIE-Embedding V1, M3E, BGE, etc. are provided for the vector model; different databases such as FAISS, Chromadb, ES, milvus, etc. are provided for vector database storage. Thus, the appropriate data reading, parsing, slicing, vectorization, and database storage methods can be selected by comprehensively considering multiple factors such as business scenarios, hardware, and performance requirements.

[0069] RAG Retrieval: This module provides retrieval modes such as vector similarity, text similarity, and hybrid retrieval. It adopts advanced retrieval algorithms such as BM25 and TF-IDF, and combines semantic similarity calculation to recall the most relevant laws and regulations from the knowledge base. At the same time, RAG enhancement techniques such as query enhancement, index enhancement, retriever enhancement, generator enhancement, and pipeline enhancement are provided for selection.

[0070] LLM Enhancement Module: This module integrates advanced retrieval enhancement technologies such as hyde and self-rag, aiming to accurately evaluate the relevance and support of retrieved information, thus significantly improving the answer quality of the LLM and effectively reducing misjudgment or hallucination phenomena. In addition, this module has an intelligent refusal-to-answer function in scenarios where there are factual conflicts to ensure the accuracy of the output. To meet different needs, this module also provides a variety of LLM model options for users to flexibly choose.

[0071] In summary, the present invention realizes efficient knowledge management and accurate retrieval of a large number of laws and regulations by combining RAG technology and LLM technology, providing an innovative and efficient solution for the field of legal consultation services. This technology enables developers to improve the accuracy of answers by connecting relevant knowledge bases and providing additional inputs to the LLM, rather than retraining the entire LLM for each specific task.

[0072] The specific process of building a legal knowledge Q&A system based on LLM is as Figure 1 shown. It mainly includes three key modules: legal knowledge base construction, RAG retrieval, and LLM enhancement. The interaction between the user input and the AI is displayed through the user interface.

[0073] Legal Knowledge Base Construction Module:

[0074] (1) Data Collection: It is necessary to collect legal data from multiple sources, including but not limited to national laws and regulations, local regulations, judicial interpretations, administrative regulations, etc. Data sources can include government official websites, legal databases, professional legal institutions, etc., and also include proprietary data provided by customers.

[0075] (2) Extract text: We need to convert the collected data (in formats such as PDF, HTML, Word, etc.) into a standardized plain text format. At this time, various data parsing methods configured in this patent system can be selected to ensure the integrity and accuracy of information during the conversion process. This module configures different methods for data reading, such as Word, Excel, PDF, etc., and different parsing methods can be selected according to the format of the collected data. For documents such as PDF, due to the inclusion of a large number of illustrations, the text information extracted from most documents is fragmented and incomplete. Therefore, the text parsing tool designed in this patent effectively guarantees the integrity of the text by using excellent OCR and layout recognition technologies.

[0076] (3) Divide paragraph blocks: Blocking refers to splitting the input long text into concise and meaningful units because the LLM has limitations on the length of the context and cannot accept long text, and it will also add interfering extra information. Effective blocking can promote the retrieval system to accurately locate relevant context paragraphs to form a response. The quality and structure of these blocks are crucial for the effectiveness of the system and can ensure that the retrieved text is precisely customized for the user's query. This module configures different methods for data parsing, such as book, full text, layout recognition, abstract retrieval, etc., and configures slicing methods such as fixed-length splitting and semantic splitting for data blocking. This system mainly uses semantic splitting instead of fixed-length splitting of text to avoid the loss of context semantic information. Semantic splitting mainly uses the discourse parsing tool of NLP to extract the main relationships between paragraphs and merge all paragraphs containing a master-slave relationship into one paragraph. After splitting the article in this way, it is ensured that each text block is talking about the same thing. By selecting different data parsing and blocking methods, the optimal blocking result can be obtained for data from different sources to ensure the atomicity and integrity of the clauses. This process is similar to parameter tuning, and the blocking result can be visually displayed through the system page.

[0077] (4) Text embedding: Text embedding refers to a process of converting text data into a vector matrix, which directly affects the subsequent retrieval effect. Common encoding models such as ERNIE-Embedding V1, M3E, BGE, etc. After experiments, it is found that BGE has the best encoding effect in the legal field. Therefore, this system selects the BGE model for text embedding. This model supports multiple versions of Chinese and English and also supports fine-tuning and local deployment.

[0078] (5) Vector database ingestion: The process of constructing an index after data vectorization and writing it into a database can be summarized as the data ingestion process. Databases suitable for RAG scenarios include: FAISS, Chromadb, ES, Milvus, etc. Considering the legal business scenario, the magnitude of the knowledge base (in the millions) and retrieval efficiency, this system selects ES as the vector database for the legal regulations knowledge base.

[0079] RAG Retrieval:

[0080] (1) User question embedding: The user question is preprocessed and then embedded. Natural language processing is performed on the user input question, including word segmentation, part-of-speech tagging, semantic understanding, etc., to accurately capture the intent and key information of the user question. The BGE model of the knowledge base construction module is selected as the embedding vector model to avoid bias between different models.

[0081] (2) Similarity matching: This module provides retrieval modes such as vector similarity, text similarity, and hybrid retrieval. Advanced retrieval algorithms are adopted, such as BM25, TF-IDF, etc., combined with semantic similarity calculation, to recall the most relevant legal regulations articles from the knowledge base. At the same time, RAG enhancement techniques such as query enhancement, index enhancement, retriever enhancement, generator enhancement, and pipeline enhancement are provided for selection. This system selects the hybrid retrieval method to retrieve vector knowledge. By intelligently mixing techniques such as keyword-based search, semantic search, and vector search, it takes advantage of each method to enable the system to adapt to different query types and information needs, ensuring that it always retrieves the most relevant and context-rich information.

[0082] LLM Enhancement:

[0083] (1) Template filling: This module receives the relevant legal knowledge returned by the RAG retrieval module and combines it with the specific context of the user question to generate personalized and accurate legal advice or answers. The user query and the retrieved context are filled into a prompt template.

[0084] (2) Retrieval enhancement: This module integrates advanced retrieval enhancement techniques such as hyde and self-rag. Among them, self-rag mainly adopts a self-feedback mechanism. If there are some ambiguous confidences in the top k text blocks recalled initially, then these text blocks can be recalled again to confirm whether they can really answer this question. Hyde is to generate an answer by the LLM based on the user question, and then both the question and the generated answer are converted into vectors for re-retrieval of the knowledge base. The above two retrieval enhancement techniques significantly improve the answer quality of the LLM and effectively reduce misjudgment or hallucination phenomena by accurately evaluating the relevance and support of the retrieved information in the legal field.

[0085] (3) Intelligent refusal to answer: When encountering scenarios with factual conflicts, this module has the function of intelligent refusal to answer to ensure the accuracy of the output.

[0086] User interface: As the window for users to interact with the system, it allows users to input legal questions or query keywords and displays the answer results generated by the system.

[0087] The following describes a specific example of the legal knowledge Q&A of this system:

[0088] Step 1. User inputs a legal question: The user inputs a specific legal question through the user interface, such as "What are the evidences in criminal cases?".

[0089] Step 2. Preprocessing and natural language understanding: The preprocessing module performs word segmentation, part-of-speech tagging, etc. on the user's input question, and identifies key information such as "criminal", "case", and "evidence". At the same time, the semantic understanding module analyzes the intention of the user's question to determine that it needs to obtain relevant information about the evidences in criminal cases.

[0090] Step 3. RAG retrieval and legal knowledge recall: The RAG retrieval module uses hyde and self-rag technologies to retrieve the most relevant legal regulations and provisions in the legal knowledge base for the user's question. The retrieval results may include the relevant provisions on the evidences in criminal cases in the Criminal Procedure Law.

[0091] Step 4. Legal knowledge integration: The legal knowledge integration module integrates the retrieval results, extracts key legal provisions and explanations, and constructs a format suitable for LLM processing.

[0092] Step 5. LLM generates an answer: The LLM enhancement module selects a suitable LLM model (such as InternLM-20B), receives the integrated legal knowledge, and combines the specific context of the user's question to generate personalized and accurate legal advice or answers. For example, the answer includes "physical evidence; documentary evidence; witness testimony; victim's statement; confession and defense of criminal suspects and defendants; expert opinion; transcripts of inquests, examinations, identifications, and investigative experiments; audio-visual materials and electronic data." At the same time, the LLM enhancement module also performs factual conflict detection on the generated answer to ensure the consistency and accuracy of the answer content. If a factual conflict is detected, the system will intelligently refuse to answer to avoid misleading users.

[0093] Step 6. Result display: The result display module displays the answer results generated by the LLM enhancement module to the user through the user interface. The user can see clear and definite legal advice or answers, as well as relevant legal regulations and provisions and explanations.

[0094] Those of ordinary skill in the art can understand that the above are only preferred examples of the invention and are not used to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, etc. made within the spirit and principles of the invention shall be included within the protection scope of the invention. All technical features in this embodiment can be freely combined according to actual needs.

[0095] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A legal knowledge question-answering system based on a large language model, characterized by: It includes the legal and regulatory knowledge base construction module, RAG search module, LLM enhancement module and user interaction interface; Among them, the legal and regulatory knowledge base construction module is used to manage the massive amount of laws and regulations from different sources through the knowledge management framework; knowledge management includes parsing, cleaning, sharding, data vectorization and vector storage of data from different data sources and formats; RAG retrieval module, used to efficiently search for information in the legal and regulatory knowledge base for user questions; The LLM enhancement module is used to generate natural and fluent legal consulting service answers based on large language model technology; it receives relevant legal knowledge returned by the RAG retrieval module and generates personalized and accurate legal advice or answers based on the specific context of the user's question; The user interaction interface is used to provide users with a friendly interactive experience, allowing users to enter legal questions or query keywords and display the answer results generated by the LLM enhancement module.

2. According to claim 1, a legal knowledge question-answering system based on a large language model is characterized by: The legal and regulatory knowledge base construction module specifically includes the following: (1) Data collection: Collect legal and regulatory data from multiple sources, including but not limited to national laws and regulations, local regulations, judicial interpretations, and administrative rules; data sources include official government websites, legal databases, professional legal institutions, and also proprietary data provided by customers; (2) Extract text: Convert the collected data including PDF, HTML, and Word formats into standardized plain text format; use excellent OCR and layout recognition technology to effectively ensure the integrity of the text; (3) Divide the paragraphs into blocks: Block segmentation refers to dividing the input long text into concise and meaningful units. Semantic segmentation is used instead of fixed-length segmentation to avoid the loss of contextual semantic information. Semantic segmentation uses NLP paragraph analysis tools to extract the main relationship between paragraphs and merge all paragraphs containing master-slave relationships into one paragraph. After the article is segmented, it is ensured that each text block is talking about the same thing. By selecting different data analysis and block segmentation methods, the optimal block segmentation results can be obtained for data from different sources to ensure the atomicity and integrity of the terms. This process is similar to parameter adjustment, and the block segmentation results can be visualized on the system page. (4) Text embedding: Text embedding refers to the process of converting text data into a vector matrix. The BGE model is used for text embedding. The model supports multiple versions in Chinese and English, and also supports fine-tuning and local deployment. (5) Vector storage: The process of constructing an index after data vectorization and writing it into the database is summarized as the data storage process. ES is selected as the vector library of the legal and regulatory knowledge base.

3. According to claim 1, a legal knowledge question-answering system based on a large language model is characterized by: The RAG retrieval module specifically includes the following: (1) User question embedding: User questions are preprocessed and then embedded. Natural language processing is performed on the questions input by users, including word segmentation, part-of-speech tagging, and semantic understanding, to accurately capture the intention and key information of the user questions. The embedding vector model uses the BGE model of the knowledge base building module to avoid deviations between different models. (2) Similarity matching: A hybrid retrieval method is used to retrieve vector knowledge. By intelligently mixing keyword-based search, semantic search, and vector search techniques to leverage the strengths of each method, the system can adapt to different query types and information needs, ensuring that it always retrieves the most relevant and context-rich information.

4. According to claim 1, a legal knowledge question-answering system based on a large language model is characterized by: The LLM enhancement module specifically includes the following: (1) Template filling: Receive the relevant legal knowledge returned by the RAG search module and combine it with the specific context of the user's question to generate personalized and accurate legal advice or answers; the user query and the retrieved additional context are filled into a prompt template; (2) Retrieval enhancement: It integrates the advanced retrieval enhancement technologies of hyde and self-rag to accurately evaluate the relevance and support of the retrieved information, thereby significantly improving the quality of LLM answers and effectively reducing misjudgments or phantom phenomena; (3) Intelligent refusal to answer: When encountering a scenario of factual conflict, it has an intelligent refusal to answer function to ensure the accuracy of the output.

5. A method for constructing a legal knowledge question-answering system based on a large language model according to any one of claims 1 to 4, characterized in that: The specific steps include: Step 1, user inputs question: the user inputs legal question or query keyword through the user interaction interface; Step 2: Question understanding and preprocessing: The system performs natural language processing on the questions input by the user, including word segmentation, part-of-speech tagging, semantic understanding and other steps, to accurately capture the intent and key information of the user's question; Step 3, RAG retrieval: The RAG retrieval module performs efficient information retrieval in the legal and regulatory knowledge base based on the pre-processed user questions. It uses advanced retrieval algorithms, including the BM25 retrieval algorithm and the TF-IDF retrieval algorithm, combined with semantic similarity calculation, to recall the most relevant legal and regulatory provisions from the knowledge base. Step 4, legal knowledge integration: Integrate the relevant legal knowledge returned by the RAG search module, extract key information, and construct it into a format suitable for LLM processing; Step 5, LLM generates answers: The LLM enhancement module receives the integrated legal knowledge and generates personalized and accurate legal advice or answers based on the specific context of the user's question. It is based on the industry's leading open source big models, including ChatGLM, Qwen, InternLM, etc., to ensure that the generated answers are natural, fluent and in line with legal logic; Step 6, result display: The answer results generated by the LLM enhancement module are displayed to the user through the user interaction interface to provide clear and unambiguous legal consulting services.

Images