Legal provision retrieval method for legal data embedding optimization and retrieval effect evaluation

Abstract

The invention provides a legal provision retrieval method and retrieval effect evaluation based on legal data embedding optimization, and is characterized in that the method comprises the following steps: obtaining a user query request, extracting related keywords by utilizing a semantic information extension prompt-based keyword method, constructing a legal query-legal provision fine adjustment data set training word embedding model, and establishing a legal query-legal provision fine adjustment data set training word embedding model; the model is used for constructing a vector database, the weight of retrieval results is dynamically adjusted through retrieval legal provisions combining sparse retrieval and dense retrieval so as to merge the results, and a reference data set is constructed to evaluate the retrieval effect. Compared with the prior art, the legal service system has rigorous words and a fixed structure, is suitable for being systematically stored in a knowledge base, effectively supports various application scenes such as legal consultation, case analysis and legal education, and further improves the efficiency and accuracy of legal services. The invention further provides a reference data set construction method for evaluating the retrieval effect, and the problem that effective evaluation standards are lacked in the field of legal provision retrieval is solved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and information retrieval technology, and in particular to a legal text retrieval method and retrieval effect evaluation based on legal data embedding optimization. Background Art

[0002] Personalized legal consultation usually requires legal professionals to handle. Such problems can be found everywhere in life and are one of the main tasks of legal workers. Whether the answer to the question is correct depends on whether the appropriate and complete legal provisions are matched. However, the law has the characteristics of high rigor, large amount of knowledge, and fast update. How to match the appropriate legal provisions for personalized legal issues is a very challenging problem, which brings great challenges to human problem solving. Even experienced legal professionals may make mistakes. Therefore, automatic retrieval of legal provisions for personalized issues has become a technology of concern in the industry. However, the accuracy and completeness of existing related technologies in legal retrieval are still not enough for practical use.

[0003] At present, the retrieval of legal provisions mainly relies on manually formulated rules or machine learning technology. Common retrieval methods include: keyword-based exact matching, vector space model (Vector Space Model), and BM25 (BestMatching 25) and other information retrieval technologies. These methods have achieved certain results in improving retrieval efficiency, but because they only rely on the similarity of the vocabulary of the sentence itself for information retrieval, they cannot effectively deal with legal issues with diverse and complex language expressions. Especially for ordinary users, due to the lack of professional legal knowledge, it is difficult to accurately use legal terms to describe the problem, resulting in poor retrieval results. There are significant differences between professional legal documents and daily language expressions. For example, users may use expressions such as "reducing heavy pollution", while legal provisions may use more professional terms such as "environmental protection", which increases the difficulty of retrieval. In recent years, the introduction of large language models (LLM) has become a new trend in the legal field, bringing opportunities for processing more complex natural languages ​​to solve legal consulting problems. LLM has the ability to understand and transform professional legal terms, and can effectively process the diverse expressions of ordinary users and convert them into legal professional terms. This conversion not only improves the accuracy of retrieval, but also reduces the threshold for ordinary users to use it. At the same time, the combination of vector databases further enhances the performance of large language models. By storing vector representations of text, vector databases can deeply understand and retrieve semantics, thereby overcoming the limitations of traditional keyword-based matching methods. At the same time, vector databases can also update legal knowledge in a timely manner, overcoming the limitations of model knowledge timeliness.

[0004] Combining a large language model with a vector database can significantly improve the accuracy of personalized question information retrieval to better serve a wide range of user groups. This technology is called Retrieval Augmented Generation technology. The application of this technology in the legal field is still in its early stages and has not yet been widely adopted. Most legal retrieval systems still rely on traditional methods such as exact keyword matching and predefined rules. The current mainstream retrieval methods include dense retrieval and sparse retrieval. Dense retrieval uses vector representations generated by deep learning to capture the deep semantic relationship between queries and documents, but it does not work well in accurate matching. Sparse retrieval is based on the traditional keyword matching method, which emphasizes the matching of explicit features, but is difficult to deal with problems such as synonymous matching. Legal retrieval needs to take into account both deep semantic understanding and accurate legal term matching. Combining the advantages of both can more efficiently meet personalized needs. Combined with the semantic understanding ability of the large model in RAG, it can better make up for the shortcomings of traditional methods and achieve more accurate and comprehensive information retrieval. Therefore, it is very necessary to introduce RAG technology in the legal field.

[0005] The legal text retrieval of the prior art has the following problems and shortcomings: 1) The wording of personalized legal issues is different from professional legal terms. Directly using unprocessed personalized legal issues for retrieval may lead to irrelevant retrieval results. It is necessary to expand and refine the semantic information.

[0006] 2) Converting text into a semantic representation in vector form relies on word embedding models. Although mainstream open source word embedding models perform well on general corpora, they are still insufficient in capturing semantic relationships in the legal field.

[0007] 3) Large language models have context length limitations. Too many retrieved documents may lead to problems such as "forgetting", affecting the quality of generation. Therefore, it is necessary to balance the recall rate and precision rate to ensure that the most relevant legal provisions are included while minimizing the amount of retrieved text. Summary of the invention

[0008] The purpose of the present invention is to provide a legal text retrieval method and retrieval effect evaluation that is optimized with legal data embedding in response to the deficiencies of the prior art. It adopts a legal text retrieval enhancement technology that combines dense retrieval with sparse retrieval, optimizes for the unique needs in legal text retrieval, and improves the recall and precision of legal text retrieval, so that legal texts can provide authoritative legal basis for them. They have rigorous wording and a fixed structure, and are suitable for being systematically stored in a knowledge base. They can effectively support a variety of application scenarios such as legal consulting, case analysis, and legal education, thereby improving the efficiency and accuracy of legal services. The present invention dynamically adjusts the weights of retrieval results to merge results, thereby improving retrieval accuracy, and better solves the problem of lack of effective evaluation standards in the field of legal text retrieval. It is particularly suitable for legal text queries by non-professional users, greatly The quality and reliability of retrieval are improved, the method is simple and has good application prospects.

[0009] The specific technical solution to achieve the purpose of the present invention is: a legal article retrieval method for embedding and optimizing legal data, which is characterized by using a legal article retrieval enhancement technology combining dense retrieval and sparse retrieval to The search for legal texts is optimized to meet the unique needs of legal text retrieval. The search for legal texts specifically includes the following steps: S1: Obtaining the original query request of the user, wherein the original query request is a natural language text such as a legal question, case consultation, legal provision query, etc.; S2: Using the keyword extraction method based on semantic information extension prompts, extract legal keywords related to the original query request; S3: Build a legal query-legal text fine-tuning dataset to train a legal semantics-sensitive word embedding model; S4: construct a vector database using the word embedding model obtained in S3 to store the original text of legal provisions, corresponding vectors, and related metadata; S5: Use the keywords extracted in S2 to perform sparse search through the vector database constructed in step S4, and the word embedding model obtained in step S3 converts the original query into word embedding and performs dense search through the vector database constructed in step S4; S6: By dynamically adjusting the weights of the sparse search and dense search results, the two search results are merged to obtain the final search result.

[0010] The keyword extraction method based on semantic information expansion prompt in step S2 comprises the following steps: S2-1: Obtain the original query request input by the user; S2-2: construct a first prompt word in combination with the original query, input the first prompt word into the large model, obtain a preliminary answer result, and integrate the original query and the preliminary answer result; S2-3: constructing a second prompt word based on the integrated content obtained in step S2-2, inputting the second prompt word into the model, and obtaining a preliminary keyword list extracted from the integrated content and merged by synonymous substitution; S2-4: constructing a third prompt word based on the preliminary keyword list obtained in step S2-3, inputting the third prompt word into the model, and obtaining a legal term prompt word list converted by performing hierarchical structure modeling and association on the preliminary keywords; S2-5: Combine the above keywords to obtain a keyword list related to the original query request.

[0011] The legal query-legal provision fine-tuning dataset in step S3 includes legal queries and positive and negative samples corresponding to the legal queries, wherein the legal queries are obtained by collecting real administrative law enforcement cases; the positive samples are constructed based on the legal provisions corresponding to the cases; the negative samples are obtained by calculating the relevance scores between the candidate legal provision set and the legal query, screening the 10 legal provisions with the highest scores, and removing the data overlapping with the positive samples.

[0012] The candidate legal text set is constructed in the following way: obtain the latest legal texts from public legal databases; clean the texts, remove redundant symbols and perform standardization; segment the texts based on legal texts; add metadata to each legal text, including legal name, text number, chapter number, whether it is invalid, legal category and keywords; and combine the above to obtain a structured data set suitable for legal text retrieval.

[0013] The dynamic weight adjustment method in step S6 takes into account the query length, the search repetition rate, and the characteristics of the legal issue, and performs weight adjustment based on the word characteristics of the user's query to obtain a re-ranked mixed search result.

[0014] A retrieval effect evaluation of a legal article retrieval method with legal data embedding and tuning, wherein a benchmark data set is constructed for effect evaluation of different retrieval methods. The purpose is to: collect public multiple-choice question data in the legal field; use regular expressions to extract question numbers, titles, answers, corresponding legal provisions and parsing information from the data source, and convert them into JSON format; for each question, determine whether its candidate options are exclusive options; if the options are exclusive, convert the question into a question form, and set the correct answer to the corresponding option; if the options are non-exclusive, for each option in each question, extract the option content and substitute it into the question stem, merge it into a judgment question, and mark the correct or incorrect information according to the question content and fill it into the answer; according to the legal provisions and parsing information, extract the legal provision information corresponding to each option; merge all the above data to form a structured legal question-answering dataset containing legal provision information.

[0015] Compared with the prior art, the present invention integrates RAG technology into the legal field and optimizes it for the unique needs in legal text retrieval. Legal practice is highly dependent on accurate information sources. Legal provisions can provide authoritative legal basis for it, and have rigorous wording and fixed structure, which are suitable for being systematically stored in the knowledge base. It can effectively support various application scenarios such as legal consultation, case analysis, and legal education, thereby improving the efficiency and accuracy of legal services. It better solves the differences between personalized legal issues input by users and standard legal terms, as well as the lack of in-depth understanding of legal semantics by the retrieval system. The method is simple and effective, which greatly improves the recall rate and precision of legal text retrieval in the legal RAG scenario, optimizes the overall effect of legal text retrieval, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the present invention; Figure 2 Extract flow chart for keywords; Figure 3 Construct a flowchart for the benchmark dataset. DETAILED DESCRIPTION

[0017] The following takes administrative law data as an example (including 1,500 case data and 85 laws and regulations), assuming that a large amount of legal question and answer data and legal and regulatory data have been collected, and the vector database and large language model have been deployed. The present invention is further explained in conjunction with the accompanying drawings and embodiments. Example

[0018] See also Figure 1 , the legal text search is carried out according to the following steps: S101: Receive a user's query request, which can be a legal question, case consultation, or direct legal text query expressed in natural language. For example, "Can the agreement between auto parts supplier Company A and auto parts supplier Company B to reduce the production of heavily polluting models, which is clearly evidenced, not be considered a monopoly agreement?" This type of question is an original query request.

[0019] S102: Use a keyword extraction method based on semantic information expansion prompts to identify and extract the keywords most relevant to the original query request. This method is a prompt method based on a large language model. The prompt model performs semantic information expansion on the original query request to extract as many keywords as possible related to the original query, and further constructs a hierarchy and associates the keywords, converting them into legal professional terms, and finally obtains a list of keywords most relevant to the original query for further sparse retrieval. For example, in the example mentioned above, keywords such as "monopoly agreement", "environmental protection", "conclusive evidence", "cooperative behavior", and "market competition" may be extracted.

[0020] S103: In order to further optimize the effect of legal text retrieval, the present invention constructs a legal query-legal text fine-tuning dataset. The collected administrative law enforcement data includes cases and corresponding legal texts, and some data where the problem is unclear and the corresponding legal texts are outdated are removed, and the remaining data is uniformly formatted. The fine-tuning dataset is constructed by contrastive learning, including legal queries, positive samples corresponding to legal queries, and negative samples corresponding to legal queries. Among them, the legal query is the collected case problem. For the positive samples corresponding to the legal query, the legal text corresponding to the case is directly sampled; for the negative samples, the candidate legal text dataset is first constructed, and the similarity between the legal query and each law in the legal text dataset is calculated, and the 10 legal texts with the highest similarity are selected, and the legal texts that are not semantically relevant to the case are screened out and sampled as negative samples. After the construction is completed, the constructed fine-tuning dataset is used to fine-tune the word embedding model bge-large-zh-v1.5. The fine-tuning method in FlagEmbedding is used to obtain the fine-tuned word embedding model ft-law, which can provide better semantic representation for legal natural language texts.

[0021] S104: The natural language text in the candidate legal provision dataset is vectorized using the word embedding model obtained in step S103. Next, a new database statute-ft is created in the vector database milvus, and the vector vector corresponding to the legal provision obtained after vectorization, the text content before processing, and the metadata information id, title, provision, etc. of the legal provision are stored in statute-ft together, and an index is constructed for the vector using the HNSW method.

[0022] S105: Use the word embedding model ft-law obtained in step S103 to convert the user's original query request into a word embedding representation. The word embedding representation obtained here is a dense vector. Then, a dense search is performed through the vector database statute-ft constructed in step S104, and the top_k parameter is set to 5 to obtain the 5 data with the highest semantic similarity to the word embedding representation of the user's query. Next, a sparse search is performed using the keywords extracted in step S102. For example, for the aforementioned query, keywords such as "monopoly agreement", "environmental protection", "conclusive evidence", "cooperative behavior", and "market competition" will be used to search for matching items in the legal text database. The BM25 method is used here for retrieval, and top_k is set to 5. The 5 most relevant data are retrieved through keyword matching.

[0023] S106: A dynamic weight adjustment method is used to merge the results of sparse search and dense search. Here, the initial weights of sparse search and dense search are set to α and β, and their sum should be 1. Initially, the weights can be equally divided, that is, α = 0.5, β = 0.5. Next, the query length of the user's query request is calculated, and the query type is identified. For example, for the aforementioned query, it is a legal issue consultation with complex semantics, involving the applicable conditions and specific situational judgments of the antitrust law, which requires an in-depth understanding of semantics and legal principles, and is more suitable for dense search; at the same time, because the query includes multiple phrases and legal terms, it is longer but not complex as a whole, and it requires semantic association and understanding of multiple concepts, so that dense search has an advantage in capturing its core meaning. Therefore, here, the weight of dense search is set higher, and the weight of sparse search is relatively low. In addition, the repetition rate of the results in the two searches is calculated. If the repetition rate is high, the weight difference of the two searches is set to be low; if the repetition rate is low, the opposite is true. For queries with shorter length or more legal terms, keyword matching is more efficient, so the weight of sparse search is increased. Finally, proceed to step S107 and end.

[0024] See also Figure 2 Taking the query request "Can the agreement between auto parts supplier Company A and auto parts supplier Company B that reduces the production of heavy-polluting models with solid evidence not be considered a monopoly agreement?" as an example, keyword extraction based on semantic information expansion prompts is performed in the following steps: S201: Obtaining an original query request input by a user.

[0025] S202: Construct the first prompt word in combination with the original query: "Please understand the following question, think about it step by step and answer it: [original query request]", input the first prompt word into the pre-trained large model, and obtain the preliminary answer result: "This question involves the identification of monopoly agreements in the Anti-Monopoly Law. According to the Anti-Monopoly Law, agreements between competitors that restrict competition are generally identified as monopoly agreements. However, in this case, the following points need to be considered: Purpose of the agreement: Reducing the production volume of heavily polluting models may be for environmental protection reasons rather than simply restricting competition. Market impact: It is necessary to evaluate the actual impact of the agreement on the relevant market competition. Reasonableness: If it can be proved that the agreement is conducive to public interests such as technological progress and environmental protection, it may constitute an exemption. Therefore, although it appears to be an agreement between competitors, it may not be identified as a monopoly agreement if it can be proved that its purpose is legitimate, the impact is limited and reasonable. However, the final determination requires the regulatory agency to conduct a comprehensive assessment based on the specific circumstances." Then, the original query and the preliminary answer results are integrated together to form a merged query text.

[0026] S203: Based on the merged query text obtained in step S202, construct the second prompt word: "Please extract keywords from the following text, and then replace the extracted keywords with professional terms that are more suitable for legal text retrieval. And output in list form: [merged query text]", input the second prompt word into the large language model to obtain a preliminary keyword list. For example: "auto parts, monopoly agreement, heavy pollution model, production quantity, antitrust law, competitor, restriction of competition, environmental considerations, market impact, technological progress, environmental protection, public interest".

[0027] S204: Based on the preliminary keyword list obtained in step S203, construct the third prompt word, "Please perform hierarchical modeling on the following keywords and convert them into more professional legal terms: [preliminary keyword list]", input the third prompt word into the model, and obtain the hierarchical prompt word list obtained by performing hierarchical modeling and association on the preliminary keywords. For example: "Anti-monopoly law 1.1 Monopoly agreement 1.1.1 Horizontal monopoly agreement 1.1.2 Vertical monopoly agreement 1.2 Market dominance 1.3 Concentration of operators; Agreement content 2.1 Restriction on production 2.2 Restriction on sales; Industry characteristics 3.1 Auto parts industry 3.2 Heavy pollution industry; Defense reasons 4.1 Environmental protection 4.2 Technological progress 4.3 Public interest; Legal evaluation elements 5.1 Purpose of the agreement 5.2 Market impact 5.3 Reasonableness analysis".

[0028] S205: Combine all the keywords obtained in the above steps to obtain the final keyword list related to the original query request. For example: "Antitrust Law, Monopoly Agreement, Horizontal Monopoly Agreement, Vertical Monopoly Agreement, Market Dominance, Concentration of Operators, Restriction of Production, Restriction of Sales, Auto Parts Industry, Heavy Pollution Industry, Environmental Protection, Technological Progress, Public Interest, Purpose of Agreement, Market Impact, Reasonableness Analysis". These keywords will be used for subsequent sparse retrieval to find the legal provisions most relevant to the original query. Finally, enter step S206 and end.

[0029] See also Figure 3 , a benchmark dataset is constructed to evaluate the effectiveness of different retrieval methods. Taking the 500-question data of market supervision practice training as an example (including 500 legal questions in the form of multiple-choice questions), the construction of the benchmark dataset is carried out according to the following steps: S301: Obtain legal question and answer data. The data obtained here includes the original questions of 500 questions in market supervision practice training, relevant laws and answers. By cleaning the data, removing unnecessary punctuation, and formatting the data, the legal question and answer text is obtained.

[0030] S302: Extracting various components of each question through a predefined regular expression pattern, including question number, title, answer, legal provisions corresponding to the question, and analysis.

[0031] S303: Start traversing each question.

[0032] S304: Determine whether there are unprocessed questions. If yes, proceed to step S305 to process the current question; if no, end.

[0033] S305: Process the current question.

[0034] S306: Determine whether the options are exclusive, that is, whether the question must be a single-choice question. If there is a mutually exclusive relationship between the options, it is considered exclusive. If it is exclusive, execute step S307 to convert the question into a question sentence, and execute step S308 to mark the answer to the question; if it is not exclusive, enter step S309 to process each option and generate a judgment question, and extract the corresponding legal provisions and answers of the options in step S310.

[0035] S307: Convert the question into a question sentence.

[0036] S308: Mark the answer to the question.

[0037] S309: Process each option and generate a true or false question.

[0038] S310: Extract the legal provisions and answers corresponding to the options.

[0039] S311: Merge the data and save them into a json file to form a structured data set, and then proceed to S312 to end.

[0040] The above is only a further explanation of the present invention and is not intended to limit this patent. Any equivalent implementation of the present invention should be included in the scope of the claims of this patent.

Claims

1. A legal article retrieval method with optimized legal data embedding, characterized in that: The method of combining sparse retrieval and dense retrieval is adopted to realize the personalized needs of legal text retrieval through keyword prompts and legal data embedding optimization. The method specifically includes the following steps: S1: Get the original query request input by the user; S2: Extract keywords related to the original query request using a keyword extraction method based on semantic information extension prompts; S3: Build a legal query-legal text fine-tuning dataset to train a legal semantics-sensitive word embedding model; S4: Build a vector database based on the word embedding model to store the original text of legal provisions, corresponding vectors and related metadata; S5: Perform sparse search on the extracted keywords through the vector database, convert the original query into word embedding using the word embedding model, and perform dense search through the vector database; S6: Use the dynamic weight adjustment method to merge the results of sparse retrieval and dense retrieval to obtain the final retrieval results.

2. According to the legal data embedding and tuning method of legal clause retrieval according to claim 1, it is characterized in that: The keyword extraction method based on semantic information expansion prompt in step S2 comprises the following steps: S2-1: Obtain the original query request input by the user; S2-2: construct a first prompt word based on the original query, input it into the large language model, obtain a preliminary answer result, and integrate the original query and the preliminary answer result using a text splicing method to obtain integrated content after semantic expansion; S2-3: constructing a second prompt word based on the integrated content obtained in step S2-2, inputting it into the large language model for extraction and merging with synonymous substitution to obtain a preliminary keyword list; S2-4: constructing a third prompt word based on the preliminary keyword list, inputting it into the large language model, modeling the preliminary keywords hierarchically, and converting them into a legal term prompt word list after association; S2-5: Combine the above keywords to obtain a keyword list related to the original query request.

3. According to the legal data embedding and tuning method of legal clause retrieval according to claim 1, it is characterized in that: The legal query-legal provision fine-tuning dataset of step S3 includes legal queries and positive and negative samples corresponding to the legal queries. The legal queries are obtained by collecting real administrative law enforcement cases; the positive samples are constructed based on the legal provisions corresponding to the cases; the negative samples are obtained by calculating the relevance scores between the candidate legal provision set and the legal queries, screening the 10 legal provisions with the highest scores, and removing the data overlapping with the positive samples.

4. The legal text retrieval method with optimized legal data embedding according to claim 3 is characterized in that: The candidate legal provisions set is constructed as follows: 1) Obtain the latest legal texts from public legal databases; 2) Clean the text by removing redundant symbols and performing standardization, and segment the text based on legal provisions; 3) Add metadata to each legal provision, including: legal name, provision number, chapter number, whether it is invalid, legal category and keywords; The results obtained in the above steps are combined to obtain a structured dataset suitable for legal text retrieval.

5. According to the legal data embedding and tuning method of legal clause retrieval according to claim 1, it is characterized in that: The dynamic weight adjustment method of step S6 considers the query length, query characteristics and retrieval repetition rate, and uses the word characteristics and term usage of the user query to adjust the weight, and obtains the re-ranked mixed retrieval results by weighting.

6. The legal article retrieval method according to claim 1 or claim 3, characterized in that: The step S3 uses the constructed fine-tuning dataset to fine-tune the word embedding model bge-large-zh-v1.5 to obtain a fine-tuned word embedding model ft-law, and the fine-tuning uses the fine-tuning method in FlagEmbedding.

7. An evaluation of the retrieval effect of the legal article retrieval method with legal data embedding optimization as claimed in claim 1, characterized in that: A benchmark dataset is constructed to evaluate the effects of different retrieval methods. The benchmark dataset is constructed as follows: 1) Collect public multiple-choice question data in the legal field, use regular expressions to extract question numbers, titles, answers, corresponding legal provisions and parsing information from the data source, and convert them into JSON format; 2) For each question, determine whether its candidate options are exclusive options. If the options are exclusive, convert the question into a question form and set the correct answer to the corresponding option. If the options are non-exclusive, for each option in each question, extract the option content and substitute it into the question stem to form a true or false question, and mark the true or false information according to the question content and fill it into the answer; 3) Based on the legal provisions and parsing information, extract the legal provision information corresponding to each option, and merge all the above data to form a structured legal question-and-answer dataset containing legal provision information.

Images