Fine-grained class case retrieval method based on large language model and application of fine-grained class case retrieval method in construction project contract dispute

Abstract

The invention relates to a fine-grained class case retrieval method based on a large language model and application of the fine-grained class case retrieval method in construction project contract disputes, and adopts a generative large language model to extract key plots in cases so as to realize fine-grained class case retrieval through semantic similarity calculation. Comprising the following steps: S1, constructing a fine-grained class case retrieval model; s2, constructing a fine-grained class case retrieval algorithm; s3, constructing a synthetic case algorithm; s4, evaluating indexes; s5, constructing an optimized fine-grained class case retrieval model; s6, summarizing and generating a judgment rule; according to the method, the LLM is adopted to generate the synthetic case, the synthetic case is used for calculating the retrieval evaluation indexes and optimizing the retrieval model, the cue word optimization module is adopted to optimize and extract cue words of key plots, the hyper-parameter optimization module is adopted to optimize the similarity threshold value, and the calculation result shows that the optimized retrieval model remarkably improves the evaluation indexes and has high reliability.

Description

Technical Field

[0001] This invention relates to the field of classification retrieval technology, and in particular to a fine-grained case retrieval method based on a large language model and its application in construction project contract disputes. Background Technology

[0002] The "Guiding Opinions of the Supreme People's Court on Unifying the Application of Law and Strengthening Case Search (Trial Implementation)," which came into effect on July 31, 2020, clearly defines similar cases and strictly regulates the applicable circumstances, scope, methods, and application of search results for similar cases. Implementing case search helps unify the application of law, enhances judicial credibility, and plays an important role in my country's judicial practice. On May 7, 2024, the Supreme People's Court issued the "Work Procedures for the Construction and Operation of People's Court Case Databases," stipulating that people's courts at all levels should search the People's Court Case Database when hearing cases and use similar cases in the database as references for judgments. This measure further highlights the important role of case search in case handling.

[0003] Automated case retrieval using computer technology can improve retrieval efficiency and quality. Computer-based similar case matching techniques include shallow text representation methods that extract fine-grained text information such as characters, words, and phrases, and deep text representation methods that employ neural network models. The precision and intelligence of case retrieval are crucial guarantees for achieving the goal of consistent judgments in similar cases.

[0004] Current intelligent case retrieval methods often treat cases as a whole, searching for the most similar candidate cases based on the query case. However, the facts of cases are often complex, making it difficult or even impossible to find completely identical cases. For example, a construction contract dispute query case may involve multiple aspects such as contract invalidity, disguised price reduction, claims, and modifications. In reality, only partially similar candidate cases can be found, or the query case may only have partial similarities with several candidate cases. In such cases, simply determining that a candidate case as a whole constitutes a similar case to the query case may carry the risk of bias. In other words, if a candidate case is only partially similar to the query case, the similarities can only be used to summarize judicial patterns and as a basis for adjudicating cases. Therefore, using fine-grained case retrieval methods can more accurately identify similar cases, thereby providing more precise data support for judicial work.

[0005] Meanwhile, with the development of natural language processing technology, deep neural networks and pre-trained models have been applied in case retrieval. In recent years, some progress has been made in applying generative large language models to case retrieval. Summary of the Invention

[0006] Based on the above technical background, the purpose of this invention is to provide a fine-grained case retrieval method based on a large language model and its application in construction contract disputes. It uses a generative large language model (hereinafter referred to as LLM) to extract key plots in cases, and then achieves fine-grained case retrieval through semantic similarity calculation.

[0007] The technical solution provided by this invention is a fine-grained case retrieval method based on a large language model, comprising the following steps: S1: Constructing a fine-grained case retrieval model

[0008] The task of fine-grained case retrieval is to find similarities between a candidate case (adjudicated case) and the query case (pending case) from a number of candidate cases (adjudicated cases), and then to summarize the rules of judgment for similar circumstances based on the judgment results of the candidate cases, which can be used as a reference for the judgment of the query case. The above process can be described as follows:

[0009] assumed To search for cases, , ,…, for One candidate case, A collection of plots = , To and Similar plot collection = .exist Contains plot , Contains plot And so on, the task of fine-grained case retrieval is based on... turn up And obtain the rules of the referee;

[0010] The fine-grained case retrieval model constructed is as follows: (1) Using LLM to search based on prompt words Extracting the details that significantly influenced the judgment. = (2) Using LLM to analyze prompts Extracting the details that significantly influenced the judgment. = (3) Perform text embedding on each key plot point to obtain a text embedding vector. The corresponding set of text embedding vectors = , The corresponding set of embedding vectors = (4) and The similarity of each element in the query is calculated, and a fine-grained case retrieval is performed based on the similarity to obtain the plot of the candidate case that is most similar to the query case.

[0011] S2: Constructing a fine-grained case retrieval algorithm

[0012] Key words were extracted from the key plot points of the case. and Constructing prompts Query Cases Case input Extract Key plot points will be considered in the candidate cases. Case input Extract This key plot point will The key plot points are input line by line into the large text vector model to obtain a sequence of text embedding vectors. =( , ,…, ),Will The key plot points are input line by line into the large text vector model to obtain a sequence of text embedding vectors. =( , ,…, ), obtain the similarity matrix = ,in, =cosine similarity( Cosine similarity is the cosine similarity of vectors. From the candidate cases, a similarity matrix sequence can be obtained. , ,…, ;

[0013] Constructing a fine-grained similarity retrieval algorithm: Algorithm 1: First, concatenate the similarity matrices of the query case and multiple candidate cases column-wise into a single matrix. ,if The maximum value of each row is greater than the similarity threshold. Then, the column number corresponding to this maximum value is mapped to a tuple consisting of candidate case numbers and key plot numbers, thereby retrieving the candidate case number and key plot number most similar to the key plot of the query case, realizing key plot retrieval. The output includes two dictionaries: The key is the key plot number of the query case, and the value is a tuple consisting of the candidate case number and the key plot number, used to generate the judgment pattern. The key is the candidate case number, and the value is a list of key plot numbers used to calculate the retrieval performance metrics.

[0014] S3: Constructing the Synthetic Case Algorithm

[0015] Currently, there is limited research on fine-grained case retrieval, and publicly available fine-grained case retrieval datasets are lacking. This invention employs dual LLM to generate synthetic cases for the evaluation and optimization of fine-grained case retrieval, constructing a synthetic case algorithm Algorithm 2: from the candidate case set... = Several case texts were randomly selected from the data, and then... For each selected case text, key plot points are extracted, and... Record the selected case number and key plot number as the retrieval target value, and then input all the extracted key plots. Combine a query case and use Check the quality of the synthesized cases. If they meet the requirements, add them to the synthesized case set. If they do not meet the requirements, regenerate them until the maximum number of synthesized cases can be reached.

[0016] S4: Evaluation Indicators

[0017] The synthesis case set can be obtained from Algorithm 2. , The Middle element = : , : , To provide the true results of fine-grained case retrieval, enter Several key plot points are obtained; input the key plot points. Obtain vector sequence This allows us to obtain the similarity matrix with each candidate case. ( = Then, Algorithm 1 can be used to obtain fine-grained retrieval results for the query cases. ,make = According to the following formula:

[0018] = (2)

[0019] It is a set of tuples, each tuple consisting of candidate case numbers and key plot numbers obtained from fine-grained case retrieval, which is the actual result of case retrieval;

[0020] = (3)

[0021] It is a set of tuples, each consisting of a candidate case number and a key plot number randomly selected when generating the synthetic case, and is the target result of the case retrieval;

[0022] Then the true example can be obtained.

[0023] (4)

[0024] False positives

[0025] (5)

[0026] False negatives

[0027] (6)

[0028] Accuracy

[0029] (7)

[0030] Recall rate

[0031] (8)

[0032] value

[0033] (9)

[0034] , and It can be used to evaluate the quality of fine-grained case retrieval.

[0035] S5: Construct an optimized fine-grained case retrieval model

[0036] To improve the accuracy and reliability of fine-grained case retrieval, this invention will... and As an optimization target:

[0037] optimization The module is a declarative programming optimization module that is configured to receive task signatures and training samples, and through an automated compilation process, generate and optimize instruction sequences to guide large language models to complete specific tasks.

[0038] optimization The module is a hyperparameter automatic search module, which is configured to: dynamically propose and evaluate new parameter combinations based on the historical results of multiple rounds of trials in a preset parameter space, using an optimized sampling algorithm to maximize the target index;

[0039] S6: Inductive generation of referee rules

[0040] Use prompt words and large models Build prompts Based on the fine-grained search results, the judgment rules for query cases are generated, specifically:

[0041] The output of Algorithm 1 The most relevant candidate case number and key plot number for each key plot of the query case are given. Given the index, we get variables query , candidate and reason The specific content, input these specific contents You can then proceed under the guidance of the prompt words. The rules of refereeing were discovered.

[0042] Furthermore, in S1, the text embedding vector is obtained by inputting the key plot points represented by the text into a pre-trained text vector large model, as shown in equation (1):

[0043] = , R 1×dim (1)

[0044] In the formula, Key plot points represented in the text; For large text vector models, It is the vector dimension.

[0045] Furthermore, in S2, key plot points of the query case are extracted using prompt words. As shown in the table below:

[0046] .

[0047] Furthermore, in step S3, key plot points of candidate cases are extracted using prompt words. As shown in the table below:

[0048] ; The prompt words for the synthesized case As shown in the table below:

[0049] ; The prompts for quality checks of the synthesized cases mentioned above. As shown in the table below:

[0050] .

[0051] Furthermore, in S5, the optimization process uses an average... The value is used as an optimization metric, and the process involves two stages. The first stage uses a prompt word optimization module to find the optimal value. The second stage involves finding the optimal... Based on this, a hyperparameter optimization module is used to find the optimal... to further improve value;

[0052] Synthetic Case Set It is divided into three parts: training set, ... and verification set test set ;

[0053] In the optimization calculation process, using optimization Then use optimization , It is not used in the optimization process, but is used to calculate the optimized model. , and To evaluate the generalization ability of the model.

[0054] Furthermore, in step S6, prompts for the referee's rules are generated. As shown in the table below:

[0055] .

[0056] Based on the same inventive concept, the present invention also provides a computer electronic device corresponding to the above-described fine-grained case retrieval method, the device comprising the following components:

[0057] Memory: Used to store computer programs and data. The computer programs include instruction modules for implementing the fine-grained case retrieval method of the present invention, and interface modules for interacting with external large model services. The memory may include cache and non-volatile storage units to support efficient data access and persistence.

[0058] Processor: Coupled to the memory for executing the computer program, thereby being configured to perform the following functional operation flow:

[0059] (1) Call the large model API via the network;

[0060] (2) Extract key details from the case;

[0061] (3) Generate synthetic cases;

[0062] (4) Generate text embedding vectors and calculate similarity;

[0063] (5) Fine-grained case retrieval;

[0064] (6) Evaluate and optimize prompt words and similarity thresholds;

[0065] (7) Generate referee rules.

[0066] The above functions are executed and invoked under the control of the processor, according to the functional operation flow.

[0067] Furthermore, the processor may be a multi-core processor with parallel computing capabilities to accelerate data processing tasks.

[0068] Communication interface: Used to establish connections with external networks and large model services, supporting protocols such as HTTP / HTTPS to ensure the security and reliability of data transmission.

[0069] This invention also provides the application of the fine-grained case retrieval method based on a large language model in construction project contract disputes.

[0070] The beneficial technical effects of this invention are as follows:

[0071] 1) By using LLM, the automated parsing and structured representation of legal texts are realized, alleviating the limitations of traditional retrieval methods; by using the similarity calculation of text embedding vectors, fine-grained case matching is realized, improving the accuracy of retrieval; dual LLM is used to generate synthetic cases to evaluate and optimize the retrieval model; based on the fine-grained case retrieval results, the judgment rules are summarized and generated with the help of LLM, providing a reference for case handlers and helping to promote the development of legal artificial intelligence.

[0072] 2) This invention employs a generative large language model to extract key plot points from cases, and then uses semantic similarity calculation to achieve fine-grained case retrieval. To verify the effectiveness of this method, LLM is used to generate synthetic cases for calculating retrieval evaluation indicators and optimizing the retrieval model. This invention uses a prompt word optimization module to optimize the prompt words for extracting key plot points, and a hyperparameter optimization module to optimize the similarity threshold. Calculation results show that the optimized retrieval model significantly improves the evaluation indicators and has high reliability. Attached Figure Description

[0073] Figure 1 This is a flowchart illustrating the fine-grained case retrieval and judgment rule generation process of this invention.

[0074] Figure 2 This is the fine-grained case retrieval model of the present invention.

[0075] Figure 3 This is a box plot showing the evaluation metrics of the model after optimization using 5-fold cross-validation in this invention. Detailed Implementation

[0076] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0077] Example 1

[0078] In a specific implementation, this invention provides a fine-grained case retrieval method based on a large language model, which may include the following steps:

[0079] S1: Constructing a fine-grained case retrieval model

[0080] The task of fine-grained case retrieval is to find similarities between a candidate case (adjudicated case) and the query case (pending case) from a number of candidate cases (adjudicated cases), and then to summarize the rules of judgment for similar circumstances based on the judgment results of the candidate cases, which can be used as a reference for the judgment of the query case. The above process can be described as follows:

[0081] assumed To search for cases, , ,…, for One candidate case, A collection of plots = , To and Similar plot collection = .exist Contains plot , Contains plot And so on, fine-grained case retrieval retrieves tasks based on... turn up And obtain the rules of the referee;

[0082] The fine-grained case retrieval model constructed is as follows: (1) Using LLM to search based on prompt words Extracting the details that significantly influenced the judgment. = (2) Using LLM to analyze prompts Extracting the details that significantly influenced the judgment. = (3) Perform text embedding on each key plot point to obtain a text embedding vector. The corresponding set of text embedding vectors = , The corresponding set of embedding vectors = (4) and The similarity of each element in the query is calculated, and a fine-grained case retrieval is performed based on the similarity to obtain the plot of the candidate case that is most similar to the query case.

[0083] The text embedding vector is obtained by inputting the key plot points of the text representation into a pre-trained text vector large model, as shown in equation (1):

[0084] = , R 1×dim (1)

[0085] In the formula, Key plot points represented in the text; For large text vector models, It is the vector dimension;

[0086] S2: Constructing a fine-grained case retrieval algorithm

[0087] Key words were extracted from the key plot points of the case. and Constructing prompts Query Cases Case input Extract Key plot points will be considered in the candidate cases. Case input Extract This key plot point will The key plot points are input line by line into the large text vector model to obtain a sequence of text embedding vectors. =( , ,…, ),Will The key plot points are input line by line into the large text vector model to obtain a sequence of text embedding vectors. =( , ,…, ), obtain the similarity matrix = ,in, =cosine similarity( Cosine similarity is the cosine similarity of vectors. From the candidate cases, a similarity matrix sequence can be obtained. , ,…, ;

[0088] Constructing a fine-grained similarity retrieval algorithm: Algorithm 1: First, concatenate the similarity matrices of the query case and multiple candidate cases column-wise into a single matrix. ,if The maximum value of each row is greater than the similarity threshold. (Similarity threshold) This is a hyperparameter used to filter key plot points in candidate cases. Its value affects the quantity and quality of search results: if the threshold is set too low, plot points with insufficient similarity may be incorrectly retained, introducing noise and returning too many results; if the threshold is too high, it may filter out potentially similar plot points with some reference value, resulting in insufficient recall. The value of has a significant impact on the accuracy of the search results. Therefore, the column number corresponding to this maximum value is mapped to a tuple consisting of candidate case numbers and key plot numbers. This retrieves the candidate case numbers and key plot numbers most similar to the key plot of the query case, thus achieving key plot retrieval. The output includes two dictionaries: The key is the key plot number of the query case, and the value is a tuple consisting of the candidate case number and the key plot number, used to generate the judgment pattern. The key is the candidate case number, and the value is a list of key plot numbers used to calculate the retrieval performance metrics.

[0089] To further clarify the core logic of similarity matrix concatenation and matching association preprocessing in the fine-grained case retrieval algorithm, and to provide a unified data foundation for subsequent threshold screening, this embodiment provides the following core pseudocode snippet. This pseudocode is merely an exemplary implementation, intended to clearly demonstrate the core algorithm flow. Those skilled in the art can implement it equivalently using any programming language such as Python or Java, or make adaptive adjustments to the loop method, data storage structure, etc.

[0090]

[0091] The design is relatively simple, which makes it easy to make fine adjustments and performance improvements in later stages.

[0092] Table 1. Key words extracted from the query case.

[0093] ;

[0094] S3: Constructing the Synthetic Case Algorithm

[0095] Currently, there is limited research on fine-grained case retrieval, and publicly available fine-grained case retrieval datasets are lacking. This invention employs dual LLM to generate synthetic cases for the evaluation and optimization of fine-grained case retrieval, constructing a synthetic case algorithm Algorithm 2: from the candidate case set... = Several case texts were randomly selected from the data, and then... For each selected case text, key plot points are extracted, and... Record the selected case number and key plot number as the retrieval target value, and then input all the extracted key plots. Combine a query case and use Check the quality of the synthesized cases. If they meet the requirements, add them to the synthesized case set. If they do not meet the requirements, regenerate them until the maximum number of synthesized cases can be reached.

[0096] To further clarify the core logic of synthetic case generation in the fine-grained case retrieval algorithm, this embodiment provides the following core pseudocode snippet. This pseudocode is merely an exemplary implementation, intended to clearly demonstrate the core process of the algorithm. Those skilled in the art can implement it equivalently using any programming language such as Python or Java, or make adaptive adjustments to the loop method, data storage structure, etc., based on this logic.

[0097]

[0098] Furthermore, the aforementioned The design is as follows:

[0099] Table 2. Key words for extracting key plot points from candidate cases

[0100]

[0101] Furthermore, the aforementioned The design is as follows:

[0102] Table 3. Cue words for synthesized cases

[0103]

[0104] Furthermore, The design is as follows:

[0105] Table 4. Critical words for quality checks of synthetic cases

[0106]

[0107] S4: Evaluation Indicators

[0108] Algorithm 2 can be used to obtain a list of synthetic query cases. , The Middle element = : , : , To provide the true results of fine-grained case retrieval, enter Several key plot points are obtained; input the key plot points. Obtain vector sequence This allows us to obtain the similarity matrix with each candidate case. ( = Then, Algorithm 1 can be used to obtain fine-grained retrieval results for the query cases. ,make = According to the following formula:

[0109] = (2)

[0110] It is a set of tuples, each consisting of candidate case numbers and key plot numbers obtained from fine-grained case retrieval, which is the actual result of case retrieval.

[0111] = (3)

[0112] It is a set of tuples, each consisting of a candidate case number and a key plot number randomly selected when generating the synthetic case, and is the target result of the case retrieval.

[0113] Then the true example can be obtained.

[0114] (4)

[0115] False positives

[0116] (5)

[0117] False negatives

[0118] (6)

[0119] Accuracy

[0120] (7)

[0121] Recall rate

[0122] (8)

[0123] value

[0124] (9)

[0125] , and It can be used to evaluate the quality of fine-grained case retrieval.

[0126] S5: Construct an optimized fine-grained case retrieval model

[0127] To improve the accuracy and reliability of fine-grained case retrieval, this invention will... and As an optimization target.

[0128] optimization The module is a declarative programming optimization module configured to receive task signatures and training samples, and through an automated compilation process, generate and optimize instruction sequences to guide large language models to complete specific tasks.

[0129] optimization The module is a hyperparameter automatic search module, configured to: dynamically propose and evaluate new parameter combinations in a preset parameter space based on historical results from multiple rounds of trials, using an optimized sampling algorithm to maximize the target index.

[0130] The optimization process uses average The value is used as an optimization metric, and the process involves two stages. The first stage uses a prompt word optimization module to find the optimal value. The second stage involves finding the optimal... Based on this, a hyperparameter optimization module is used to find the optimal... to further improve value.

[0131] Synthetic Case Set It is divided into three parts: training set, ... and verification set test set .

[0132] In the optimization calculation process, using optimization Then use optimization . It is not used in the optimization process, but is used to calculate the optimized model. , and To evaluate the generalization ability of the model.

[0133] S6: Inductive generation of referee rules

[0134] Use prompt words and large models Build prompts Based on the fine-grained search results, the rules of judgment for query cases are generated.

[0135] Prompt words for generating referee patterns As shown in Table 5.

[0136] Table 5. Prompt words for generating referee patterns

[0137]

[0138] The output of Algorithm 1 The most relevant candidate case number and key plot number for each key plot of the query case are given. Given the index, we get variables query , candidate and reason The specific content, input these specific contents You can then proceed under the guidance of the prompt words. The rules of refereeing were discovered.

[0139] The fine-grained case retrieval method based on a large language model of this invention is illustrated by the following fine-grained case retrieval example in practical implementation:

[0140] 1. Basic Data

[0141] The actual case search comes from China Judgments Online, case number (2019) Gan Min Chu 57, and the cause of action is a construction project contract dispute. The candidate case set consists of 37 cases with the cause of action of construction project contract disputes from the People's Court Case Database. The number of candidate cases randomly selected when generating synthetic cases. The number of key plot points randomly selected for each case is 3. It is 2. and Both use the DeepSeek-chat large model. Adopting the Tongyi Thousand Questions-plus large model, The dataset uses a generalized text vector model, v3, with a vector dimension of 1024. 1000 synthetic test cases are generated using Algorithm 2. Similarity threshold The initial value is 0.9.

[0142] 2. Model Optimization

[0143] 2.1 Dataset Partitioning and Cross-Validation

[0144] To better evaluate the model's stability and generalization ability, 5-fold cross-validation was used. The dataset was partitioned as follows:

[0145] First, synthesize the case set. It is divided into two parts, with 200 cases constituting the test set. The other 800 entries constitute the training and validation set. .

[0146] use Perform 5-fold cross-validation, that is, The data was randomly divided into 5 mutually exclusive subsets of approximately equal size. In each round of the experiment, one of these subsets was selected as the validation set. Other data are used as the training set. This process is repeated 5 times to ensure that each subset is used as the validation set exactly once.

[0147] In each round of optimization calculations, using optimization Then use optimization . Not participating in optimization, will be used after optimization is complete. Calculate the optimized model , and This is used to evaluate the model's capabilities.

[0148] 2.2 Optimization Process

[0149] The optimization process uses average The value is used as an optimization metric, and the process involves two stages. The first stage involves using the prompt word optimization module to find the optimal value. In the second stage, the hyperparameter optimization module is used to find the optimal... Based on this, find the optimal to further improve value.

[0150] The prompt word optimization module can utilize existing declarative programming frameworks such as DSPy. This module defines task signatures and model pipelines using DSPy and leverages its compiler to automatically optimize prompt words and model weights based on a small number of examples.

[0151] The hyperparameter optimization module can employ existing hyperparameter optimization frameworks, such as Optuna. This module defines the search space using Optuna and leverages its tree-structured Parzen estimator sampling algorithm to automatically conduct multiple rounds of trials to find the optimal parameter combination.

[0152] 2.3 Model optimization performance on the validation set

[0153] Input the validation set data into the model before and after each round of optimization to obtain the average evaluation index, as shown in the table below:

[0154] Table 6. Average values ​​of experimental evaluation indicators obtained from the validation set.

[0155]

[0156] It can be seen that the model significantly improved the F1 score and Recall score after optimization, while the Precision remained at a high level, and the overall model performance was better.

[0157] 2.4 Calculation results of the model on the test set

[0158] To evaluate the generalization ability of the proposed method, the test set is input into the model before and after optimization to obtain the average value of the evaluation index.

[0159] Meanwhile, to evaluate the effectiveness of the two-stage optimization method adopted in this invention, an ablation experiment was conducted. The ablation experiment included two types of models: one was a model that only optimized the prompt words, and the other was a model that only optimized the similarity threshold. The average values ​​of the obtained evaluation indicators are shown in the table below.

[0160] Table 7. Average values ​​of experimental evaluation metrics for the test set.

[0161]

[0162] Comparing the calculation results of the validation set and the test set, it can be seen that both the validation set and the test set can achieve good results on the optimized model, and the model has good generalization ability.

[0163] Based on the experimental results on the test set, in 5-fold cross-validation, the unoptimized original model had the highest precision but the lowest F1 score. The optimized model had the highest F1 score in all aspects, showing a significant improvement compared to the unoptimized model, while maintaining a high level of precision and a noticeable increase in recall.

[0164] Optimizing only the similarity threshold or only the prompt words improves both the F1 score and recall, but both are still lower than the optimized model. This indicates that optimizing only the similarity threshold or only the prompt words can improve model performance, but optimizing both simultaneously yields better results. This also proves the effectiveness and necessity of the two-stage optimization method used in this invention.

[0165] Will The optimized model used for 5-fold cross-validation, box plots of the mean of the evaluation metrics and the coefficient of variation (COP). )See Figure 3 .from Figure 3 As can be seen, the evaluation index of the model after cross-validation optimization has less dispersion, and the model has higher stability and reliability.

[0166] 2.5 Supplementing the model's computational results on the test set

[0167] To further test the model's generalization ability, another 35 adjudicated cases with the cause of action of construction contract disputes were randomly selected from the China Judgments Online website to form a candidate case set. The supplementary test set consists of 200 synthesis cases generated by Algorithm 2. ,Will The optimized model obtained by the aforementioned 5-fold cross-validation is used to calculate the following metrics:

[0168] Table 8 Average value of experimental evaluation index

[0169]

[0170] It can be seen that, in The optimized model still performs well, with no significant decrease in any of the indicators, demonstrating that the optimized model has good generalization ability and can still achieve good results in new candidate cases and query cases, indicating that the model proposed in this invention has significant practical value.

[0171] 2.6 The effect of the comparison method

[0172] To further evaluate the effectiveness of the method proposed in this invention, the following method was used as a comparison method.

[0173] Depend on and Build prompts It simulates how humans retrieve relevant key plot points from candidate cases based on a query case. Using the DeepSeek-chat large model, the design as follows:

[0174] Table 9. Suggested keywords for key plot searches

[0175]

[0176] use deal with For each case, obtain the key plot set of candidate cases, and then... Each query case uses The key plot points of the candidate cases are retrieved from the set of key plot points to find key plot points related to the query case. The calculated index values ​​are shown in the table below:

[0177] Table 10 Evaluation index values ​​of the comparison methods

[0178]

[0179] It is evident that the evaluation index value of the comparative method is relatively low, while the method proposed in this invention has significant advantages.

[0180] 3. The generation of refereeing patterns

[0181] Real query cases are input into the optimized model, and Algorithm1 performs fine-grained case retrieval to obtain candidate cases corresponding to the key elements of the real query cases and their indexes. The key details and evidence from real-life search cases. Key plot inputs for candidate cases in the index The referee rules are generated by LLM under the guidance of prompt words. The DeepSeek-chat large model is used. The generated referee rules are as follows:

[0182] Rule 1: In the case of the plaintiff requesting confirmation of priority right to payment for construction work, based on the 10th candidate case, the following rule of judgment is drawn: the contractor has the right to priority payment from the proceeds of the discounted price or auction of the project if the employer fails to pay the project price as agreed. The scope of priority payment is limited to the actual project cost and does not include interest or breach of contract damages.

[0183] The case for reference is: Shanghai XX Construction Company v XX Medical Company, a construction contract dispute.

[0184] Analysis of the applicability of the rule: In this case, the plaintiff's request to confirm a priority right to payment for the construction project is highly consistent with the "confirmation of a priority right to payment for the discounted or auctioned proceeds of a construction project" in the candidate cases in terms of legal nature, applicable conditions, and scope of rights. Both are based on the facts that the contractor has completed its construction obligations, the employer has failed to pay the project funds on time, and the contractor has asserted its rights within the statutory period. The court also explicitly supported the establishment of the contractor's priority right and limited its scope. Therefore, this rule of judgment is applicable to the plaintiff's claim for priority payment in this case.

[0185] Other patterns are omitted.

[0186] 4 Computer Electronic Devices and Implementation Examples

[0187] 4.1 Computer electronic equipment

[0188] Based on the same inventive concept, the present invention also provides a computer electronic device corresponding to the case retrieval method provided in the above embodiments. This device includes the following components:

[0189] 4.1.1 Memory

[0190] This device is used to store computer programs and data. The programs include instruction modules for implementing the methods of this invention, and interface modules for interacting with external large-scale model services. The memory may include cache and non-volatile storage units to support efficient data access and persistence.

[0191] 4.1.2 Processor

[0192] Coupled to the memory for executing the program, thereby being configured to perform the following functional operations:

[0193] (1) Call the large model API via the network;

[0194] (2) Extract key details from the case;

[0195] (3) Generate synthetic cases;

[0196] (4) Generate text embedding vectors and calculate similarity;

[0197] (5) Fine-grained case retrieval;

[0198] (6) Evaluate and optimize prompt words and similarity thresholds;

[0199] (7) Generate referee rules.

[0200] The above functions are executed and invoked under the control of the processor, according to the process described in the method embodiment.

[0201] Specific implementation: The processor can be a multi-core processor with parallel computing capabilities to accelerate data processing tasks.

[0202] 4.1.3 Communication Interface

[0203] Used to establish connections with external networks and large-scale model services, supporting protocols such as HTTP / HTTPS to ensure the security and reliability of data transmission.

[0204] 4.2 Implementation Example

[0205] In actual deployment, the above-mentioned device can run on a local server or a cloud platform. After the user initiates a request through the client, the device sequentially performs the following steps: calling the large model API, extracting key plot points of the case, generating synthetic cases, generating text embedding vectors and calculating similarity, fine-grained case retrieval, optimizing prompt words and similarity thresholds, and generating judgment patterns (see the method implementation example for details), and finally returns a structured case analysis report.

[0206] In this device, the memory stores initial case data, intermediately generated key plots, embedding vectors, synthesized cases, and the final analysis report. The processor, through program execution, controls the reading of data from the memory, processes the data through various functional modules, writes the results back to the memory, and communicates with external services and returns results to the client via the communication interface. All components work collaboratively to complete the entire process from case input to case analysis report output.

[0207] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A fine-grained case retrieval method based on a large language model, characterized in that, Includes the following steps: S1: Constructing a fine-grained case retrieval model The task of fine-grained case retrieval is to find similarities between the candidate cases and the pending cases from a pool of adjudicated candidate cases, and then to summarize the adjudication patterns of similar cases based on the adjudication results of the candidate cases, which serve as a reference for the adjudication of the pending cases. The above process is described as follows: assumed To search for cases, , ,…, for One candidate case, A collection of plots = , To and Similar plot collection = .exist Contains plot , Contains plot And so on, fine-grained case retrieval retrieves tasks based on... turn up And obtain the rules of the referee; The fine-grained case retrieval model constructed is as follows: (1) Using LLM to search based on prompt words Extracting the details that significantly influenced the judgment. = (2) Using LLM to analyze prompts Extracting the details that significantly influenced the judgment. = (3) Perform text embedding on each key plot point to obtain a text embedding vector. The corresponding set of text embedding vectors = , The corresponding set of embedding vectors = (4) and The similarity of each element in the query is calculated, and a fine-grained case retrieval is performed based on the similarity to obtain the plot of the candidate case that is most similar to the query case. S2: Constructing a fine-grained case retrieval algorithm Key words were extracted from the key plot points of the case study. and Constructing prompts Query Cases Case input Extract Key plot points will be considered in the candidate cases. Case input Extract This key plot point will The key plot points are input line by line into the large text vector model to obtain a sequence of text embedding vectors. =( , ,…, ),Will The key plot points are input line by line into the large text vector model to obtain a sequence of text embedding vectors. =( , ,…, ), obtain the similarity matrix = ,in, =cosine similarity( Cosine similarity is the cosine similarity of vectors. From the candidate cases, a similarity matrix sequence can be obtained. , ,…, ; Constructing a fine-grained similarity retrieval algorithm: Algorithm 1: First, concatenate the similarity matrices of the query case and multiple candidate cases column-wise into a single matrix. ,if The maximum value of each row is greater than the similarity threshold. Then, the column number corresponding to this maximum value is mapped to a tuple consisting of candidate case numbers and key plot numbers, thereby retrieving the candidate case number and key plot number most similar to the key plot of the query case, realizing key plot retrieval. The output includes two dictionaries: The key is the key plot number of the query case, and the value is a tuple consisting of the candidate case number and the key plot number, used to generate the judgment pattern. The key is the candidate case number, and the value is a list of key plot numbers used to calculate the retrieval performance metrics. S3: Constructing the Synthetic Case Algorithm A dual LLM approach is employed to generate synthetic cases for evaluation and optimization of fine-grained case retrieval. This leads to the construction of Synthetic Case Algorithm 2: which generates synthetic cases from a candidate case set. = Several case texts were randomly selected from the data, and... For each selected case text, key plot points are extracted, and... Record the selected case number and key plot number as the retrieval target value, and then input all the extracted key plots. Combine a query case and use Check the quality of the synthesized cases. If they meet the requirements, add them to the synthesized case set. If they do not meet the requirements, regenerate them until the maximum number of synthesized cases can be reached. S4: Evaluation Indicators The synthesis case set can be obtained from Algorithm 2. , The Middle element = : , : , To provide the true results of fine-grained case retrieval, enter Several key plot points are obtained; input the key plot points. Obtain vector sequence This allows us to obtain the similarity matrix with each candidate case. ( = Then, Algorithm 1 can be used to obtain fine-grained retrieval results for the query cases. ,make = According to the following formula: = （2） It is a set of tuples, each tuple consisting of candidate case numbers and key plot numbers obtained from fine-grained case retrieval, which is the actual result of case retrieval; = （3） It is a set of tuples, each consisting of a candidate case number and a key plot number randomly selected when generating the synthetic case, and is the target result of the case retrieval; Then the true example can be obtained. （4） False positives （5） False negatives （6） Accuracy （7） Recall rate （8） value （9） , and It can be used to evaluate the quality of fine-grained case retrieval; S5: Construct an optimized fine-grained case retrieval model To improve the accuracy and reliability of fine-grained case retrieval, and As an optimization target: optimization The module is a declarative programming optimization module that is configured to receive task signatures and training samples, and through an automated compilation process, generate and optimize instruction sequences to guide large language models to complete specific tasks. optimization The module is a hyperparameter automatic search module, which is configured to: dynamically propose and evaluate new parameter combinations based on the historical results of multiple rounds of trials in a preset parameter space, using an optimized sampling algorithm to maximize the target index; S6: Inductive generation of referee rules Use prompt words and large models Build prompts Based on the fine-grained search results, the judgment rules for query cases are generated, specifically: The output of Algorithm 1 The most relevant candidate case number and key plot number for each key plot of the query case are given. Given the index, we get variables query , candidate and reason The specific content, input these specific contents You can then proceed under the guidance of the prompt words. The rules of refereeing were discovered.

2. The fine-grained case retrieval method based on a large language model according to claim 1, characterized in that, In S1, the text embedding vector is obtained by inputting the key plot points of the text representation into a pre-trained text vector large model, as shown in Equation (1): = , R 1×dim （1） In the formula, Key plot points represented in the text; For large text vector models, It is the vector dimension.

3. The fine-grained case retrieval method based on a large language model according to claim 1, characterized in that, In step S2, key clue words for the query case are extracted. As shown in the table below:

4. The fine-grained case retrieval method based on a large language model according to claim 1, characterized in that, In step S3, key plot points of candidate cases are extracted using prompt words. As shown in the table below: The prompt words for the synthesized case As shown in the table below: The prompts for quality checks of the synthesized cases mentioned above. As shown in the table below:

5. The fine-grained case retrieval method based on a large language model according to claim 1, characterized in that, In S5, the optimization process uses average The value is used as an optimization metric, and the process involves two stages. The first stage uses a prompt word optimization module to find the optimal value. The second stage involves finding the optimal... Based on this, a hyperparameter optimization module is used to find the optimal... to further improve value; Synthetic Case Set It is divided into three parts: training set, ... and verification set test set ; In the optimization calculation process, using optimization Then use optimization , It is not used in the optimization process, but is used to calculate the optimized model. , and To evaluate the generalization ability of the model.

6. The fine-grained case retrieval method based on a large language model according to claim 1, characterized in that, In step S6, prompts for the rules of the referee are generated. As shown in the table below:

7. A computer electronic device corresponding to the fine-grained case retrieval method based on a large language model as described in any one of claims 1-6, characterized in that, The device includes the following components: Memory: Used to store computer programs and data. The computer programs include instruction modules that implement the fine-grained case retrieval method of the present invention, and interface modules that interact with external large model services. The memory may include cache and non-volatile storage units to support efficient data access and persistence. Processor: Coupled to the memory for executing the computer program; Communication interface: Used to establish connections with external networks and large model services, supporting protocols such as HTTP / HTTPS to ensure the security and reliability of data transmission.

8. A computer electronic device corresponding to the fine-grained case retrieval method based on a large language model according to claim 7, characterized in that, The processor is configured to perform the following functional operation procedures: (1) Call the large model API via the network; (2) Extract key details from the case; (3) Generate synthetic cases; (4) Generate text embedding vectors and calculate similarity; (5) Fine-grained case retrieval; (6) Evaluate and optimize prompt words and similarity thresholds; (7) Generate referee rules.

9. A computer electronic device corresponding to the fine-grained case retrieval method based on a large language model according to claim 7, characterized in that, The processor is a multi-core processor with parallel computing capabilities to accelerate data processing tasks.

10. The application of the fine-grained case retrieval method based on a large language model as described in any one of claims 1-6 in construction project contract disputes.

Images