Retrieval method and device based on man-machine cooperation type case and storage medium

Through the combination of K-means algorithm and ProtoEM algorithm, a joint decision-making model is established, and the implicit knowledge of legal practitioners is integrated with machine decision-making, solving the problem of difficult to express implicit knowledge and evaluation of decision-making uncertainty, achieving more accurate and efficient The search for similar cases has improved the efficiency of legal practice and the quality of decision-making.

CN118916489APending Publication Date: 2024-11-08SICHUAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411001915.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

It is difficult for the prior art to effectively integrate the implicit knowledge of legal practitioners into human-computer collaboration, especially in case searches. It is difficult to capture their professional judgments and experience, and the difficulty of estimating the uncertainty of human decision-making will affect the search effect.

Method used

Using the K-means algorithm to cluster sentences in historical legal case documents, joint probabilistic decisions are obtained through the ProtoEM algorithm, confusion matrix is ​​established to quantify the uncertainty of legal practitioners' decisions, and discrete decisions are converted into probabilistic decisions through joint decision models. Realize collaborative retrieval between legal practitioners and machines.

Benefits of technology

It improves the accuracy and efficiency of similar case searches, improves the work efficiency and decision-making quality in legal practice, and is significantly better than the search performance of relying solely on machines or legal practitioners.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916489A_ABST
    Figure CN118916489A_ABST
Patent Text Reader

Abstract

The invention discloses a retrieval method and device based on man-machine cooperation type cases and a storage medium. The retrieval method comprises the following steps that S1, sentences in historical legal case documents are clustered; s2, a joint probability decision of a legal practitioner and a machine is obtained through a ProtoEM algorithm, and the confusion matrix is updated by using the joint probability decision; s3, matching the discrete decision of the legal practitioner into the confusion matrix by combining the decision of the legal practitioner for the key sentences and the decision of the machine for the key sentences in a probabilistic manner, and establishing a joint decision model; and S4, retrieving a matched legal case by using the joint decision model. The retrieval result is more accurate and comprehensive, and the working efficiency and decision-making quality in law practice are effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence application technology, and in particular to a retrieval method, device and storage medium based on human-computer collaborative similar cases. Background Art

[0002] In the legal field, legal practitioners usually need to refer to a large number of legal cases similar to the pending cases (i.e., similar cases) to help make a more objective judgment on the cases. Similar cases are defined as cases that have been adjudicated and are similar to the pending cases in terms of basic legal facts, controversial issues, and legal applicability. When lawyers defend, similar cases can provide references for defense and are of great significance to the rationality of the defense. Unlike general information retrieval, the retrieval of legal similar cases relies heavily on the implicit knowledge of legal practitioners, such as professional knowledge and work experience. Legal practitioners are willing to invest more energy to obtain accurate and comprehensive results to ensure the fairness and impartiality of judicial decisions. However, although legal practitioners have rich legal implicit knowledge, they lack the ability to retrieve relevant cases from large legal case databases, especially under limited time constraints in judicial decision scenarios. Recently, the use of artificial intelligence models (referred to as "machines" in this invention) to support legal case retrieval has become increasingly common. These machines are favored because they can accurately identify small differences between numerical values ​​and consider complex feature sets when determining relevance, making them the first choice for automatically analyzing and retrieving large corpora. Although machines have many advantages, new research has found that machines lack sufficient legal knowledge, and their retrieval performance in legal scenarios will be affected. This becomes particularly obvious considering that the implicit knowledge of legal practitioners is usually difficult to express in machine encoding. It is challenging to accurately express and encode the implicit knowledge of legal practitioners, such as professional knowledge and work experience, into a form that can be understood by machines. Therefore, relying entirely on machines and ignoring the implicit knowledge of legal practitioners will hardly provide reliable and correct results.

[0003] In the existing technology, there are two main challenges in human-computer collaborative case retrieval: one is how to establish a human-computer collaborative paradigm, and the other is how to estimate the uncertainty of human decision-making.

[0004] How to establish a human-machine collaboration paradigm and utilize the tacit knowledge of legal practitioners: Tacit knowledge usually comes from legal practitioners' long-term practical experience, professional qualities and deep understanding of the legal field. Nevertheless, how to effectively integrate this knowledge into machines to improve the efficiency and accuracy of legal case retrieval remains a major challenge. In the process of achieving human-machine collaboration, traditional quantitative measurement methods often fail to capture the intuition, experience and professional judgment of legal practitioners, especially when dealing with complex cases. Therefore, it is particularly important to explore a flexible and intelligent mechanism that allows legal practitioners to express their professional judgment and knowledge in a machine-understandable way so as to facilitate effective interaction and collaboration with machines. In addition, since tacit knowledge is usually unstructured and individualized and difficult to describe clearly in traditional language, the machine interpretation of this knowledge requires more research and exploration. In general, to achieve effective cooperation between legal practitioners and machines, it is necessary to find a new cooperation model that can not only utilize the professional knowledge and experience of legal practitioners, but also take into account the computing power and intelligence of machines. This model is expected to bring fundamental changes to information retrieval and case handling in the legal field, significantly improving efficiency, accuracy and intelligence.

[0005] How to estimate the uncertainty of human decision-making, combine discrete human decisions with continuous model outputs, and improve the effect of similar case retrieval: In similar case retrieval tasks, legal practitioners face complex and changeable decision-making scenarios, which are full of uncertainty. This uncertainty may come from the diversity and complexity of the case itself, or from the legal practitioners' interpretation and understanding of the case details and relevant legal provisions. In this context, it is particularly difficult to achieve effective human-machine collaboration. First, the decisions of legal practitioners are usually based on their rich professional knowledge and experience, involving many subjective factors, usually in the form of category labels, ratings or binary choices (such as: relevant or irrelevant). These decisions are discrete, belong to a limited range of choices, and lack continuity. Therefore, when collaborating with machines, how to convert these subjective and discrete decisions into continuous numerical results, such as similarity scores, probabilities or predicted values, becomes a key issue. Second, due to the complexity and diversity of legal cases, it is often difficult for machines to accurately simulate the decision-making process of legal practitioners, which further affects the effect of human-machine collaboration. Currently, methods for estimating the uncertainty of human decision-making often rely on the assistance of true labels, but in the legal field, key sentences often lack such labels, making these methods difficult to apply. Therefore, the present invention proposes a new method to estimate the uncertainty of human decision-making that does not rely on the existence of true labels. This requires a deep understanding and modeling of the human decision-making process, as well as a detailed analysis and interpretation of human behavior and language. Summary of the invention

[0006] The purpose of the present invention is to provide a retrieval method, device and storage medium for similar cases based on human-machine collaboration, so as to achieve more accurate retrieval of similar cases by integrating the implicit knowledge of legal practitioners into the machine.

[0007] To achieve the above object, the present invention provides the following technical solution: a retrieval method based on human-computer collaborative similar cases, comprising the following steps:

[0008] S1. Cluster the sentences in the historical legal case documents using the K-means algorithm;

[0009] S2. The joint probabilistic decision of legal practitioners and machines is obtained through the ProtoEM algorithm, and a confusion matrix is ​​created for each prototype using the joint probabilistic decision. The confusion matrix is ​​continuously updated by maximizing the objective function, which quantifies the uncertainty of legal practitioners in similar decision-making behaviors without knowing the true label.

[0010] S3. Combining the decision of legal practitioners on key sentences with the decision of the machine on key sentences in a probabilistic way, converting the discrete decision of legal practitioners into probabilistic decision through confusion matrix, and establishing a joint decision model;

[0011] S4. Use the selected key sentences to retrieve matching legal cases using the joint decision-making model.

[0012] In some embodiments, S1 includes the following steps:

[0013] First, the text in the historical legal case documents was segmented into individual sentences;

[0014] Then, the sentences are input into the legal-bert model to encode the features of each sentence;

[0015] Finally, the number of clusters is determined, and the K-means algorithm is used to cluster the encoded sentence features.

[0016] In some embodiments, in S2, the confusion matrix calculation expression of each prototype is:

[0017]

[0018] In the formula, 1[·] is an indicator function; H i For legal practitioners to make decisions on key sentences; i is the machine's decision on key sentences; i is the real decision; k is the kth sentence; φ t-1 The objective function value of the previous step, It is a joint decision made by legal practitioners and machines.

[0019] In some embodiments, in S3, the confusion matrix for:

[0020]

[0021] Where, d i is the i-th sentence; is the average sentence embedding over prototypes pk.

[0022] In some embodiments, in S3, the joint decision model is:

[0023]

[0024] In the formula, H i For legal practitioners to make decisions on key sentences; i is the machine's decision on key sentences; i is the real decision; k is the kth sentence; M ij M i The jth dimension of ; the sentence category of j real label; C is the category of the sentence; [φ i ] kq For each element, the decision q at this time and the decision H of the legal practitioner for the key sentence i In sentence d i The inconsistency rate on i ] kj For each element, pass the true decision y i and legal practitioners’ decisions on key sentences i In sentence d i The inconsistency rate on .

[0025] A device, characterized in that the device includes a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the retrieval method based on human-computer collaborative cases.

[0026] A computer storage medium, characterized in that the computer storage medium is used to store program data, and when the program data is executed by a processor, it is used to implement the retrieval method based on human-computer collaborative cases.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention uses a probability-based approach through the Co-Retrieval algorithm to fuse the decisions of legal practitioners and machines, and uses the ProtoEM algorithm to evaluate the uncertainty of legal practitioners' decisions in the absence of real labels; the retrieval results of the present invention are more accurate and comprehensive, effectively improving work efficiency and decision-making quality in legal practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the overall structure of the present invention. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0031] See also Figure 1 , a retrieval method based on human-computer collaborative similar cases, comprising the following steps:

[0032] S1. Cluster the sentences in the historical legal case documents using the K-means algorithm.

[0033] First, the text in the historical legal case document is segmented into individual sentences; the historical document is represented as Then, each sentence

[0034] Then, the semantic embedding of the sentence is obtained through the legal-bert algorithm, using The concatenation of is taken as input and feature encoding is performed on each sentence.

[0035] Finally, the K-means algorithm is used to cluster the encoded sentence features and cluster all embeddings into P prototypes. Each prototype captures similar decision-making behaviors of legal practitioners in subtle contexts.

[0036] S2. Obtain the joint probability decision of legal practitioners and machines through the ProtoEM algorithm, and use the joint probability decision to update the confusion matrix.

[0037] For each decision class p from the jth person j Sentences The goal of the ProtoEM algorithm is to estimate the corresponding decision uncertainty at the category level, and its parameters are given by the confusion matrix In the absence of true labels, this embodiment iteratively uses the available data (i.e., prototype p j ) to provide a maximum likelihood estimation solution that includes latent variables (i.e., the unknown truth in this embodiment).

[0038] In step E, this embodiment uses the confusion matrix To calculate the human-machine joint decision. In step M, this embodiment uses the human-machine joint decision as the truth to update the prototype p j The confusion matrix In the tth round, the objective function of this embodiment is expressed as follows:

[0039]

[0040] In the formula, The objective function of is the confusion matrix; The objective function of the previous iteration; y i For real decision; H i For legal practitioners to make decisions on key sentences; i For the machine to make decisions about key sentences; It is a joint probability decision.

[0041] Since legal practitioners and machines make independent decisions on key sentences and the confusion matrix It captures the category-level information, so the joint probability decision It can be decomposed into Substitute it into the objective function and omit it from not depending on the confusion matrix The constant term of , we get the new objective function:

[0042]

[0043] because, is a joint decision made by the legal practitioner and the machine, and it is a constant that can be calculated in step E. Therefore, the optimization problem can be easily solved in closed form.

[0044] Specifically, the confusion matrix element of the j-th prototype is Determined by the following equation.

[0045]

[0046] In the formula, 1[·] is an indicator function; H i For legal practitioners to make decisions on key sentences; i is the machine's decision on key sentences; i is the real decision; k is the kth sentence; It is a joint decision made by legal practitioners and machines.

[0047] This embodiment provides each prototype p j Create a confusion matrix It quantifies the uncertainty of legal practitioners in similar decision-making behaviors without knowing the true labels.

[0048] S3. Combining the decision of legal practitioners on key sentences with the decision of the machine on key sentences in a probabilistic way, adapting the discrete decision of legal practitioners to the confusion matrix, and establishing a joint decision model;

[0049] In this embodiment, according to different retrieval mechanisms, the keyness of each sentence can be classified into different categories, such as elements constituting a crime, controversial points, or non-critical sentences. Given an existing retrieval machine, consider C different categories. Then, both legal practitioners and machines need to decide sentence d i This process is called identifying key sentences in this embodiment. i The decision is denoted by H i , is a discrete variable drawn from a set of categories, and the decision of the machine is denoted by H i ∈RC is a C-dimensional probability vector.

[0050] In collaborative retrieval tasks, there are usually only two parties involved, namely legal practitioners and machines, and simple ensemble learning methods cannot outperform the best decisions of both parties. This is because simple weighted majority voting methods do not take into account the differences in decision quality and credibility of different parties, which may cause the final decision to be affected by low-quality decisions.

[0051] In this embodiment, human and machine jointly identify the key legal sentences d i The decision is expressed as p(y i |M i ,H i ), assuming that given y i In the case of i and machine decision M i are conditionally independent, then the joint decision p(y i |M i ,H i ) follows Bayes' rule.

[0052] p(y i |M i ,H i )∝p(H i |M i ,y i )p(y i |Mi )∝p(H i |y i )p(y i |M i )

[0053] In the formula, y i is the real decision, p(y i |M i ) is the predicted distribution of the retrieval machine; p(H i |y i ) is the probability of discrete human decisions, capturing the decision uncertainty of legal practitioners at the category level. This uncertainty can be represented as the confusion matrix φ i , then each element [φ i ] jk = p(H i =j|y i = k) can be evaluated by evaluating the true decision y i and legal practitioners’ decision making i In the key sentence d i In this case, the joint decision p(y i |M i ,H i ) can be reformulated in closed form, where the denominator term is used for renormalization:

[0054]

[0055] In the formula, H i For legal practitioners to make decisions on key sentences; i is the machine's decision on key sentences; i is the real decision; k is the kth sentence; M ij M i The jth dimension of ; the sentence category of j real label; C is the category of the sentence; [φ i ] kq For each element, the decision q at this time and the decision H of the legal practitioner for the key sentence i In sentence d i The inconsistency rate on i ] kj For each element, pass the true decision y i and legal practitioners’ decisions on key sentences i In sentence d i The inconsistency rate on .

[0056] This embodiment introduces implicit knowledge through legal practitioners' decisions on key sentences and combines them with machine decisions, thereby arriving at a joint decision on key sentences.

[0057] S4. Use the joint decision-making model to retrieve matching legal cases.

[0058] The present application also provides a computer storage medium, which is used to store program data. When the program data is executed by a processor, it is used to implement the retrieval method based on human-computer collaboration cases as described in the above embodiment.

[0059] The present application also provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to execute the human-computer collaborative case retrieval method as described in the embodiment of the present application. The computer program product can be a software installation package.

[0060] The retrieval method based on human-computer collaboration cases described in the above embodiments of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0061] In a specific embodiment, the effect of the present application is illustrated by the following comparative experiments;

[0062] Table 1. Comparison of retrieval performance evaluation with existing methods

[0063]

[0064]

[0065] As shown in Table 1, compared with all baseline models, this embodiment significantly improves the retrieval performance, and this improvement is very significant in all evaluation indicators. When the average evaluation is performed on all evaluation indicators and data sets, the effect of the framework of this embodiment is improved by 6.82% compared with the effect of the Bert-Pli model, by 11.21% compared with the effect of the IOT-Match model, and by 8.11% compared with the effect of the GIEOT-Match model. These results show that (1) for the recognition of key sentences in legal cases, the decision of this embodiment is independent of the machine-based key sentence recognition. (2) The results of this embodiment demonstrate the positive significance of involving legal practitioners in the legal case retrieval process. Although the implicit knowledge of legal practitioners is difficult to explicitly integrate with machine decisions, it is of vital importance to improving the accuracy of similar case retrieval results.

[0066] Table 2 Evaluation of human-robot collaboration and uncertainty estimation

[0067]

[0068]

[0069] The present invention can improve the retrieval performance, which is better than the retrieval performance based on legal practitioners alone and the retrieval performance of machines. As shown in Table 2, the similar case retrieval results of the present invention are always better than the results of Co-Retrieval w / o Legal Practitioner and Co-Retrieval w / o Machine. Specifically, when the data of different machine retrieval models, data sets, and evaluation indicators are averaged in this paper, the retrieval results of the present invention are 5.51% better than those of legal practitioners alone and 8.71% better than those of machine retrieval alone. This improvement in performance highlights the potential for cooperation between legal practitioners and machines in identifying and selecting key sentences, and proves the complementarity of legal practitioners and machine decision-making.

[0070] Compared with other alternative methods of human-machine collaborative similar case retrieval in the prior art, namely Co-Retrievalw / Intersection, Co-Retrievalw / Union and Co-Retrievalw / NaiveEM, the present invention has better similar case retrieval effect. Based on the average effect of all different machine retrieval models, data sets, and evaluation indicators, the effect of the present invention is 5.89% better than that of the Co-Retrievalw / Intersection method, 10.15% better than that of the Co-Retrievalw / Union method, and 3.44% better than that of the Co-Retrievalw / NaiveEM method. Compared with the above-mentioned baseline human-machine collaborative similar case retrieval method, the better effect of the present invention shows that explicit evaluation and modeling of human decision-making uncertainty is an effective way to improve the effect of human-machine collaborative similar case retrieval.

[0071] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A retrieval method based on human-computer collaborative similar cases, characterized in that: The following steps are involved: S1. Clustering sentences in historical legal case documents; S2, obtain the joint probability decision of the legal practitioner and the machine through the ProtoEM algorithm, and create a confusion matrix for each prototype using the joint probability decision; S3. Probabilistically combine the legal practitioner’s decision on key sentences with the machine’s decision on key sentences, and adapt the legal practitioner’s discrete decision to the confusion matrix In the process of joint decision-making, a joint decision-making model is established; S4. Use the joint decision-making model to retrieve matching legal cases.

2. A retrieval method based on human-computer collaborative similar cases according to claim 1, characterized in that: S1 includes the following steps: First, the text in the historical legal case documents was segmented into individual sentences; Then, the sentences are input into the legal-bert model to encode the features of each sentence; Finally, the number of clusters is determined, and the K-means algorithm is used to cluster the encoded sentence features.

3. The method for retrieving similar cases based on human-computer collaboration according to claim 1, characterized in that: In S2, the joint probability decision creates a confusion matrix for each prototype: In the formula, 1[·] is an indicator function; H i For legal practitioners to make decisions on key sentences; i is the machine's decision on key sentences; i is the real decision; k is the kth sentence; It is a joint decision made by legal practitioners and machines.

4. The method for retrieving similar cases based on human-computer collaboration according to claim 1, characterized in that: In S3, the confusion matrix for: Where, d i is the i-th sentence; is the average sentence embedding on prototype pk.

5. The method for retrieving similar cases based on human-computer collaboration according to claim 1, characterized in that: In S3, the joint decision model is: In the formula, H i For legal practitioners to make decisions on key sentences; i is the machine's decision on key sentences; i is the real decision; k is the kth sentence; M ij M i The jth dimension of ; the sentence category of j real label; C is the category of the sentence; [φ i ] kq For each element, the decision q at this time and the decision H of the legal practitioner for the key sentence i In sentence d i The inconsistency rate on i ] kj For each element, pass the true decision y i and legal practitioners’ decisions on key sentences i In sentence d i The inconsistency rate on .

6. A device, characterized in that: The device includes a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the retrieval method based on human-computer collaborative cases as described in any one of claims 1 to 5.

7. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by the processor, it is used to implement the retrieval method based on human-computer collaborative similar cases as described in any one of claims 1 to 5.