Judicial document index extraction method based on few-shot contrast learning

Abstract

A kind of element index extraction method for judicial text, comprising the following steps: 1) obtaining judicial document data, data cleaning is carried out on judicial document data, and training data based on judicial documents are constructed.2) a kind of judicial document index structured language (JDISL) is proposed for judicial document index.Using JDISL for prompt guidance, the construction form of index based on prompt is obtained, and the training corpus is further processed into prompt training corpus.3) the prompt training corpus is subjected to few-shot negative sampling augmentation.4) on the basis of open source Unilm basic pre-training model, the prompt training corpus is used as input.5) the Unilm model hidden layer vector is processed into Gaussian embedding.6) the contrast learning loss of Gaussian embedding is calculated, and the final model is obtained by iterative updating.7) the judicial document to be extracted index is input into the trained model, the index extraction model carries out information extraction, and the information extraction result of each label type is obtained.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of judicial document data processing, and specifically designs a judicial document index extraction method based on few-shot contrast learning (i.e., based on a knowledge graph). BACKGROUND

[0002] In the information age, the amount of Internet data grows exponentially. With the rapid development of Internet technology, information on the network is growing explosively. Not only is the information scale expanding, but the information types are also increasing. At the same time, the successful application of a large amount of data in various fields heralds the arrival of the era of big data. Big data plays an increasingly important role in social development, and its value has been universally recognized by society. In recent years, with the continuous deepening of China's legal construction, the trial of judicial cases has become increasingly transparent, and the public disclosure of judicial documents on the Internet is a typical example. As a "judicial product" that carries the court case trial process and the trial result, judicial documents contain rich judicial information, including the judgment court, case number, litigant's claim, case name, judgment result, applicable law, etc. These exactly gather the core elements of the court "big data". By deeply mining these information, the trial rule of the case can be summarized, the trial trend can be predicted, and the judicial credibility can be improved, which provides technical support for realizing judicial justice and building a legal society. However, judicial documents are a kind of semi-structured domain text, which has both programmatic legal language and daily ordinary language. At the same time, the writing of judicial documents is largely determined by judges, which makes judicial documents have a series of characteristics such as polymorphism, heterogeneity and randomness. Therefore, how to extract valuable information from this kind of special text is an important topic with great value and significance.

[0003] There is a wide range of index extraction needs in the field of justice, which is used to evaluate the effectiveness of judicial procedures or judicial reform. With the construction of a legal society in China and the development of information technology, the number of judicial documents in the field of justice is growing exponentially. The situation of more cases and fewer people is difficult to meet the needs of case handling, and the differences in professional competence of judicial personnel will also have a certain impact on case judgment. Therefore, the intelligent extraction of key elements of judicial documents can provide reference for judicial personnel, assist in case handling, improve work efficiency, and is also the key to subsequent in-depth analysis and efficient case judgment.

[0004] Currently, there is a problem of scarcity of available data resources in the field of judicial documents, and the annotation difficulty is great, which belongs to a low-resource few-shot application scenario; the judicial document index extraction task includes entity extraction, relation extraction, event extraction and other sub-tasks, and different sub-tasks require different annotation data and algorithm models, which leads to the application demand of training multiple sub-task models in a low-resource few-shot scenario. In summary, the judicial document index extraction faces the problems of difficulty in converging of multi-task algorithm model in a low-resource few-shot application scenario, low extraction accuracy and poor generalization, and such a judicial document index extraction model cannot meet the goal of assisting judicial personnel to handle cases and improving work efficiency. Therefore, the present application provides a judicial document index extraction method based on few-shot contrast learning (i.e., based on a knowledge graph) to effectively solve the above problems. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a judicial document index extraction method based on few-shot contrast learning (i.e., based on a knowledge graph). First, judicial document data is obtained, the judicial document data is cleaned, and training data based on judicial documents is constructed. Then, the judicial document index is prompted using a designed judicial document index structured language (JDISL), and the training corpus is further processed into prompt training corpus. Subsequently, the prompt training corpus is subjected to few-shot negative sampling augmentation. On the basis of the open-source UniLM pre-training model, the prompt training corpus is taken as input, the UniLM model hidden layer vector is processed into Gaussian embedding, the contrast learning loss of the Gaussian embedding is calculated, and the final model is obtained by iterative updating. The judicial document to be extracted is input into the trained model to obtain the information extraction result of each label type.

[0006] In view of the deficiencies of the prior art, the present application provides a judicial document index extraction method based on few-shot contrast learning.

[0007] According to one embodiment of the present application, a judicial document index extraction method based on few-shot contrast learning is provided, which comprises the following steps:

[0008] 1) Obtain judicial document data (such as publicly disclosed judicial documents or judicial document data), clean the judicial document data, and construct training data based on judicial documents (i.e., judicial documents) containing training corpus;

[0009] 2) A judicial document index structured language (JDISL) is proposed for the judicial document index, prompt guidance is performed using JDISL, the index is constructed in the form of prompt, and the training corpus is further processed into prompt training corpus;

[0010] 3) The prompt training corpus is amplified by few-shot negative sampling to obtain an amplified prompt training corpus;

[0011] 4) On the basis of the open-source Unilm basic pre-training model, the amplified prompt training corpus is taken as input;

[0012] 5) Take the Unilm model hidden layer vector and process it into a Gaussian embedding;

[0013] 6) Calculate the contrastive learning loss of the Gaussian embedding, and iteratively update to obtain the final model;

[0014] 7) Input the judicial document to be extracted into the trained final model, and the index extraction model performs information extraction to obtain the information extraction results of each label type.

[0015] Preferably, in step 2), the prompt is used to guide the index of the judicial document, and the index is constructed based on the prompt. The process of further processing the training corpus into a prompt training corpus includes the following two sub-steps 2.1) and 2.2):

[0016] Step 2.1) uses information extraction structured generation means (or information extraction means) including entity extraction, relation extraction, event extraction, and response extraction to construct the prompt of the court session element index in the judicial document. In order to unify the modeling into a text-to-prompt structure generation framework, the entity index, relationship index, event index, and response index in the judicial document are unified.

[0017] Information extraction is affected by different targets, heterogeneous structures, and specific demand patterns, and is divided into entity extraction, relation extraction, event extraction, and response extraction. The prompt of the court session element index is constructed. The unified modeling is a text-to-prompt structure generation framework, which realizes the unified modeling of the entity index, relationship index, event index, and response index in the judicial document.

[0018] Information extraction structured generation can be decomposed into two atomic operations.

[0019] 1) Positioning: locating the target information segment from the sentence, which includes, for example, entities and events in the judicial document.

[0020] 2) Association: connecting different information segments according to the required association, such as the connection between an entity and another entity, the connection between an entity and an event, or the connection between an event and another event.

[0021] Different information extraction task structures can be generated by combining atomic operations.

[0022] According to the structural characteristics of the judicial document index, a judicial document index structured language (JDISL) is designed, wherein the symbol ":" of the JDISL represents the mapping from information to its positioning or associated segment, and the symbols "[] and ()" are used to represent the hierarchy between the extracted index information.

[0023] The JDISL consists of three parts:

[0024] 1) info_name: represents a specific information segment existing in the source text

[0025] 2) relation_name: represents a specific information segment existing in the source text, which has an association with the upper-layer info information in the structure.

[0026] 3) info_span: represents the text segment span corresponding to the specific information or associated segment in the source text.

[0027] Through the JDISL, different information extraction structures can be uniformly coded, and different judicial document index information extraction tasks can be uniformly modeled as the same generation process from text to structure.

[0028] For entity recognition and event detection tasks, they can be modeled as:

[0029] (info_name:info_span),

[0030] For relationship extraction and event extraction tasks, they can be uniformly modeled as:

[0031] (info_name:info_span(relation_name:info_span),…),

[0032] Taking the court session elements as an example, the indexes to be extracted include witnesses, plaintiffs, defendants, evidence, courts, and judges, and the relationships between the indexes include (plaintiff) sues (defendant). After being prompted, for entities, they are represented as:

[0033] [witness, plaintiff, defendant, judge]

[0034] For attributes and relationships, they are represented as:

[0035] [court: evidence, witness: (testimony), [plaintiff: (sues)], defendant, judge]

[0036] Step 2.2) After constructing the prompt for the court session element index in the judicial document, the training data containing the training corpus are processed based on the prompt construction, thereby obtaining a prompt training corpus.

[0037] After constructing the prompt of the court trial element indicators, the training corpus needs to be processed based on the prompt construction to obtain the prompt training corpus.

[0038] As for the original corpus:

[0039] Content: "The plaintiff Cui and the defendant Zhang Labor Dispute Case, witness Liu's testimony shows that Cui has fault."

[0040] Label: [2, 3, "plaintiff"], [7, 8, "defendant"], [20, 21, "witness"], [27, 32, "testimony"], [(2, 3), (7, 8), "sue"]

[0041] Based on the prompt construction processing, the prompt structured corpus is obtained:

[0042] Content: "The plaintiff Cui and the defendant Zhang Labor Dispute Case"

[0043] Result: [(2, 3, "Cui"): ("sue"), (7, 8, "Zhang"): ("sued"), (20, 21, "Liu"): (27, 32, "Cui has fault")]

[0044] The prompt construction is a pre-defined pattern, which processes the training text into a prompt structured corpus.

[0045] Preferably, the process of step 3) of performing few-shot negative sampling augmentation on the prompt training corpus more specifically includes the following steps:

[0046] Step 3.1) for the prompt construction corpus:

[0047] Content: text

[0048] Result: [[info_type_1], [info_type_2], …, [info_type_n]]

[0049] Where [info_type_n] represents the extracted label type n structure form of the information set [[info_name], [relation_name], [info_span]].

[0050] The Result is processed into multiple result subsets based on the prompt:

[0051] Result_1: [info_type_1] [prompt: label_1]

[0052] Result_2: [info_type_2] [prompt: label_2]

[0053] …

[0054] Result_n: [info_type_n] [prompt: label_n]

[0055] wherein label_n represents the label type n corresponding to the [info_type_n] set;

[0056] Subsequently, based on the prompt type, negative sampling is performed, and the data volume is expanded from n to n^2, such as expanding Result_1 to n times:

[0057] Result_1: [info_type_1] [prompt: label_1]

[0058] Result_1: [info_type_1] [prompt: label_2]

[0059] …

[0060] Result_1: [info_type_1] [prompt: label_n].

[0061] Thus, the expanded prompt training corpus is obtained.

[0062] Preferably, in step 4), based on the open source Unilm base pre-training model, the process of taking the prompt training corpus as input is more specifically comprised of:

[0063] Step 4.1) processing the prompt training corpus into the input form required by the Unilm model. The input form that can be accepted by the Unilm model is:

[0064] {x1, x2, x3, …, x m-1 , x m}

[0065] The prompt structured information obtained by JDISL is [[info_name], [relation_name], [text]], wherein the text is the information referred to within the info_span span in the source text.

[0066] The structured information of the prompt is then used as a prefix and associated with the fragment information to obtain the model input corpus: {info_name_1,…info_name_n,…,relation_name_1,…,relation_name_n,…,text_1,…,text_n,x1,x2,x3,...,x m-1 x m};

[0067] In the above, the value of n refers to the number of entity / relation fragments. The value of m refers to the sequence length of the input document.

[0068] Unilm can be used for both generation and classification tasks. After input, the input is first segmented using a pre-trained Unilm tokenizer, followed by vector embedding, which includes token embedding, segment embedding, and position embedding.

[0069] Token embedding refers to inserting a [CLS] token at the beginning of each sentence and a [SEP] token at the end of each sentence. The [CLS] token represents the vector of the current sentence, and the [SEP] token represents the clause used to segment sentences in the text.

[0070] Segment embedding is used to distinguish between two sentences, with A and B tags preceding the different sentences respectively. Therefore, the input sentence is represented as (E A E B E A E B ,……).

[0071] Position embedding adds positional information based on the relative position of the sentence's index, such as [0,1,2,3…].

[0072] After embedding, the hidden layer output vector is calculated through 24 cascaded transformer layers and a multi-head attention mechanism.

[0073] h={H; h1, h2, h3,..., h m-1 h m}

[0074] Among them, h m H represents the hidden state of the m-th judicial document sequence, and H represents the state set of the prompt structured information.

[0075] Preferably, step 5) involves taking the hidden layer vectors of the Unilm model and processing them into Gaussian embeddings. More specifically, this process includes:

[0076] According to step 5.1), a projection network f consisting of fully connected layers and nonlinear activation functions is used.u and f ∑ to generate Gaussian distribution parameters:

[0077] u i = f u (h i );

[0078] Σ i = ELu(f ∑ (h i )) + (1 + ∈)

[0079] where u i , Σ i represent the mean and diagonal covariance (non-zero elements only along the diagonal of the matrix) of the Gaussian embedding respectively, f u and f ∑ are both composed of single linear layers with Relu as the activation function, and ∈ takes e^(-14) for stability.

[0080] The hidden layer output vector is processed into Gaussian embedding (u i , Σ i ).

[0081] Preferably, the process of iteratively updating to obtain the final model of the contrastive learning loss calculated in step 6) to calculate the Gaussian embedding more specifically includes the following two sub-steps 6.1) and 6.2):

[0082] Step 6.1) To calculate the contrastive loss, consider the KL divergence between all valid token pairs in the sampling corpus, if two tokens x q and x p have the same label y q = y p , they are considered as positive examples, for their Gaussian embedding N(u p , Σ p ) and N(u q , ∑ q ), the KL divergence is calculated for it:

[0083]

[0084] Both directions of KL divergence are calculated because it is not symmetric.

[0085] d(p, q) = 1 / 2(D KL |N p ||N q | + D KL |N q ||N p |)

[0086] For positive samples xp , compute x p Gaussian embedding loss with respect to other valid tokens in the batch:

[0087] χ p = {(χ q , y q ) e χ | y p = y q · p≠ q}

[0088] Finally, the contrastive learning loss is calculated using the KL divergence and the Gaussian embedding loss:

[0089]

[0090] Step 6.2) Backpropagation based on the contrastive learning loss function, iterative update of the UniLM model parameters, and obtain the final information extraction model that can be used for judicial document index extraction, i.e., the final model.

[0091] Currently, there is a lack of available annotated data in the field of judicial documents, and the annotation is difficult, which is a low-resource and few-sample application scenario. The judicial document index extraction task includes entity extraction, relation extraction, event extraction, and other sub-tasks. Different sub-tasks require different annotated data and algorithm models, which leads to the application demand of training multiple sub-task models in the low-resource and few-sample scenario. Therefore, there is an urgent need for a judicial document index extraction method to solve the problems of low-resource and few-sample application scenarios, such as difficulty in converging, low extraction accuracy, and poor generalization of multi-task algorithm models.

[0092] In the present application, prompt guidance is used for judicial document indexes, and the index is constructed based on prompt to further process the training corpus into prompt training corpus.

[0093] Information extraction is affected by different targets, heterogeneous structures, and specific demand patterns, and can be divided into entity extraction, relation extraction, event extraction, and response extraction. The prompt structure is constructed for the elements of the court trial. The unified modeling is a text-to-prompt structure generation framework, which realizes the unified modeling of entity indexes, relationship indexes, event indexes, and response indexes in judicial documents.

[0094] Information extraction and structured generation can be decomposed into two atomic operations.

[0095] 1) Positioning: locating the target information segment from the sentence, such as entities and events in the document.

[0096] 2) Association: connecting different information segments according to the required association, such as the connection between entities and events.

[0097] Different information extraction task structures can be generated by combining atomic operations.

[0098] According to the structural characteristics of judicial document indicators, a judicial document indicator structured language (JDISL) is designed. The JDISL represents the mapping from information to its positioning or associated segments, and [ ], ( ) are used to represent the hierarchy between extracted indicator information.

[0099] The JDISL consists of three parts:

[0100] 1) info_name: represents a specific information segment existing in the source text

[0101] 2) relation_name: represents a specific information segment existing in the source text, which has a correlation with the upper layer info information in the structure.

[0102] 3) info_span: represents the text segment span corresponding to the specific information or associated segment in the source text.

[0103] Through JDISL, different information extraction structures can be uniformly coded, and different judicial document indicator information extraction tasks can be uniformly modeled as the same generation process from text to structure.

[0104] For entity recognition and event detection tasks, they can be modeled as:

[0105] (info_name:info_span),

[0106] For relationship extraction and event extraction tasks, they can be uniformly modeled as:

[0107] (info_name:info_span(relation_name:info_span),…),

[0108] Taking the court hearing elements as an example, the indicators to be extracted include witnesses, plaintiffs, defendants, evidence, courts, and judges, and the relationships between the indicators include (plaintiff) sues (defendant). The prompt is guided, and for entities, it is represented as:

[0109] [witness, plaintiff, defendant, judge]

[0110] For attributes and relationships, they are represented as:

[0111] [court: evidence, witness: (testimony), [plaintiff: (sue)], defendant, judge]

[0112] After constructing the prompt of the court trial element index, the training corpus needs to be processed based on the prompt construction to obtain a prompt training corpus.

[0113] As for the original corpus:

[0114] Content: "The plaintiff Cui and the defendant Zhang Labor Dispute Case, witness Liu's testimony shows that Cui has fault."

[0115] Label: [2, 3, "plaintiff"], [7, 8, "defendant"], [20, 21, "witness"], [27, 32, "testimony"], [(2, 3), (7, 8), "sue"]

[0116] Based on the prompt construction processing, the prompt structured corpus is obtained:

[0117] Content: "The plaintiff Cui and the defendant Zhang Labor Dispute Case"

[0118] Result: [(2, 3, "Cui"): ("sue"), (7, 8, "Zhang"): ("sue"), (20, 21, "Liu"): (27, 32, "Cui has fault")]

[0119] The prompt construction is a pre-defined mode, which processes the training text into a prompt structured corpus.

[0120] In the present application, the prompt training corpus is subjected to few-shot negative sampling augmentation.

[0121] For the prompt structured corpus:

[0122] Content: text

[0123] Result: [[info_type_1], [info_type_2], …, [info_type_n]]

[0124] Where [info_type_n] represents the extracted label type n structure form [[info_name], [relation_name], [info_span]] information set.

[0125] The Result is processed into multiple result subsets based on the prompt:

[0126] Result_1: [info_type_1] [prompt: label_1]

[0127] Result_2: [info_type_2] [prompt: label_2]

[0128] …

[0129] Result_n: [info_type_n] [prompt: label_n]

[0130] wherein label_n represents the label type n corresponding to the [info_type_n] set;

[0131] Subsequently, based on the prompt type, negative sampling is performed, and the data amount is expanded from n to n^2, such as expanding Result_1 to n times:

[0132] Result_1: [info_type_1] [prompt: label_1]

[0133] Result_1: [info_type_1] [prompt: label_2]

[0134] …

[0135] Result_1: [info_type_1] [prompt: label_n].

[0136] The expanded prompt training corpus is obtained above.

[0137] In the present application, on the basis of the open source Unilm base pre-training model, the prompt training corpus is taken as input.

[0138] The prompt training corpus is processed into the input form required by the Unilm model. The input form that can be accepted by the Unilm model is:

[0139] {x1, x2, x3, …, x m-1 , x m}

[0140] The prompt structured information obtained by JDISL is [[info_name], [relation_name], [text]], wherein the text is the information referred to within the info_span span in the source text.

[0141] Subsequently, the prompt structured information is associated as a prefix with the segment information to obtain the model input corpus:

[0142] {info_name_1,..., info_name_n,..., relation_name_1,..., relation_name_n,..., text_1,..., text_n, x1, x2, x3,..., xn m-1 , x m}

[0143] Unilm can be used for both generation tasks and classification tasks. After input, the input is first segmented using the tokenizer of the Unilm pre-training, and then vector embedding is performed, which includes token embedding, segment embedding, and position embedding.

[0144] Token embedding refers to inserting [CLS] markers at the beginning of each sentence and [SEP] markers at the end of each sentence, where the [CLS] marker represents the vector of the current sentence, and the [SEP] marker represents the division of the sentence in the text.

[0145] Segment embedding is used to distinguish between two sentences, and different sentences are marked with A and B respectively, so the input sentence is represented as (E A , E B , E A , E B ,...).

[0146] Position embedding is to add position information according to the index position of the sentence, such as [0, 1, 2, 3,...].

[0147] After embedding, it will pass through 24 transformer structure layers in series, and the hidden layer output vector

[0148] h = {H; h1, h2, h3,..., hn} is calculated through a multi-head attention mechanism. m-1 , h m}

[0149] Where h m represents the hidden layer state of the mth judicial document sequence, and H represents the state set of the prompt structured information.

[0150] In the present application, the Unilm model hidden layer vector is processed as a Gaussian embedding.

[0151] The projection network f u and f ∑ composed of a fully connected layer and a nonlinear activation function are used to generate Gaussian distribution parameters:

[0152] u i = f u (h i );

[0153] ∑ i =ELU(f ∑ (h i ))+(1+∈)

[0154] Where u i , ∑ i Let f represent the mean covariance and diagonal covariance of the Gaussian embedding, respectively (non-zero elements are only along the diagonal of the matrix). u and f ∑ All are composed of single linear layers, with ReLU as the activation function, and e^(-14) is chosen for stability.

[0155] The hidden layer output vector is processed into a Gaussian embedding (u i , Σ i ).

[0156] In this invention, the contrastive learning loss of Gaussian embedding is calculated, and the final model is obtained by iterative update.

[0157] To calculate the contrast loss, consider the KL divergence among all valid token pairs in the sampled corpus. If two tokens x q and x p Having the same label y q =y p Then they are considered positive examples for their Gaussian embedding N(u p , ∑ p ) and N(u q , Σ q ), calculate its KL divergence:

[0158]

[0159]

[0160] Both directions of the KL divergence are calculated because it is not symmetrical.

[0161] d(p,q)=1 / 2(D KL |N p ||N q |+D KL |N q ||N P |)

[0162] For positive sample x p Calculate x p Gaussian embedding loss relative to other valid tokens in the batch:

[0163] χ p ={(χ q y g)∈χ[y p =y q , p≠q

[0164] Finally, the contrastive learning loss is calculated using KL divergence and Gaussian embedding loss:

[0165]

[0166] The UniLM model parameters are iteratively updated based on the backpropagation of the contrastive learning loss function, and the final information extraction model for judicial document index extraction is obtained.

[0167] Compared with the prior art, the technical scheme of the present application has the following beneficial technical effects:

[0168] 1. In the present application, a judicial document index structured language (JDISL) is proposed, and the JDISL is used for prompt guidance to obtain a prompt-based construction form of the index. The training corpus is further processed into prompt training corpus, effectively encoding different information extraction structures, thereby enabling effective joint extraction, effectively solving the problem of unified modeling difficulty of multiple extraction sub-tasks of the judicial document index extraction multi-task algorithm model and excessive training cost.

[0169] 2. In the present application, the prompt training corpus is subjected to few-shot negative sampling augmentation, effectively solving the problem of few-shot learning difficulty caused by too few available labeled data in the low-resource field of judicial documents.

[0170] 3. In the present application, the Unilm model hidden layer vector is processed into Gaussian embedding, and the information class distribution is explicitly modeled by Gaussian embedding, promoting the generalized feature representation of various indicators, which is helpful for the self-adaptation of the few-shot target domain, effectively solving the problems of poor generalization performance and low extraction precision of the few-shot learning model.

[0171] 4. In the present application, the contrastive learning loss function is used to calculate the loss of the training model, and the final model is obtained by iterative updating, effectively solving the problems of model training convergence difficulty and low extraction precision. BRIEF DESCRIPTION OF DRAWINGS

[0172] Figure 1 The present application is based on a few-shot contrastive learning method for extracting judicial document indexes.

[0173] Figure 2 The few-shot data processing module structure diagram of the present application based on a few-shot contrastive learning method for extracting judicial document indexes.

[0174] Figure 3The invention is based on a few-shot contrast learning network structure diagram of a judgment document index extraction method based on few-shot contrast learning. DETAILED DESCRIPTION

[0175] The technical solutions of the invention are illustrated below, and the scope of protection requested by the invention includes but is not limited to the examples.

[0176] A judgment document index extraction method based on few-shot contrast learning, which comprises the following steps:

[0177] 1) Obtain judgment document data, clean the judgment document data, and construct training data based on judgment documents.

[0178] 2) A judgment document index structured language (JDISL) is proposed for the judgment document index, prompt guidance is performed using JDISL, the index is constructed based on the prompt, and the training corpus is further processed into prompt training corpus.

[0179] 3) The prompt training corpus is subjected to few-shot negative sampling augmentation.

[0180] 4) On the basis of the open source Unilm basic pre-training model, the prompt training corpus is taken as input.

[0181] 5) Take the Unilm model hidden layer vector and process it into Gaussian embedding.

[0182] 6) Calculate the contrast learning loss of the Gaussian embedding, and iteratively update to obtain the final model.

[0183] 7) Input the judgment document to be extracted into the trained model, and perform information extraction on the index extraction model to obtain the information extraction results of each label type.

[0184] As a preferred, in step 2), the judgment document index is guided by prompt, the index is constructed based on prompt, and the training corpus is further processed into prompt training corpus, which comprises:

[0185] Step 2.1) Information extraction is affected by different targets, heterogeneous structures and specific demand patterns, and is divided into entity extraction, relation extraction, event extraction, response extraction, etc. The prompt structure of the court session element index is constructed. Unified modeling is performed as a text-to-prompt structure generation framework, and unified modeling is realized for entity indexes, relationship indexes, event indexes and response indexes in judgment documents.

[0186] Information extraction structured generation can be decomposed into two atomic operations.

[0187] 1) Positioning: Locate the target information segment from the sentence, such as entities and events in documents, etc.

[0188] 2) Association: Connect different information segments according to the required association, such as the connection between entities and entities, the connection between entities and events.

[0189] Different information extraction task structures can be generated by combining atomic operations.

[0190] According to the structure of the judicial document index, a judicial document index structured language (JDISL) is designed. JDISL: represents the mapping from information to its positioning or association segment, [ ], () is used to represent the hierarchy between extracted index information.

[0191] JDISL consists of three parts:

[0192] 1) info_name: represents a specific information segment existing in the source text

[0193] 2) relation_name: represents a specific information segment existing in the source text, which has an association with the upper layer info information in the structure.

[0194] 3) info_span: represents the text segment span corresponding to the specific information or association segment in the source text.

[0195] Through JDISL, different information extraction structures can be uniformly coded, and different judicial document index information extraction tasks can be uniformly modeled as the same generation process from text to structure.

[0196] For entity recognition and event detection tasks, they can be modeled as:

[0197] (info_name:info_span)

[0198] For relationship extraction and event extraction tasks, they can be uniformly modeled as:

[0199] (info_name:info_span(relation_name:info_span),…)

[0200] Taking the court hearing elements as an example, the indexes to be extracted include witnesses, plaintiffs, defendants, evidence, courts, and judges, and the relationships between the indexes include (plaintiff) suing (defendant). The prompt is guided, and for entities, it is represented as:

[0201] [ witness, plaintiff, defendant, judge ]

[0202] For attributes and relationships, they are represented as:

[0203] [Court: evidence, witness: (testimony), [plaintiff: (sue)], defendant, judge]

[0204] Step 2.2) After constructing the prompt for the court element index, the training corpus needs to be processed based on the prompt construction to obtain the prompt training corpus.

[0205] As for the original corpus:

[0206] Content: “The plaintiff Cui and the defendant Zhang labor contract dispute case, the witness Liu’s testimony shows that Cui has fault.”

[0207] Label: [2, 3, “plaintiff”], [7, 8, “defendant”], [20, 21, “witness”], [27, 32, “testimony”], [(2, 3), (7, 8), “sue”]

[0208] Based on the prompt construction processing, the prompt structured corpus is obtained:

[0209] Content: “The plaintiff Cui and the defendant Zhang labor contract dispute case”

[0210] Result: [(2, 3, “Cui”): (“sue”), (7, 8, “Zhang”): (“sue”), (20, 21, “Liu”): (27, 32, “Cui has fault”)]

[0211] Prompt construction as a predefined pattern, processing training text into prompt structured corpus form.

[0212] As a preferred, step 3) prompt training corpus is amplified by few-shot negative sampling:

[0213] Step 3.1) for prompt construction corpus:

[0214] Content: text

[0215] Result: [[info_type_1], [info_type_2], …, [info_type_n]]

[0216] Where [info_type_n] represents the extracted label type n structure form [[info_name], [relation_name], [info_span]] information set.

[0217] The Result is processed into multiple result subsets based on promt:

[0218] Result_1: [info_type_1] [prompt: label_1]

[0219] Result_2: [info_type_2] [prompt: label_2]

[0220] …

[0221] Result_n: [info_type_n] [prompt: label_n]

[0222] where label_n represents the label type n corresponding to the [info_type_n] set

[0223] Subsequently, based on the prompt type, negative sampling is performed, expanding the data volume from n to n^2, such as expanding Result_1 to n times:

[0224] Result_1: [info_type_1] [prompt: label_1]

[0225] Result_1: [info_type_1] [prompt: label_2]

[0226] …

[0227] Result_1: [info_type_1] [prompt: label_n]

[0228] As a preferred, step 4) is based on the open source Unilm base pre-training model, and the prompt training corpus is taken as input:

[0229] Step 4.1) processes the prompt training corpus into the input form required by the Unilm model. The input form that the Unilm model can accept is:

[0230] {x1, x2, x3, …, x m-1 , x m}

[0231] The prompt structured information obtained by JDISL is [[info_name], [relation_name], [text]], where text is the information referred to within the info_span span in the source text.

[0232] Subsequently, the prompt structured information is associated as a prefix with the segment information to obtain the model input corpus:

[0233] {info_name_1, info_name_n,..., relation_name_1,..., relation_name_n,..., text_1,..., text_n, x1, x2, x3,..., xn m-1 , x m}

[0234] Unilm can be used for both generation tasks and classification tasks. After input, the tokenizer of Unilm is first used to segment the input, and then vector embedding is performed, including token embedding, segment embedding, and position embedding.

[0235] Token embedding refers to inserting [CLS] markers at the beginning of each sentence and [SEP] markers at the end of each sentence, where the [CLS] marker represents the vector of the current sentence, and the [SEP] marker represents the division of the sentences in the text.

[0236] Segment embedding is used to distinguish between two sentences, and different sentences are marked with A and B respectively, so the input sentence is represented as (E A , E B , E A , E B , …).

[0237] Position embedding is to add position information according to the index position of the sentence, such as [0, 1, 2, 3…].

[0238] After embedding, it will pass through 24 layers of transformer structure in series, and the hidden layer output vector

[0239] h = {H; h1, h2, h3,..., hn} is obtained by calculating the multi-head attention mechanism. m-1 , h m}

[0240] Where h m represents the hidden layer state of the mth judicial document sequence, and H represents the state set of the prompt structured information.

[0241] As a preferred, step 5) takes the Unilm model hidden layer vector and processes it into Gaussian embedding:

[0242] According to step 5.1), the projection network f u and f ∑ composed of a fully connected layer and a nonlinear activation function are used to generate Gaussian distribution parameters:

[0243] u i = f u (h i );

[0244] ∑ i =ELU(f ∑ (h i ))+(1+∈)

[0245] Where u i , ∑ i Let f represent the mean covariance and diagonal covariance of the Gaussian embedding, respectively (non-zero elements are only along the diagonal of the matrix). u and f Σ All are composed of single linear layers, with ReLU as the activation function, and e^(-14) is chosen for stability.

[0246] The hidden layer output vector was then processed into a Gaussian embedding (u i , Σ i ).

[0247] Preferably, step 6) calculates the contrastive learning loss of the Gaussian embedding and iteratively updates to obtain the final model:

[0248] Step 6.1) To calculate the contrast loss, consider all valid t values ​​in the sampled corpus. o If the two tokens x diverge in KL, then... q and x p Having the same label y q =y p Then they are considered positive examples for their Gaussian embedding N(u p , Σ p ) and N(u q , ∑ q ), calculate its KL divergence:

[0249]

[0250] Both directions of the KL divergence are calculated because it is not symmetrical.

[0251] d(p, q) = 1 / 2(D KL |N p ||N q |+D KL |N q ||N p |)

[0252] For positive sample x p Calculate x p Gaussian embedding loss relative to other valid tokens in the batch:

[0253] χ p =((χ) q y q )∈χ|y p= y q , p≠q

[0254] Finally, the contrastive learning loss is calculated using KL divergence and Gaussian embedding loss:

[0255]

[0256] Step 6.2) Backpropagation based on the contrastive learning loss function, iterative update of UniLM model parameters, and obtain the final information extraction model for judicial document index extraction.

[0257] Embodiment 1

[0258] As shown in Figure 1 , a judicial document index extraction method based on few-shot contrastive learning includes the following steps:

[0259] 1) Obtain judicial document data, clean the data, and construct training data based on judicial documents.

[0260] 2) A judicial document index structured language (JDISL) is proposed for judicial document index. Prompt guidance is used to obtain the index based on prompt construction, and the training corpus is further processed into prompt training corpus.

[0261] 3) Few-shot negative sampling augmentation is performed on the prompt training corpus.

[0262] 4) Based on the open-source Unilm base pre-training model, the prompt training corpus is used as input.

[0263] 5) Take the Unilm model hidden layer vector and process it into Gaussian embedding.

[0264] 6) Calculate the contrastive learning loss of Gaussian embedding and iteratively update to obtain the final model.

[0265] 7) Input the judicial document to be extracted into the trained model, and perform information extraction using the index extraction model to obtain the information extraction results of each label type.

[0266] Embodiment 2

[0267] Repeat Embodiment 1, except that in step 2), prompt guidance is used for judicial document index to obtain the index based on prompt construction, and the training corpus is further processed into prompt training corpus, as shown in Figure 2

[0268] ​Step 2.1) Information extraction is affected by different goals, heterogeneous structures, and specific demand patterns, and is divided into entity extraction, relationship extraction, event extraction, response extraction, etc. The prompt structure is constructed for the court hearing element indicators. Unified modeling is generated from text to prompt structure, and unified modeling is realized for entity indicators, relationship indicators, event indicators, and response indicators in judicial documents.

[0269] Information extraction and structured generation can be divided into two atomic operations.

[0270] 1) Positioning: locating the target information segment from the sentence, such as entities and events in the document.

[0271] 2) Association: connecting different information segments according to the required association, such as the connection between entities and events.

[0272] Different information extraction task structures can be generated by combining atomic operations.

[0273] According to the structural characteristics of judicial document indicators, a judicial document indicator structured language (JDISL) is designed. JDISL represents the mapping from information to its positioning or association segment, and [ ], () are used to represent the hierarchical structure between extracted indicator information.

[0274] JDISL consists of three parts:

[0275] 1) info_name: represents a specific information segment in the source text

[0276] 2) relation_name: represents a specific information segment in the source text, which has an association with the upper layer info information in the structure.

[0277] 3) info_span: represents the text segment span corresponding to the specific information or associated segment in the source text.

[0278] Through JDISL, different information extraction structures can be uniformly coded, and different judicial document indicator information extraction tasks can be uniformly modeled as the same generation process from text to structure.

[0279] For entity recognition and event detection tasks, they can be modeled as:

[0280] (info_name: info_span)

[0281] For relationship extraction and event extraction tasks, they can be uniformly modeled as:

[0282] (info_name: info_span(relation_name: info_span),...)

[0283] Taking court elements as an example, the indicators to be extracted include witnesses, plaintiffs, defendants, evidence, courts, and judges, and the relationships between the indicators include (plaintiff) sues (defendant). Prompt guidance is performed on them, and for entities, they are listed as:

[0284] [witness, plaintiff, defendant, judge]

[0285] For attribute and relationship dependencies, they are represented as:

[0286] [court: evidence, witness: (testimony), [plaintiff: (sues)], defendant, judge]

[0287] Step 2.2) After constructing the prompt for the court element indicators, the training corpus needs to be processed based on the prompt construction, so as to obtain a prompt training corpus.

[0288] As for the original corpus:

[0289] Content: “The case of plaintiff Cui and defendant Zhang labor contract dispute, the testimony of witness Liu shows that Cui has fault.”

[0290] Label: [2, 3, “plaintiff”], [7, 8, “defendant”], [20, 21, “witness”], [27, 32, “testimony”], [(2, 3), (7, 8), “sues”]

[0291] Based on prompt construction processing, the prompt structured corpus is:

[0292] Content: “The case of plaintiff Cui and defendant Zhang labor contract dispute”

[0293] Result: [(2, 3, “Cui”): (“sues”), (7, 8, “Zhang”): (“sued”), (20, 21, “Liu”): (27, 32, “Cui has fault”)]

[0294] Prompt construction is a pre-defined pattern, which processes the training text into a prompt structured corpus form.

[0295] Embodiment 3

[0296] Repeat Embodiment 2, except that step 3) performs few-shot negative sampling augmentation on the prompt training corpus, as shown in: Figure 2

[0297] ​Step 3.1) For prompt construction corpus:

[0298] Content: text

[0299] Result: [[info_type_1], [info_type_2], …, [info_type_n]]

[0300] Where [info_type_n] represents the extracted information set of the structure type n in the form of [[info_name], [relation_name], [info_span]].

[0301] Process Result based on prompt into multiple result subsets:

[0302] Result_1: [info_type_1] [prompt: label_1]

[0303] Result_2: [info_type_2] [prompt: label_2]

[0304] …

[0305] Result_n: [info_type_n] [prompt: label_n]

[0306] Where label_n represents the label type n corresponding to the [info_type_n] set

[0307] Subsequently, based on the prompt type, negative sampling is performed, expanding the data amount from n to n^2, such as expanding Result_1 by n times:

[0308] Result_1: [info_type_1] [prompt: label_1]

[0309] Result_1: [info_type_1] [prompt: label_2]

[0310] …

[0311] Result_1: [info_type_1] [prompt: label_n]

[0312] Example 4

[0313] Repeat Example 3 as shown in Figure 2 Step 4) on the basis of the open-source Unilm base pre-training model, input the prompt training corpus, such asFigure 3 as shown:

[0314] Step 4.1) Process the prompt training corpus into the input form required by the Unilm model. The input form that can be accepted by the Unilm model is:

[0315] {x1, x2, x3,..., xn, x m-1 , x m}

[0316] The prompt structured information obtained by JDISL is [[info_name], [relation_name], [text]], where text is the information referred to within the info_span span in the source text.

[0317] Then the prompt structured information is associated with the segment information as a prefix to obtain the model input corpus:

[0318] {info_name_1,... info_name_n,..., relation_name_1,..., relation_name_n,..., text_1,..., text_n, x1, x2, x3,..., xn, x m-1 , x m}

[0319] Unilm can be used for both generation tasks and classification tasks. After input, the Unilm pre-trained tokenizer is first used to tokenize the input, and then vector embedding is performed, including token embedding, segment embedding, and position embedding.

[0320] Token embedding refers to inserting [CLS] markers at the beginning of each sentence and [SEP] markers at the end of each sentence, where [CLS] markers represent the vector of the current sentence, and [SEP] markers represent the division of sentences in the text.

[0321] Segment embedding is used to distinguish between two sentences, and different sentences are marked with A and B markers before them, so the input sentence is represented as (E A , E B , E A , E B ,...).

[0322] Position embedding is to add position information according to the index position of the sentence, i.e., or for example [0, 1, 2, 3,...].

[0323] After embedding, it will pass through 24 transformer structure layers in series, and the hidden layer output vector

[0324] h={H; h1, h2, h3,..., h m-1 h m}

[0325] Among them, h m H represents the hidden state of the m-th judicial document sequence, and H represents the state set of the prompt structured information.

[0326] Example 5

[0327] Repeat Example 4, as follows Figure 3 As shown, step 5) simply involves taking the hidden layer vectors of the Unilm model and processing them into Gaussian embeddings.

[0328] According to step 5.1), a projection network f consisting of fully connected layers and nonlinear activation functions is used. u and f ∑ To generate Gaussian distribution parameters, such as Figure 3 As shown:

[0329] u i =f u (h i );

[0330] ∑ i =ELU(f ∑ (h1)+(1+∈)

[0331] Where u i , ∑ i Let f represent the mean covariance and diagonal covariance of the Gaussian embedding, respectively (non-zero elements are only along the diagonal of the matrix). u and f ∑ All are composed of single linear layers, with ReLU as the activation function, and e^(-14) is chosen for stability.

[0332] The hidden layer output vector was then processed into a Gaussian embedding (u i , ∑ i ).

[0333] Example 6

[0334] Repeat Example 5, as follows Figure 3 As shown, step 6) involves calculating the contrastive learning loss of the Gaussian embedding and iteratively updating to obtain the final model.

[0335] Step 6.1) To calculate the contrast loss, consider the KL divergence among all valid token pairs in the sampled corpus. If two tokens x q and x p Having the same label y q =y q, they are considered positive examples, for their Gaussian embedding N(u p ,∑ p ) and N(u q ,∑ q ), the KL divergence is calculated for it:

[0336]

[0337] Both directions of KL divergence are calculated, as it is not symmetric.

[0338] d(p,q) = 1 / 2(D KL |N p ||N q | + D KL |N q ||N p |)

[0339] For positive samples x p , the Gaussian embedding loss of x p relative to other valid tokens in the batch is calculated:

[0340] x p = {(x q , y q ) ∈ χ | yp = y q , p ≠ q}

[0341] Finally, the contrastive learning loss is calculated using the KL divergence and the Gaussian embedding loss:

[0342]

[0343] Step 6.2), as shown in Figure 3 , based on the contrastive learning loss function, the UniLM model parameters are iteratively updated by backpropagation, and the final information extraction model that can be used for judicial document index extraction is obtained.

Claims

1. A method for extracting indicators from judicial documents based on few-sample contrastive learning, the method comprising the following steps: 1) Obtain judicial document data, perform data cleaning on the judicial document data, and construct training data based on judicial documents that includes training corpus; 2) A structured language for judicial document indicators, JDISL, is proposed. JDISL is used for prompt guidance, resulting in a prompt-based structure for the indicators. The training corpus is further processed into prompt training corpus; specifically including: Step 2.1) Using information extraction and structured generation methods, including entity extraction, relation extraction, event extraction, and response extraction, the trial element indicators in judicial documents are constructed into prompts to create a unified modeling framework from text to prompt structure. This achieves unified modeling of entity indicators, relation indicators, event indicators, and response indicators in judicial documents. Based on the specific structure of judicial document indicators, a judicial document indicator structured language, JDISL, is designed. The symbol ":" in JDISL represents the mapping from information to its location or related fragments, and the symbols "[]" and "()" are used to represent the relationships between extracted indicator information. The hierarchical structure of JDIS consists of three parts: 2.1.1) info_name: representing a specific information fragment in the source text; 2.1.2) relation_name: representing a specific information fragment in the source text that is related to the upper-level info information in the structure; 2.1.3) info_span: representing the span of the text fragment corresponding to the specific information or related fragment in the source text. By using JDIS to uniformly encode different information extraction structures, different judicial document indicator information extraction tasks are uniformly modeled into the same text-to-structure generation process. Step 2.2) After constructing the prompt for the trial element indicators in the judicial documents, the training data containing the training corpus is processed based on the prompt construction to obtain the prompt training corpus. 3) Perform few-sample negative sampling amplification on the prompt training corpus to obtain the amplified prompt training corpus; 4) Based on the open-source Unilm pre-trained model, the augmented prompt training corpus is used as input; 5) Take the hidden layer vectors of the Unilm model and process them into Gaussian embeddings; 6) Calculate the contrastive learning loss of the Gaussian embedding and iteratively update to obtain the final model; 7) Input the judicial documents for which the indicators to be extracted into the trained final model. The indicator extraction model extracts information and obtains the information extraction results for each label type.

2. The method for extracting judicial document indicators according to claim 1, characterized in that, In step 2.1), the information extraction structure generation is decomposed into two atomic operations: 1) Locating: Locating the target information fragment from the sentence, which includes entities and events in legal documents; 2) Association: Connect different information fragments according to the required association, that is, the connection between entities, the connection between entities and events, or the connection between events.

3. The method for extracting judicial document indicators according to claim 1, characterized in that, In step 2.1.3), different tasks of extracting judicial document indicator information are uniformly modeled into the same text-to-structure generation process. The model for entity recognition and event detection tasks is as follows: (info_name:info_span) The unified model for relation extraction and event extraction tasks is as follows: (info_name:info_span(relation_name:info_span),…).

4. The method for extracting judicial document indicators according to claim 1, characterized in that, The Prompt construct, as a predefined pattern, processes the training text into a structured corpus of prompts.

5. The method for extracting judicial document indicators according to claim 1, characterized in that, Step 3) involves amplifying the prompt training corpus using few-sample negative sampling, which includes: Step 3.1) Construct the corpus for prompt: Content:text Result:[[info_type_1],[info_type_2],…,[info_type_n]] Where [info_type_n] represents the extracted information set with tag type n and structure [[info_name], [relation_name], [info_span]]. Process the Result into multiple subsets of the result based on the promise: Result_1:[info_type_1][prompt:label_1] Result_2:[info_type_2][prompt:label_2] … Result_n:[info_type_n][prompt:label_n] Where label_n represents the label type n corresponding to the [info_type_n] set; Then, negative sampling is performed based on the prompt type, expanding the data volume from n to n^2, such as expanding Result_1 to n times: Result_1:[info_type_1][prompt:label_1] Result_1:[info_type_1][prompt:label_2] … Result_1:[info_type_1][prompt:label_n], This yields the amplified prompt training corpus.

6. The method for extracting judicial document indicators according to claim 1, characterized in that, Step 4) involves using the prompt training corpus as input based on the open-source Unilm pre-trained model. Step 4.1) Process the prompt training corpus into the input format required by the Unilm model; the input format that the Unilm model can accept is: {x1,x2,x3,…,x m-1 ,x m } The structured information of the prompt obtained from JDISL is [[info_name], [relation_name], [text]], where text is the information referred to within the span of info_span in the source text; The structured information of the prompt is then used as a prefix and associated with the fragment information to obtain the model input corpus: {info_name_1,…info_name_n,…,relation_name_1,…,relation_name_n,…,text_1,…,text_n,x1,x2,x3,…,x m-1 ,x m }; Unilm is used for both generation and classification tasks. After input, the input is first tokenized using a tokenizer pre-trained with Unilm, followed by vector embedding, which includes token embedding, segment embedding, and position embedding. In this context, token embedding refers to inserting a [CLS] token at the beginning of each sentence and a [SEP] token at the end of each sentence. The [CLS] token represents the vector of the current sentence, and the [SEP] token represents the clause used to segment sentences in the text. Segment embedding is used to distinguish between two sentences, with A and B tags preceding the different sentences respectively. Therefore, the input sentence is represented as (E A E B E A E B ,……); Among them, position embedding adds position information based on the relative position of the sentence's subscript index; After embedding, the hidden layer output vector is calculated through 24 cascaded transformer layers and a multi-head attention mechanism. h={H;h1,h2,h3,…,h m-1 ,h m } Among them, h m H represents the hidden state of the m-th judicial document sequence, and H represents the state set of the prompt structured information.

7. The method for extracting judicial document indicators according to claim 1, characterized in that, Step 5) involves retrieving the hidden layer vectors of the Unilm model and processing them into Gaussian embeddings, which includes: Step 5.1) Use a projection network f consisting of fully connected layers and nonlinear activation functions. u and f ∑ To generate Gaussian distribution parameters: u i =f u (h i ); ∑ i =TWO(f ∑ (h i ))+(1+∈); Where u i , ∑ i These represent the mean covariance and diagonal covariance of the Gaussian embedding, respectively, i.e., the non-zero elements are only along the diagonal of the matrix. u and f ∑ All layers consist of a single linear layer, using ReLU as the activation function, and for stability, e^(-14) is chosen; thus, the hidden layer output vector is processed into a Gaussian embedding (u i , ∑ i ).

8. The method for extracting judicial document indicators according to any one of claims 1-7, characterized in that, Step 6) involves calculating the contrastive learning loss of the Gaussian embedding and iteratively updating to obtain the final model, which includes the following two sub-steps: 6.1) and 6.2). Step 6.1) To calculate the contrast loss, consider the KL divergence among all valid token pairs in the sampled corpus. If two tokens x q and x p Having the same label y q =y p Then they are considered positive examples for their Gaussian embedding N(u p , ∑ p ) and N(u q , ∑ q ), calculate its KL divergence: Both directions of the KL divergence are calculated because it is not symmetrical: d(p,q)=1 / 2(D KL |N p ||N q |+D KL |N q ||N p |) For positive sample x p Calculate x p Gaussian embedding loss relative to other valid tokens in the batch: χ p ={(χ q ,and q )∈χ|y p / and q ,p≠q} Finally, the contrastive learning loss is calculated using KL divergence and Gaussian embedding loss: Step 6.2) Backpropagation is performed based on the contrastive learning loss function to iteratively update the parameters of the UniLM model, resulting in the final information extraction model that can be used for judicial document indicator extraction.

Images