Law document sensitive relation identification method based on global pointer

By adopting the GPLinker model based on global pointers in the judicial field, combined with the BERT model, combined with the knowledge graph embedding and enhanced pre-training method, the accuracy and robustness of the recognition of sensitive relationships in the legal document in the existing technology are solved, and more efficient recognition effect.

CN118917306APending Publication Date: 2024-11-08CHONGQING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411026092.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art methods for identifying sensitive relationships of legal documents in the judicial field use a simple BERT+ fully connected layer model, which performs poorly, especially in sparse training data and complex information environments, making it difficult to accurately identify semantic associations between entities.

Method used

The entity relationship joint extraction model based on global pointers is adopted, and the GPLinker model is combined with the BERT model, and the semantic information and knowledge graph embedding between entities is used to identify sensitive relationships of legal documents. The pre-pre-type of the model is enhanced by using R-Drop, EMA, FGM and other methods. The training process improves the generalization ability and robustness of the model.

Benefits of technology

It significantly improves the accuracy of identification of entity-sensitive relationships in legal documents, enhances the ability to model complex relationship patterns, reduces the risk of overfitting, and improves the robustness and efficiency of identification results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917306A_ABST
    Figure CN118917306A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of text processing, and particularly relates to a legal instrument sensitive relation recognition method based on a global pointer, which comprises the following steps: performing sentence segmentation processing on legal instrument data, and performing entity relation extraction on each text to obtain entity relation data; preprocessing and enhancing the entity relation data to obtain enhanced data; processing the enhanced data by adopting a data loader to obtain standard data for pre-training the BERT model; taking BERT model parameters as initial parameters of a GPLinker model, and adopting pre-trained BERT model coding standard data to train the GPLinker model; constructing a legal document sensitive relation identification model by using the BERT model and the GPLinker model; carrying out five-fold cross validation on the legal document sensitive relation identification model to obtain an optimal model, and carrying out identification by adopting the optimal model; according to the method, the entity relationship can be automatically and accurately extracted from a large number of legal instruments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of text processing, and in particular relates to a method for identifying sensitive relationships in legal documents based on a global pointer. Background Art

[0002] In recent years, with the rapid development of the Internet and the enhancement of people's legal awareness, the number of litigation cases has increased year by year, bringing greater work pressure to judicial staff. In judicial practice, processing a large number of legal documents is an important part of the daily work of courts and related institutions. However, due to the large number of legal documents and complex information, traditional manual processing methods are inefficient and prone to errors. At the same time, legal documents contain rich legal information. Effective extraction and analysis of relevant information can provide important decision-making support and information reference for judicial practitioners such as judges and lawyers. In particular, for the identification of sensitive relationships, accurate extraction and analysis of entity relationships in legal documents can help timely discover and deal with potential legal risks and problems.

[0003] However, the simple BERT+ fully connected layer model is currently commonly used in the judicial field to identify sensitive relationships in legal documents. This model performs poorly on sparse training data and does not consider the semantic associations between entities. It has a weak ability to identify sensitive relationships in legal documents with complex information. Therefore, the entity relationship joint extraction model based on the global pointer of the present invention effectively utilizes the semantic information between entities, greatly improves the accuracy of sensitive relationship identification results, and helps to improve the work efficiency of judicial personnel in handling documents. Summary of the invention

[0004] To solve the above problems, the present invention provides a method for identifying sensitive relationships of legal documents based on global pointers, comprising the following steps:

[0005] S1. Obtain legal document data and perform sentence segmentation to obtain a text dataset;

[0006] S2. Perform entity relationship extraction on each text in the text dataset to obtain corresponding entity relationship data;

[0007] S3. Preprocess and enhance all entity relationship data to obtain an enhanced data set;

[0008] S4. Use the data loader to process the enhanced dataset to obtain the standard dataset, use the standard dataset to pre-train the BERT model and save the parameters;

[0009] S5. Use the parameters obtained in step S4 as the initial parameters of the GPLinker model, use the pre-trained BERT model to encode the standard data set, and use the encoded standard data set to train the GPLinker model;

[0010] S6. The pre-trained BERT model and the trained GPLinker model are used to construct a legal document sensitive relationship recognition model;

[0011] S7. Use the standard data set to perform a five-fold cross validation on the legal document sensitive relationship recognition model to obtain the optimal legal document sensitive relationship recognition model;

[0012] S8. Obtain the document data to be identified and input it into the optimal legal document sensitive relationship identification model, and output the identification result.

[0013] Beneficial effects of the present invention:

[0014] Accurate identification of entity-sensitive relationships: Traditional methods for identifying sensitive relationships in legal documents often rely on word-level representations and ignore the semantic information of entities and the relationships between entities. The present invention uses the GPLinker model to deeply understand the entity semantic information and the relationships between entities, and construct a knowledge graph. It can more accurately capture the complex semantic associations between entities and is more accurate in understanding and inferring entity relationships in legal documents.

[0015] Learning more complex relationship patterns: Traditional methods for identifying sensitive relationships in legal documents are often limited by training data and find it difficult to capture the detailed and diverse relationship types between entities in legal documents. This model significantly improves the ability to model complex relationships by introducing knowledge graph embedding of entities.

[0016] Higher robustness: Traditional methods for identifying sensitive relationships in legal documents often use a simple BERT model for pre-training. This model makes full use of methods such as R-Drop, EMA, and FGM to assist in the BERT model pre-training process, effectively enhancing the generalization ability and robustness of the model and effectively reducing the risk of overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the method of the present invention;

[0018] Figure 2 It is a schematic diagram of the overall structure of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The present invention provides a method for identifying sensitive relations of legal documents based on global pointers, such as Figure 1 , Figure 2 As shown, the following steps are included:

[0021] S1. Obtain legal document data and perform sentence segmentation to obtain a text dataset.

[0022] Specifically, a variety of legal documents are obtained from the China Judgments Online website, each legal document is processed into sentences to obtain multiple texts, and all texts are combined into a text dataset.

[0023] S2. Perform entity relationship extraction on each text in the text dataset to obtain the corresponding entity relationship data.

[0024] Specifically, step S2 performs entity relationship extraction on any text, including:

[0025] S21. Determine the subject and object of the text, and regard the subject and object as the first entity and the second entity respectively; more specifically, the entity in the present invention refers to a person;

[0026] S22. confirming the entity pair relationship between the first entity and the second entity according to the text semantics; the entity pair relationship includes relationships such as friends, father and son, colleagues, etc.;

[0027] S23. Obtain the positions of the first entity and the second entity in the text, and record them as the first entity text position and the second entity text position;

[0028] S24. Concatenate text content, first entity content, first entity text position, second entity content, second entity text position, and entity pair relationship in order to form an entity relationship data.

[0029] Specifically, if a text is “(3) Testimony of witness A: I took over a new community construction project from a friend named B.”, the first entity is “A”, the second entity is “B”, and the entity pair relationship is “friend”; each Chinese character is regarded as a symbol, and the first entity text position refers to the sequence number of the first entity’s beginning symbol and the next symbol after the end in the text to which it belongs. In this example, the first entity text position is [5,8], and the second entity text position is [16,18].

[0030] S3. Preprocess and enhance all entity relationship data to obtain an enhanced dataset.

[0031] Specifically, step S3 performs preprocessing and enhancement processing on all entity relationship data, including:

[0032] S31. For each entity relationship data, if there is a blank field in its entity field, the entity relationship data is deleted; the entity field includes a first entity field for recording the first entity content, a first position field for recording the first entity text position, a second entity field for recording the second entity content, a second position field for recording the second entity text position, and a relationship field for recording the entity pair relationship;

[0033] The sensitive relationship identification of legal documents mentioned in the present invention mainly refers to identifying people related to the suspect and the relationship between the person and the suspect; if no entity and entity relationship appear in a text, there will be blank fields in the entity fields in the entity relationship data formed according to the text, which is meaningless for sensitive relationship identification and therefore needs to be deleted.

[0034] S32. Count the text length of each entity relationship data retained, and determine the maximum sequence length based on the statistical results;

[0035] Specifically, the text length of all retained entity relationship data is counted, and a maximum sequence length is determined according to the text length distribution, while ensuring that the maximum sequence length does not exceed the maximum length that the BERT model can handle, which is 512. In the embodiment of the present invention, 5186 entity relationship data are used, and statistics show that the text lengths of 5099 entity relationship data are less than 256, so the maximum sequence length is set to 256.

[0036] S33. For each retained entity relationship data, the maximum length word segmentation method is used to process its text content to obtain an original sample, and the entity field of the entity relationship data is used as the label of the original sample;

[0037] S34. For each original sample, forward translate the corresponding text content into English text through the text translation model, and then reverse translate the English text into Chinese through the same text translation model to obtain new text content; use the maximum length word segmentation method to process the new text content to obtain a new sample, and use the label of the original sample as the label of the new sample;

[0038] S35. All original samples and all new samples together constitute the enhanced dataset.

[0039] Specifically, using the maximum length word segmentation method to process any text content includes:

[0040] S311. Use a word segmenter (such as WordPiece or Byte Pair Encoding) to segment the text content to obtain a preliminary sequence;

[0041] S312. If the length of the preliminary sequence is greater than the maximum sequence length, the preliminary sequence is truncated at the end; if the length of the preliminary sequence is less than the maximum sequence length, the preliminary sequence is padded, that is, a zero value is padded at the end of the preliminary sequence to ensure that all samples obtained are of the same length.

[0042] S4. Use the data loader to process the enhanced dataset to obtain the standard dataset, use the standard dataset to pre-train the BERT model and save the parameters.

[0043] Specifically, step S4 uses a standard data set to pre-train the BERT model, including:

[0044] S41. For each standard data in the standard data set, a plurality of words are randomly selected from the standard data for MASK operation to obtain a MASK text data, and each MASK position after the MASK operation and the real vocabulary of the MASK position are recorded;

[0045] S42. Input the MASK text data into the BERT model and output the predicted word at the MASK position;

[0046] S43. The dynamic label smoothed cross entropy loss function is used to calculate the loss value between the predicted word and the real word at the MASK position, and backpropagation is performed according to the loss value to update the parameters of the BERT model until the parameters converge.

[0047] Specifically, for each MASK position in each MASK text data, the pre-trained BERT model uses the previous n words as context to obtain the predicted word of the MASK position. The process of the prediction task can be expressed as:

[0048]

[0049] Among them, w i represents the predicted word of the ith MASK position in the MASK text data, M represents the number of words in the corpus, ω i represents the real word at the ith MASK position in the MASK text data, ω j represents the word at the i-th position in the corpus, C 1:n represents the first n words of the ith MASK position in the MASK text data, f(ω i ) represents the ω in the corpus i The frequency of occurrence, f(C 1:n ,ω j ) represents the corpus in a given context C 1:n In case of j The frequency of the i-th MASK position in the MASK text data, f(C 1:n ) represents the context C in the corpus1:n The frequency of occurrence,

[0050] Specifically, the dynamic label smoothed cross entropy loss function SmoothedCELoss is expressed as:

[0051]

[0052] Where N represents the number of samples, C represents the number of categories, represents the true label of the i-th sample belonging to the c-th category after smoothing. represents the probability that the i-th sample belongs to the c-th category; represents the true label of the i-th sample belonging to the c-th category before smoothing, ∈ represents the dynamic smoothing parameter, C ntc Indicates the number of samples in the cth category.

[0053] Specifically, during the BERT model training process, the strategy of preheating the learning rate and gradually increasing the learning rate is adopted, which is expressed as:

[0054]

[0055] Among them, learning rate Represents the learning rate, lr max represents the maximum learning rate, step represents the current number of training steps, warmup steps Indicates the preset number of learning rate warmup steps.

[0056] Specifically, after each parameter update, the Fast Gradient Method (FGM) is used to calculate the gradient of the dynamic label smoothed cross entropy loss function with respect to the model input, and perturbations are added along the gradient direction for adversarial training, which is specifically expressed as:

[0057]

[0058] Among them, β represents the perturbation parameter, SmoothedCELoss represents the dynamic label smoothed cross entropy loss function, and input represents the model input, that is, the input training sample; represents the gradient of the dynamic label smoothed cross entropy loss function with respect to the model input, perturbation represents the model input after adding perturbation, sign(·) represents the sign function,

[0059] At the same time, the Exponential Moving Average (EMA) method is used to update the average value of the model parameters to smooth the update of the model parameters and reduce the jitter during the training process, which is expressed as:

[0060] θ′ EMA =α·θ EMA +(1-α)θ

[0061] Among them, θ′ EMA represents the average value of the updated model parameters (i.e. the average value of the model parameters in the current training round), θ EMA represents the average value of the model parameters before updating (i.e., the average value of the model parameters in the previous training round), α represents the smoothing coefficient, and θ represents the model parameters in the current training round.

[0062] S5. Use the parameters obtained in step S4 as the initial parameters of the GPLiner model, use the pre-trained BERT model to encode the standard data set, and use the encoded standard data set to train the GPLiner model.

[0063] Specifically, the GPLinker model takes the encoded standard data as input and entity relationship triples as output; the sparse multi-label cross entropy loss function is used to calculate the loss value during training, which is expressed as:

[0064]

[0065] Where P represents the positive category set, N represents the negative category set, and for each training sample, the first entity, the second entity, and the entity pair relationship all belong to the positive category, and the rest belong to the negative category, A = P∪Q; S z It represents the score of the sample on the category (also known as the element in the positive or negative category) z. The positive category set includes multiple elements. The score of the sample on each element in the positive category set is calculated and summed to get the positive category loss. Similarly, the negative category set also includes multiple elements. The score of the sample on each element in the negative category set is calculated and summed to get the negative category loss, and finally the loss value is obtained.

[0066] S6. The pre-trained BERT model and the trained GPLinker model are used to construct a legal document sensitive relationship recognition model.

[0067] S7. A standard data set was used to perform a five-fold cross-validation on the legal document sensitive relationship recognition model to obtain the optimal legal document sensitive relationship recognition model.

[0068] S8. Obtain the document data to be identified and input it into the optimal legal document sensitive relationship identification model, and output the identification result.

[0069] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0070] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for identifying sensitive relations of legal documents based on global pointers, characterized in that: The following steps are involved: S1. Obtain legal document data and perform sentence segmentation to obtain a text dataset; S2. Perform entity relationship extraction on each text in the text dataset to obtain corresponding entity relationship data; S3. Preprocess and enhance all entity relationship data to obtain an enhanced data set; S4. Use the data loader to process the enhanced dataset to obtain the standard dataset, use the standard dataset to pre-train the BERT model and save the parameters; S5. Use the parameters obtained in step S4 as the initial parameters of the GPLinker model, use the pre-trained BERT model to encode the standard data set, and use the encoded standard data set to train the GPLinker model; S6. The pre-trained BERT model and the trained GPLinker model are used to construct a legal document sensitive relationship recognition model; S7. Use the standard data set to perform a five-fold cross validation on the legal document sensitive relationship recognition model to obtain the optimal legal document sensitive relationship recognition model; S8. Obtain the document data to be identified and input it into the optimal legal document sensitive relationship identification model, and output the identification result.

2. According to the method for identifying sensitive relationships of legal documents based on global pointers in claim 1, it is characterized in that: Step S2 performs entity relationship extraction on any text, including: S21. Determine the subject and object of the text, and regard the subject and object as the first entity and the second entity respectively; S22. Confirming the entity pair relationship between the first entity and the second entity according to the text semantics; S23. Obtain the positions of the first entity and the second entity in the text, and record them as the first entity text position and the second entity text position; S24. Concatenate text content, first entity content, first entity text position, second entity content, second entity text position, and entity pair relationship in order to form an entity relationship data.

3. According to claim 2, a legal document sensitive relationship identification method based on global pointers is characterized in that: Step S3 performs preprocessing and enhancement processing on all entity relationship data, including: S31. For each entity relationship data, if there is a blank field in its entity field, the entity relationship data is deleted; the entity field includes a first entity field for recording the first entity content, a first position field for recording the first entity text position, a second entity field for recording the second entity content, a second position field for recording the second entity text position, and a relationship field for recording the entity pair relationship; S32. Count the text length of each entity relationship data retained, and determine the maximum sequence length based on the statistical results; S33. For each retained entity relationship data, the maximum length word segmentation method is used to process its text content to obtain an original sample, and the entity field of the entity relationship data is used as the label of the original sample; S34. For each original sample, forward translate the corresponding text content into English text through the text translation model, and then reverse translate the English text into Chinese through the same text translation model to obtain new text content; use the maximum length word segmentation method to process the new text content to obtain a new sample, and use the label of the original sample as the label of the new sample; S35. All original samples and all new samples together constitute the enhanced dataset.

4. According to claim 3, a legal document sensitive relationship identification method based on global pointers is characterized in that: Using the maximum length word segmentation method to process any text content includes: S311. Use a word segmenter to segment the text content to obtain a preliminary sequence; S312. If the length of the preliminary sequence is greater than the maximum sequence length, a tail truncation operation is performed on the preliminary sequence; if the length of the preliminary sequence is less than the maximum sequence length, a padding operation is performed on the preliminary sequence.

5. According to claim 1, a legal document sensitive relationship identification method based on global pointer is characterized in that: Step S4 uses a standard data set to pre-train the BERT model, including: S41. For each standard data in the standard data set, a plurality of words are randomly selected in the standard data for MASK operation to obtain a MASK text data, and the MASK position and the real vocabulary of the MASK position are recorded; S42. Input the MASK text data into the BERT model and output the predicted word at the MASK position; S43. The dynamic label smoothed cross entropy loss function is used to calculate the loss value between the predicted word and the real word at the MASK position, and backpropagation is performed according to the loss value to update the parameters of the BERT model until the parameters converge.

6. According to claim 5, a legal document sensitive relationship identification method based on global pointers is characterized in that: The dynamic label smoothed cross entropy loss function SmoothedCELoss is expressed as: Where N represents the number of samples, C represents the number of categories, represents the true label of the i-th sample belonging to the c-th category after smoothing. represents the probability that the i-th sample belongs to the c-th category; represents the true label of the i-th sample belonging to the c-th category before smoothing, ∈ represents the dynamic smoothing parameter, C ntc Indicates the number of samples in the cth category.

7. According to claim 5, a legal document sensitive relationship identification method based on global pointers is characterized in that: During the BERT model training process, the strategy of warming up the learning rate and gradually increasing the learning rate is adopted, which is expressed as: Among them, learning rate Represents the learning rate, lr max represents the maximum learning rate, step represents the current number of training steps, warmup steps Indicates the preset number of learning rate warmup steps.

8. According to claim 5, a legal document sensitive relationship identification method based on global pointers is characterized in that: After each parameter update, the fast gradient method is used to calculate the gradient of the dynamic label smoothed cross entropy loss function with respect to the model input, and a perturbation is added along the gradient direction, expressed as: Among them, β represents the perturbation parameter, SmoothedCELoss represents the dynamic label smoothed cross entropy loss function, input represents the model input, represents the gradient of the dynamic label smoothed cross entropy loss function with respect to the model input, perturbation represents the model input after adding perturbation, sign(·) represents the sign function, At the same time, the exponential sliding average method is used to update the average value of the model parameters, expressed as: θ′ EMA =a·θ EMA +(1-α)θ Among them, θ′ EMA represents the average value of the updated model parameters, θ EMA represents the average value of the model parameters before updating, α represents the smoothing coefficient, and θ represents the model parameters.

9. According to claim 1, a legal document sensitive relationship identification method based on global pointer is characterized in that: The GPLinker model takes the encoded standard data as input and entity relationship triples as output. During the training process, the sparse multi-label cross entropy loss function is used to calculate the loss value, which is expressed as: Among them, P represents the positive category set, Q represents the negative category set, A=P∪Q; S z It represents the score of the representative sample in category z.

Citation Information

Cited By

  • Intelligent legal document generation method and system based on large model

    CN119669485A

  • Medical text privacy information extraction and encryption method and system, terminal and medium

    CN120632126A