An intelligent method for extracting administrative law enforcement case information and identifying the cause of action
Through the DGCNN-based information extraction method, the problem of low efficiency and low accuracy of information extraction of administrative penalty decisions in the judicial field is solved, and short text information suitable for downstream NLP tasks is generated, which improves the applicability and universality of the extraction results and saves human Resources.
Patent Information
- Application Number
- CN202210171624.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-24
- Publication Date
- 2024-11-01
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The information extraction efficiency and accuracy of the existing administrative penalty decision letter in the judicial field are low, and the extraction results are not applicable to downstream NLP subtasks, which cannot effectively solve problems such as overlapping relationships and poor universality when data changes.
Using DGCNN-based information extraction method, the long text of the administrative penalty decision is extracted into input. Through corpus conversion, preprocessing and vectorization, combined with BERT and UniLM models, a short text format that conforms to mainstream NLP tasks is generated to realize entity recognition, Relationship extraction and case incident extraction, and result combination is carried out to improve the utilization and applicability of information.
It improves the accuracy and efficiency of information extraction, reduces computing resource consumption, provides short text information suitable for downstream task analysis, enhances the universality and applicability of the extraction results, and saves human resources.
Smart Images

Figure CN114547230B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an intelligent administrative law enforcement case information extraction and case cause identification method, belonging to the technical field of legal artificial intelligence. Background Art
[0002] With the continuous development of my country's legal system, legal documents have become more and more perfect. The format and content of legal documents have been more and more detailed, and the theoretical research work of legal documents has also made great progress. Therefore, legal documents are more standardized and complete. With the continuous advancement of information technology, there are a large number of administrative law enforcement document resources on the Internet. Administrative law enforcement documents, as an important carrier of legal judgments and information, contain a large amount of available information. If this information can be correctly extracted and used, it will save a lot of human resources and provide reliable help for new law enforcement judgments.
[0003] At the same time, with the continuous development of deep learning, the information extraction model that combines statistical learning and deep learning has received widespread attention. Usually, early information extraction regards entity extraction and relationship extraction as serial tasks. Such serial models are relatively simpler in modeling, but treating entity recognition and relationship extraction as two independent tasks will obviously have a series of problems: the correlation between the two subtasks is not taken into account in the solution process of the two tasks, which leads to the result of the relationship extraction task being heavily dependent on the result of the entity extraction, resulting in the problem of error accumulation. For one-to-many problems, that is, the problem of relationship overlap, the serial model cannot provide a better solution. Therefore, in recent years, many works have considered jointly modeling entity recognition and relationship extraction tasks. This end-to-end model intuitively has better results.
[0004] In order to strengthen the interaction between the two sub-models, some joint decoding algorithms have been proposed: In 2014, Li Qi et al. proposed to regard entity relationship extraction as a structured prediction problem, adopted the structured perceptron algorithm, designed global features, and used beam search for approximate joint decoding. In 2016, Arzoo Katiyar et al. proposed to use conditional random fields (CRF) to simultaneously model entity and relationship models, and obtained the output results of entities and relationships through the Viterbi decoding algorithm. In 2017, Zhang Meishan et al. proposed a decoding algorithm using global normalization. In 2017, Zheng Suncong et al. of the Institute of Automation designed a transition system for entity relationship extraction, thereby realizing joint entity relationship extraction. However, the above unified entity and relationship annotation framework cannot completely solve problems such as relationship overlap, so in specific scenarios, some post-processing rules need to be introduced for constraints. And because most of the above algorithms use sequence models, the training time is slow.
[0005] At present, when conducting text analysis on administrative law enforcement documents, it is often limited by the unstructured characteristics of the data, large information capacity, complex content representation, and diversified information field representation. These problems have put forward higher requirements for the information extraction and structuring of administrative law enforcement documents. For the information extraction of administrative law enforcement documents, Ge Jidong et al. from Nanjing University in 2018 proposed a segmented feature rule model, which treats the documents as logical segments and designs rules for them. In 2019, Dai Guangyu et al. proposed a legal document information extraction model that matches the key word set with the keyword set to improve the efficiency of users in obtaining information from legal documents. In 2020, Xi Lina et al. proposed a rule-based feature tree method for target blocks, which can automatically extract the basic elements in administrative law enforcement documents. In 2020, Bai Xiongwen et al. proposed a legal judgment information extraction method based on a sequence-to-sequence neural network with a self-attention mechanism, which can improve the extraction efficiency and accuracy of legal elements. In 2020, Li Dan et al. proposed a legal document information extraction method based on a combination of rules and models, which effectively combined the methods of rules and models to form a complementary relationship, improved the effect of legal document information extraction, and at the same time improved scalability and portability. These methods are all rule-based, simple and convenient, and can meet the accuracy requirements. However, they often need to change and add new regular statements when facing data changes, which means that they are less versatile. In addition, the content format of administrative law enforcement documents in different regions may be different, resulting in a high degree of restriction in regularization-based methods.
[0006] In recent years, natural language processing has been widely used in the judicial field, and the field of legal artificial intelligence has attracted much attention. Artificial intelligence technology can greatly improve the efficiency and accuracy of information extraction, bringing convenience to practitioners. However, simple methods based on deep learning or machine learning are affected by text length, contextual information and other aspects, and the effect needs to be improved. In addition, the extracted results are often not applicable to the analysis of all downstream tasks, because only some entities and relationships are extracted and cannot be directly applied to downstream tasks. Therefore, the information extraction accuracy of the statistical learning field model is low, and the unified deep learning extraction model cannot completely solve the problem of relationship overlap and slow speed. As well as the common problem that the extraction results are not applicable to all downstream natural language processing tasks, it is necessary to propose a new model suitable for legal document information extraction. Summary of the invention
[0007] In view of the shortcomings of the prior art, the present invention provides a high-accuracy and high-utilization information extraction and case cause identification method based on DGCNN.
[0008] In order to solve the problems existing in the existing judicial field, such as low efficiency and accuracy in information extraction of administrative penalty decisions, and the inability to further use the extracted entities, the present invention proposes an information extraction method for administrative penalty decisions, which extracts the long text of the document as input and further generates a short text format that meets the requirements of current mainstream NLP tasks. Summary of the invention:
[0010] The purpose of the present invention is to solve the problem that the accuracy of administrative penalty document information extraction in the existing judicial field is not high and the extraction results are not suitable for downstream NLP subtasks. A method for extracting administrative law enforcement document information and identifying case causes with high accuracy and high utilization is provided.
[0011] The present invention mainly includes five parts: named entity recognition of administrative law enforcement documents, extraction of legal related terms in administrative law enforcement documents, extraction of case character relationships in administrative law enforcement documents, extraction of case events in administrative law enforcement documents, and generation of short texts by combining the extraction results. Named entity recognition of administrative law enforcement documents is to study how to identify and classify all entities existing in administrative law enforcement documents, such as time, place, and case-related subjects and objects. This study aims to improve the named entity extraction effect; legal related term extraction is to study how to extract professional expressions in the legal field in the documents, such as relevant legal clauses, law enforcement judgments, etc. The task aims to study how to improve the recognition effect by integrating with the legal knowledge base; case character relationships are key feature elements in documents. This content mainly studies how to improve the effect of downstream decision-making assistance tasks by extracting character entities, attributes, and relationships in documents. Case event extraction studies how to extract text descriptions of key events in the case, that is, to find core information from the case. The purpose of combining the extraction results to generate short text is to further process the results of information extraction and generate short text that is suitable for various current NLP methods and can be analyzed without losing key information. This step greatly enhances the role of the extracted information and provides a good foundation for downstream tasks.
[0012] Terminology explanation:
[0013] 1. Jieba: Jieba is an excellent third-party Chinese word segmentation library in Python. Jieba supports three word segmentation modes: precise mode, full mode and search engine mode.
[0014] 2. Administrative Penalty Document: An administrative penalty decision is a written legal document with legal force that is prepared by the administrative authority based on the illegal evidence obtained through investigation and evidence collection in response to the illegal behavior of the party concerned. It records the illegal facts of the party concerned, the reasons, basis and decision for the penalty, etc.
[0015] 3. Information extraction: extracting specific events or factual information from natural language texts to help us automatically classify, extract and reconstruct massive amounts of content.
[0016] 4. DGCNN (Dilate Gated Convolutional Neural Network): It is a "dilated gate convolutional neural network" that combines two relatively new convolution methods: dilated convolution and gated convolution, and adds some artificial features and tricks, which ultimately makes the model achieve the best effect on the basis of lightness and speed.
[0017] 5. ROUGE (Recall-Oriented Understudy for Gisting Evaluation): It is a set of indicators for evaluating automatic summarization and machine translation. It compares the automatically generated summary or translation with a set of reference summaries (usually manually generated), and counts the number of overlapping basic units (n-grams, word sequences, and word pairs) between the two to evaluate the quality of the summary and obtain a corresponding score to measure the similarity between the automatically generated summary or translation and the reference summary.
[0018] 6. Tokenizer: It is a commonly used toolkit in tensorflow. With it, we can preprocess, serialize, vectorize, etc. Tokenizer converts each text into a sequence of integers (each integer is the index of a token in the dictionary) or into a vector (where the coefficient of each token can be binary) based on vectorized corpus, word count, TF-IDF, etc.
[0019] 7. Chinese_roberta_wwm_ext_L-12_H-768_A-12: This is an upgraded version of BERT released by HIT and iFLYTEK Lab on May 31, 2019 for Google. It mainly changes the training sample generation strategy in the original pre-training stage. The original WordPiece-based word segmentation method will divide a complete word into several subwords. When generating training samples, these separated subwords will be randomly masked. The whole word mask is used here, that is, if part of the WordPiece subwords of a complete word are masked, the other parts of the same word will also be masked.
[0020] 8. Early Stopping: It is a common method to control overfitting. When training deep learning neural networks, we usually hope to obtain the best generalization performance (i.e., the ability to fit the data well). However, all standard deep learning neural network structures such as fully connected multi-layer perceptrons are prone to overfitting: when the network performs better and better on the training set and the error rate is getting lower and lower, in fact, at some point, its performance on the test set has begun to deteriorate. Early Stopping can effectively avoid the above situation.
[0021] The technical solution of the present invention is as follows:
[0022] An intelligent administrative law enforcement case information extraction and case cause identification method includes the following steps:
[0023] A. Construction of the data set: crawl the administrative penalty decision, extract the text content, and annotate the short text in the text content;
[0024] B. Corpus conversion:
[0025] Divide all long sentences in administrative law enforcement documents into short sentences according to punctuation marks;
[0026] Segment the short text annotated in the dataset in step A according to punctuation marks;
[0027] Find the longest sentence in the short text, use this longest sentence as a standard, find sentences similar to this longest sentence in the administrative penalty document, extract and save them, and execute recursively until all sentences in the short text have been executed once;
[0028] C. Extract data preprocessing and vectorization:
[0029] Step B extracts sentences similar to the label, i.e., the short text, from the administrative law enforcement documents;
[0030] Filter the extracted text and convert it into a sequence of sentences separated by spaces; encode and index the sentence sequence;
[0031] After performing global average pooling on the obtained text data, the BERT pre-trained Chinese model with full-word mask is introduced to supplement the average pooling and convert the segmented sentence sequence into a sentence vector.
[0032] D. Training of the DGCNN-based extraction model:
[0033] The sentence vectors extracted from the administrative law enforcement documents in step B and the labels, i.e., the short texts, are used as labels, and the sentence vectors generated from the entire text of the administrative law enforcement documents are used as input to train the extraction model;
[0034] E. Short text generation based on UniLM:
[0035] The key sentences of administrative law enforcement documents extracted by the trained extraction model are used as input, and the manually annotated short texts are used as labels to train the generation model;
[0036] F. Input the law enforcement document text for prediction:
[0037] After the administrative law enforcement documents that need to be converted are processed through steps A to C, the processing results are input into the trained extraction model to generate and save the extraction results. The extraction results are further input into the trained generation model to obtain short texts suitable for analysis of other downstream tasks, and finally the extraction results and short texts suitable for analysis of other downstream tasks are obtained.
[0038] Further preferably, in step A, marking the text content means: extracting all required key fields from the administrative law enforcement document, and recombining them into a new short text that only includes key purpose information based on semantics; and marking these short texts in the administrative penalty document.
[0039] According to the preferred embodiment of the present invention, in step B,
[0040] Split all long sentences in administrative law enforcement documents into short sentences according to punctuation marks. Specifically, use jieba word segmentation to split the text of administrative law enforcement documents into sentence formats according to punctuation marks, and save them as elements in a list.
[0041] The short text annotated in the data set of step A is divided into sentences according to punctuation marks, specifically: the annotated short text is divided into short sentence formats using jieba word segmentation and saved in array format.
[0042] Extract the longest short sentence in the array, match it with the administrative law enforcement documents in the list, extract the sentence in the administrative law enforcement documents in the list that is most similar to the longest short sentence and save it, and perform this step recursively until all the sentences in the array are matched.
[0043] Further preferably, the sentence most similar to the longest short sentence in the administrative law enforcement documents in the list is extracted, and the specific implementation process is as follows:
[0044] Assume x1 is a labeled short text sequence X = [x1, x2, ... x n ] is a sentence in m It is the administrative law enforcement document sequence Y = [y1, y2, ...y m ], then Flcs is as shown in formula (1):
[0045]
[0046] In formula (1), R LCS It refers to the recall rate, which is the ratio of the number of identical characters in the extracted text and the short text to the number of characters in the manual label, i.e. the short text; P LCS It refers to the accuracy, which is the ratio of the number of identical characters in the extracted text and the short text to the total number of extracted characters; F LCS It is ROUGE-L;
[0047] For a sentence x in a short text sequence X that will be labeled n All sentences in the sequence of administrative law enforcement documents y j ,j=1,2...m;calculate F LCS , take the value that makes F LCSThe sentence y in the administrative law enforcement document sequence with the highest score k It is saved as the most similar sentence, and this step is performed recursively until all sentences in the labeled short text sequence X are matched. The extraction result is saved as R, which is the most similar sentence.
[0048] Preferably, according to the present invention, the specific implementation process of step C is as follows:
[0049] First, use the Tokenizer toolkit in tensorflow to remove punctuation and line breaks from the text for text preprocessing.
[0050] Then, use the Fit_on_texts method of the Tokenizer toolkit to learn the dictionary of the text. Word_index is the mapping relationship between the corresponding word and the number Dict. Through this Dict, each word in each sentence is converted into a number, that is, Texts_to_sequences;
[0051] Again, pad them to the same length using the Padding method; then use the embedding layer in Keras to perform a vectorization;
[0052] Finally, the BERT pre-trained Chinese model with full-word mask, namely the Chinese_roberta_wwm_ext_L-12_H-768_A-12 pre-trained model, is introduced to supplement the average pooling.
[0053] Preferably, according to the present invention, the specific implementation process of step D is as follows:
[0054] The sentence vector corresponding to the administrative law enforcement document text obtained in step C is used as the input of the extraction model. The features are extracted through a layer of DGCNN. The extracted features are input into the Attention layer to complete the integration of sequence information, including encoding the sentence vector sequence of the administrative law enforcement document into a total administrative law enforcement document text vector, and encoding the sentence vector of the annotated short text into a total annotated short text vector. The Attention layer is shown in formula (2):
[0055]
[0056] In formula (2), α, W are both trainable parameters, and Act() is the activation function, taking tanch; x i is the sequence before encoding, x is the vector after encoding, λ i is the weight during calculation, softmax i () is the activation function, also known as the normalized exponential function;
[0057] Afterwards, the sentence vector X corresponding to the total administrative law enforcement document text obtained in step C is used as input, and the extracted most similar sentence vector Y is used as the corresponding label, which is input into a 5-layer DGCNN and then connected to a fully connected layer; after activation with the Sigmoid activation function, it is compared with the specified threshold, and the sentence vectors greater than the threshold are saved as a list, and the vectors less than the threshold are discarded, and then the sentence vector is decoded into the original text to finally obtain the information extraction result.
[0058] Further preferably, DGCNN is a dilation gate convolutional neural network, which is constructed as follows: First, add a normal one-dimensional convolution
[0059] The formula is shown in formula (3):
[0060]
[0061] In formula (3), X represents the input sequence, Y represents the output sequence, Conv1D1 and Conv1D2 are two one-dimensional convolutions;
[0062] Next, use dilated convolution; specifically, the input X is first passed through a fully connected layer to extract features, then input into a dilated gate convolution layer with a dilation rate of 1, and then the output is input into a dilated gate convolution layer with a dilation rate of 2; then connect a dilated gate convolution layer with a dilation rate of 4 and a dilated gate convolution layer with a dilation rate of 8; finally, after connecting two dilated gate convolution layers with a dilation rate of 1, the result Y is output after passing through a sigmoid activated fully connected layer.
[0063] Preferably, according to the present invention, the specific implementation process of step E is as follows:
[0064] a. Group the administrative law enforcement document data Y and the corresponding annotated short text X in step B into n pieces. The grouping standard is: n-1 pieces of data are used as training sets to train the extraction model in step D. In the prediction stage, the untrained data is used as input to predict and generate short texts. This step is repeated n times until all data are trained and used as input for prediction. In this way, the extracted short texts of all administrative law enforcement documents are obtained, which are recorded as C.
[0065] b. Input the grouped data into the UniLM network for training:
[0066] First, the extracted short text sequence C is packaged, and the result is P 0 =[x1,x2,...x x ];
[0067] Then, the above packaging result P 0 Input a K-layer Transformer, that is, Pk =Transformer k (P k-1 ),k∈[1,K]; In this way, P passes through K layers of Transformer, 0 Encoded at different levels of abstraction k =[x1 k ,x2 k ,...x x k ] on the context vector representation;
[0068] For the k-th layer Transformer, its self-attention head A k The output calculation method is shown in formula (4):
[0069]
[0070] In formula (4), Q, K, and V are three vectors. is the trained weight matrix, k∈[1,K]; M is the mask matrix, M ij is the element of the mask matrix, indicating whether it is being paid attention to, 0 means being paid attention to, and negative infinity means being masked; d k represents the dimension of matrix K, A k and V k The subscript k in represents the kth layer.
[0071] In the decoding stage of the Seq2Seq model of the UniLM network, a sequence prediction task is added, that is, the distribution p(y t |y <t ,x) is changed to predict one more label, as shown in formula (5):
[0072] p(y t ,z t ∣y <t ,x)=p(y t ∣y <t ,x)p(z t ∣y <t ,x) (5)
[0073] In formula (5), z is the longest common subsequence of the annotated short text and the administrative law enforcement document text;
[0074] In the prediction phase, for each step, first predict the label z t , if z t If it does not appear in the text of the administrative law enforcement document, then there is no need to change it. tIf it appears in the text of administrative law enforcement documents and is the beginning field of a sentence, then in the distribution of tokens, all tokens that are not in the original text are masked. t If a token appears in the text of an administrative law enforcement document and is a continuous segment with the previous token, then all tokens that cannot form the corresponding n-gram in the original text are masked out in the distribution of the token.
[0075] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of an intelligent administrative law enforcement case information extraction and case cause identification method when executing the computer program.
[0076] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of an intelligent administrative law enforcement case information extraction and case cause identification method.
[0077] The beneficial effects of the present invention are:
[0078] 1. The present invention adopts DGCNN as the extraction model and utilizes its non-serialized neural network structure characteristics to extract information from text data, which greatly reduces the time and computing resources consumed and improves the accuracy of extraction.
[0079] 2. The method of generating sentence vectors by combining training data with BERT provided by the present invention reduces the dependence on the amount of data, and can also achieve good results when there is less labeled data.
[0080] 3. The information extraction idea provided by the present invention does not need to mark all the information to be extracted in the original text when constructing the training set. It only needs to provide the required short text information to realize the information extraction in the original text, thus saving human resources.
[0081] 4. The administrative law enforcement document information extraction method based on the combination of extraction and generation proposed in the present invention not only meets the traditional information extraction tasks but also provides practical short text information for other downstream tasks such as case analysis and recommendation of similar administrative law enforcement documents. BRIEF DESCRIPTION OF THE DRAWINGS
[0082] Figure 1 Schematic diagram of the model training process;
[0083] Figure 2 This is a schematic diagram of the construction of DGCNN;
[0084] Figure 3 Schematic diagram of the neural network architecture of the extraction model;
[0085] Figure 4 Schematic diagram of the Mask method of Seq2Seq in the present invention;
[0086] Figure 5 A schematic diagram of the Seq2Seq model of the UNILM neural network in the present invention;
[0087] Figure 6 This is a schematic diagram of the UniLM network architecture;
[0088] Figure 7 It is a flowchart of the intelligent administrative law enforcement case information extraction and case cause identification method; DETAILED DESCRIPTION
[0089] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.
[0090] Example 1
[0091] An intelligent administrative law enforcement case information extraction and case identification method, such as Figure 7 As shown, the steps include:
[0092] like Figure 1 As shown, the model training process includes steps A to E;
[0093] A. Construction of the data set: crawl administrative penalty decisions from the website, extract text content tagged as html, and annotate short texts in the text content;
[0094] B. Corpus conversion:
[0095] Divide all long sentences in administrative law enforcement documents into short sentences according to punctuation marks;
[0096] Segment the short text annotated in the dataset in step A according to punctuation marks;
[0097] Find the longest sentence in the short text, use this longest sentence as a standard, find sentences similar to this longest sentence in the administrative penalty document, extract and save them, and execute recursively until all sentences in the short text have been executed once;
[0098] C. Extract data preprocessing and vectorization:
[0099] Step B extracts sentences similar to the label, i.e., the short text, from the administrative law enforcement documents;
[0100] Filter the extracted text, that is, filter out all punctuation marks plus tabs and line breaks, convert the text into a sequence of sentences separated by spaces; encode the sentence sequence and index it;
[0101] After performing global average pooling on the obtained text data, the BERT pre-trained Chinese model with full-word mask from Harbin Institute of Technology is introduced to supplement the average pooling and convert the segmented sentence sequence into a sentence vector.
[0102] D. Training of the DGCNN-based extraction model:
[0103] The sentence vector generated in the above step C is equivalent to preprocessing. The sentence vector with the label, i.e., the short text, extracted from the administrative law enforcement document in step B is used as the label, and the sentence vector generated from the entire text of the administrative law enforcement document is used as input to train the extraction model. This step is the key to the extraction model, and its purpose is to train a model that can extract key knowledge from administrative law enforcement documents.
[0104] E. Short text generation based on UniLM:
[0105] In order to generate a text that best matches the short text in the label, it is necessary to combine the key contents of the extracted administrative law enforcement documents. The key sentences of the administrative law enforcement documents extracted by the trained extraction model are used as input, and the manually annotated short text is used as the label to train the generation model;
[0106] F. Input the law enforcement document text for prediction:
[0107] After the administrative law enforcement documents that need to be converted are processed through steps A to C, the processing results are input into the trained extraction model to generate and save the extraction results. The extraction results are further input into the trained generation model to obtain short texts suitable for analysis of other downstream tasks, and finally the extraction results and short texts suitable for analysis of other downstream tasks are obtained.
[0108] Example 2
[0109] The difference between the intelligent administrative law enforcement case information extraction and case cause identification method described in Example 1 is that:
[0110] Further preferably, in step A, marking the text content means: extracting all required key fields from the administrative law enforcement document, and recombining them into a new short text that only includes key purpose information based on semantics; and marking these short texts in the administrative penalty document.
[0111] Specifically, the administrative penalty decision was crawled from the administrative penalty document website, and the text content tagged as html was extracted. The text content was then annotated, specifically: the important information in the text was manually extracted, and the information was combined according to semantics, appropriate conjunctions were added, and the sentence order was modified. The annotation result was a short text that was fluent and semantically consistent. The specific information that appeared in the above annotated short text was also annotated in the original text. A total of 9,697 data were annotated here, in the format of {original administrative law enforcement document text, annotated short text}. The annotated data set was saved in json format and placed in the corresponding file directory. The specific information that appeared in the above annotated short text was also annotated in the original text. The purpose of this was to enable the model to input more accurate labels and improve the effect. However, in order to reduce labor costs, this step can also be discarded, and the final extraction effect will not be much different.
[0112] In step B, all long sentences in the administrative law enforcement document are segmented into short sentences according to punctuation marks, specifically: the jieba word segmentation is used to segment the administrative law enforcement document text into sentence format according to punctuation marks, and saved as elements in a list.
[0113] The short text annotated in the data set of step A is divided into sentences according to punctuation marks, specifically: the annotated short text is divided into short sentence formats using jieba word segmentation and saved in array format.
[0114] Extract the longest short sentence in the array, match it in the administrative law enforcement documents in the list, extract the sentence in the administrative law enforcement documents in the list that is most similar to the longest short sentence and save it, and perform this step recursively until all the sentences in the array are matched. Note that the standard for measuring whether the sentences are similar here is to combine the weighted Rouge in units of characters with the weighted Rouge in units of words to calculate. Because simply using words as units will reduce the weight of long words, resulting in longer words being less accurate in extraction, but using only words as units will reduce the accuracy of proper nouns. In order to balance the relationship between the two, the weighted Rouge weight in units of characters is finally selected as 0.7, and the weighted Rouge weight in units of words is selected as 0.3.
[0115] Extract the sentences in the administrative law enforcement documents in the list that are most similar to the longest short sentence. The specific implementation process is as follows:
[0116] ROUGE-L is a recall-based metric, where L refers to the longest common subsequence.
[0117] First, define the longest common subsequence (LCS): Assume that there exists X = [x1, x2, ... x n ] and Y=[y1,y2,...ym ], if there exists a strictly increasing sequence [i1,i2,...i k ] is the index of X, for each j=1,2...k, there is always x ij =y j , then Y is a subsequence of X; the maximum length common subsequence of sequence X and sequence Y is called LCS, denoted by LCS(X,Y). Here, the F-measure based on LCS is used to evaluate the similarity of two sentences X and Y; assuming that x1 is a labeled short text sequence X = [x1, x2, ...x n ] is a sentence in the Supreme People's Court's Several Provisions on Evidence in Civil Litigation. m It is the administrative law enforcement document sequence Y = [y1, y2, ...y m ] is a sentence in the Civil Procedure Law of the People's Republic of China; then Flcs is as shown in formula (1):
[0118]
[0119] In formula (1), R LCS It refers to the recall rate, which is the ratio of the number of identical characters in the extracted text and the short text to the number of characters in the manual label, i.e. the short text; P LCS It refers to the accuracy, which is the ratio of the number of identical characters in the extracted text and the short text to the total number of extracted characters; F LCS It is ROUGE-L; it is a method and evaluation index for calculating the overlap rate of long common subsequences. In summary, the similarity calculation result is Flcs = 0.41167*0.7 = 0.2882. The extracted vocabulary is retained.
[0120] For a sentence x in a short text sequence X that will be labeled n All sentences in the sequence of administrative law enforcement documents y j ,j=1,2...m;calculate F LCS , take the value that makes F LCS The sentence y in the administrative law enforcement document sequence with the highest score k This step is repeated recursively until all sentences in the labeled short text sequence X are matched, and the extraction result is saved as R, which is the most similar sentence. The evaluation mainly considers completeness, that is, the sufficiency and fidelity of the extraction results.
[0121] The specific implementation process of step C is as follows:
[0122] First, use the Tokenizer toolkit in tensorflow to remove punctuation and line breaks from the text for text preprocessing.
[0123] Since computers cannot understand the meaning of text when processing language and text, they need to convert a word (a single Chinese character or phrase is considered a word) into a positive integer, so a text needs to be converted into a sequence for computer processing. Then, the Fit_on_texts method of the Tokenizer toolkit is used to learn the dictionary of the text. Word_index is the mapping relationship between the corresponding word and number Dict. Through this Dict, each word in each sentence is converted into a number, that is, Texts_to_sequences;
[0124] Again, padding is used to make them of the same length; vectorization is performed using the embedding layer that comes with Keras; after vectorization, global average pooling is used to reduce the amount of data.
[0125] Finally, the BERT pre-trained Chinese model with full word mask, namely Chinese_roberta_wwm_ext_L-12_H-768_A-12 pre-trained model, is introduced to supplement average pooling. After vectorization, global average pooling is used to reduce the amount of data.
[0126] The specific implementation process of step D is as follows:
[0127] like Figure 3 As shown in Figure 2, the sentence vector corresponding to the administrative law enforcement document text obtained in step C is used as the input of the extraction model. The features are extracted through a layer of DGCNN, and the extracted features are input into the Attention layer to complete the integration of sequence information, including encoding the sentence vector sequence of the administrative law enforcement document into a total administrative law enforcement document text vector, and encoding the sentence vector of the annotated short text into a total annotated short text vector. The Attention layer is shown in formula (2):
[0128]
[0129] In formula (2), α, W are both trainable parameters, and Act() is the activation function, taking tanch; x i is the sequence before encoding, x is the vector after encoding, λ i is the weight during calculation, softmax i () is the activation function, also known as the normalized exponential function; it maps the input to a real number between 0 and 1, and normalizes it to ensure that the sum is 1.
[0130] Afterwards, the sentence vector X corresponding to the total administrative law enforcement document text obtained in step C is used as input, and the extracted most similar sentence vector Y is used as the corresponding label, which is input into a 5-layer DGCNN and then connected to a fully connected layer; after activation with the Sigmoid activation function, it is compared with the specified threshold, and the sentence vectors greater than the threshold are saved as a list, and the vectors less than the threshold are discarded, and then the sentence vector is decoded into the original text to finally obtain the information extraction result.
[0131] The specified threshold is used for EarlyStop, and the theoretical range should be between (0, 1). The setting standard is to hope that the model extraction results are more comprehensive, that is, when focusing on generalization effect, a lower value should be set, such as 0.2, 0.15, etc. When focusing on the precision of the extraction results, a higher value should be set, such as 0.3, 0.35, etc. The theoretical value range is (0, 1), but in practical applications, its value should not exceed 0.5.
[0132] Further preferably, DGCNN is a dilation gate convolutional neural network that combines a gating mechanism and a dilation mechanism. Figure 2 As shown, its
[0133] The construction method is as follows: First, add a gate to the ordinary one-dimensional convolution. The formula is shown in formula (3):
[0134]
[0135] In formula (3), X represents the input sequence, Y represents the output sequence, Conv1D1 and Conv1D2 are two one-dimensional convolutions; their forms are the same (for example, the number of convolution kernels and window size are the same), but the weights are not shared, that is, the parameters are doubled, σ means that Conv1D2 is activated with the Sigmoid function, and the other one is not activated. It means multiplying them bit by bit. Since the range of the Sigmoid function is (0, 1), a "valve" is added to each output of Conv1D to control the flow. One advantage of this is that the risk of gradient disappearance is lower, because there is a convolution without any activation function, and the gradient of this part of the convolution without the activation function is not easy to disappear.
[0136] Next, in order to enable the CNN model to capture farther distances without increasing model parameters, dilated convolution is needed; compared with ordinary convolution, in addition to the size of the convolution kernel, dilated convolution also has a dilation rate parameter, which is mainly used to indicate the size of the dilation. The similarity between dilated convolution and ordinary convolution is that the size of the convolution kernel is the same, that is, the number of parameters in the neural network remains unchanged, and the difference is that dilated convolution has a larger receptive field. That is, it can capture the connection between longer sequences without expanding the model parameters. Specifically, it means: first, the input X is extracted through a fully connected layer, and then input into a dilated gate convolution layer with a dilation rate of 1, and then the output is input into a dilated gate convolution layer with a dilation rate of 2; then, a dilated gate convolution layer with a dilation rate of 4 and a dilated gate convolution layer with a dilation rate of 8 are connected; finally, after connecting two dilated gate convolution layers with a dilation rate of 1, the result Y is output after passing through a sigmoid activated fully connected layer.
[0137] The specific implementation process of step E is as follows:
[0138] a. Before inputting the above extraction model results into the UniLM network, the data needs to be converted. Because in the training set, both the extraction and generation models will accept the label of the annotated short text. To solve the overfitting problem caused by this, the administrative law enforcement document data Y and the corresponding annotated short text X in step B are grouped into n pieces. The grouping standard is: n-1 pieces of data are used as training sets to train the extraction model in step D, and the untrained data is used as input to predict the generated short text in the prediction stage. This step is repeated n times until all data are trained and used as input for prediction. In this way, the extracted short text of all administrative law enforcement documents is obtained, denoted as C; and the consistency of the model in the training and prediction stages is improved as much as possible.
[0139] b. Input the grouped data into the UniLM network for training:
[0140] First, the extracted short text sequence C is packaged, and the result is P 0 =[x1,x2,...x x ];
[0141] Then, the above packaging result P 0 Input a K-layer Transformer, that is, P k =Transformer k (P k-1 ),k∈[1,K]; In this way, P passes through K layers of Transformer, 0 Encoded at different levels of abstraction k =[x1 k,x2 k ,...x x k ] on the context vector representation;
[0142] For the k-th layer Transformer, its self-attention head A k The output calculation method is shown in formula (4):
[0143]
[0144] In formula (4), Q, K, and V are three vectors. is the trained weight matrix, k∈[1,K]; M is the mask matrix, M ij is the element of the mask matrix, indicating whether it is being paid attention to, 0 means being paid attention to, and negative infinity means being masked; d k represents the dimension of matrix K, A k and V k The subscript k in represents the kth layer.
[0145] The UniLM network consists of 24 layers of Transformer, in which 1024 hidden sizes and 16 attention heads are set; the initialization generation model parameters used are Nezha_base, the Dropout ratio is 0.1, and the weight decay factor is 0.01;
[0146] The loss function is cross entropy loss, and the Seq2Seq method is used to generate short texts.
[0147] Use self-attention mask for pre-training, let S1 and S2 represent the extracted short text sequence and the labeled short text sequence respectively, and construct the input "[SOS]S1[EOS]S2[EOS]".
[0148] The fine-tuning of the generative model is done by randomly masking a certain proportion of the tokens in the target sequence and learning to recover the masked words.
[0149] The training goal is to maximize the likelihood of masked tokens based on the context. The training is set for 30 epochs, and the fine-tuned hyperparameters are the same as those during pre-training. The target token is masked at a rate of 0.7, the batch size is set to 8, the maximum length is 1024, and the label smoothing is set to 0.1. During training, the end marker [EOS] at the target end is also masked, so that the generative model can learn to predict and learn to end automatically, thus completing the training goal.
[0150] In order to ensure the fidelity and accuracy between the vocabulary in the generated short text and the professional vocabulary in the administrative law enforcement documents, a sequence prediction task is added to the decoding stage of the Seq2Seq model of the UniLM network, that is, the distribution p(y t |y <t ,x) is changed to predict one more label, as shown in formula (5):
[0151] p(y t ,z t ∣y <t ,x)=p(y t ∣y <t ,x)p(z t ∣y <t ,x) (5)
[0152] In formula (5), z is the longest common subsequence of the annotated short text and the administrative law enforcement document text;
[0153] In the prediction phase, for each step, first predict the label z t , if z t If it does not appear in the text of the administrative law enforcement document, then there is no need to change it. t If it appears in the text of administrative law enforcement documents and is the beginning field of a sentence, then in the distribution of tokens, all tokens that are not in the original text are masked. t If a token appears in the text of an administrative law enforcement document and is a continuous segment with the previous token, then all tokens that cannot form the corresponding n-gram in the original text are masked out in the distribution of the token.
[0154] The overall structure of the UniLM network is as follows: Figure 6 As shown in the figure, the result of the extraction model is used as input x, and the manually annotated summary is used as label y. Then we have "[SOS]x[EOS]y[EOS]". The x on the left is a known sequence, and y is an unknown sequence. Therefore, we can use Figure 4 The structure shown does not need to be predicted step by step from "should", so only the economic compensation needs to be predicted. The input and output relationship is as follows Figure 5As shown in the figure, the attention of the input part is bidirectional, and the attention of the output part is unidirectional. In summary, the specific design of the UniLM network is to input a string of word sequences "[SOS]x[EOS]y[EOS]". x is the result of the extraction model, and y is the manually annotated summary. Like BERT, it is represented in three forms: Segment Embedding, Position Embedding, and Token Embedding. It is input into a 24-layer Transformer network. In the Transformer module, multi-head self-attention is used to aggregate the output vector of the previous layer. Each layer controls the attention range through the mask matrix M, as shown in formula (4). During the training process, the source sequence and the target sequence are regarded as a continuous text sequence, and a random mask is used to encourage the model to learn the relationship between the two fragments, and the bidirectional encoder and unidirectional decoder are trained at the same time. Finally, the end mark [EOS] at the target end will also be masked, so that the generation model can learn to predict when to end and complete the training goal. After 30 epochs of training, the required parameters are finally obtained and the training is completed.
[0155] The specific implementation process of step F is as follows: crawl the text of administrative law enforcement documents and convert the text of administrative law enforcement documents into corpus. Segment the text of administrative law enforcement documents into short sentences according to punctuation marks and then remove the punctuation marks. Then use Tokenizer combined with the pre-trained BERT model to vectorize the text. Then input it into the DGCNN network and input the training parameters to extract and save the sentence vectors, then decode the vectors and restore them to text and save them. Input the extracted sentence vectors into UniLM and load the training parameters to generate the final short text result.
[0156] Example 3
[0157] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the intelligent administrative law enforcement case information extraction and case cause identification method described in Example 1 or 2 are implemented.
[0158] Example 4
[0159] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the intelligent administrative law enforcement case information extraction and case cause identification method described in Example 1 or 2.
Claims
1. An intelligent administrative law enforcement case information extraction and case identification method, characterized in that: The steps include: A. Construction of the data set: crawl the administrative penalty decision, extract the text content, and annotate the short text in the text content; B. Corpus conversion: Divide all long sentences in administrative law enforcement documents into short sentences according to punctuation marks; Segment the short text annotated in the dataset in step A according to punctuation marks; Find the longest sentence in the short text, use this longest sentence as a standard, find sentences similar to this longest sentence in the administrative penalty document, extract and save them, and execute recursively until all sentences in the short text have been executed once; C. Extract data preprocessing and vectorization: Step B extracts sentences similar to the label, i.e., the short text, from the administrative law enforcement documents; Filter the extracted text and convert it into a sequence of sentences separated by spaces; encode and index the sentence sequence; After performing global average pooling on the obtained text data, the BERT pre-trained Chinese model with full-word mask is introduced to supplement the average pooling and convert the segmented sentence sequence into a sentence vector. D. Training of the DGCNN-based extraction model: The sentence vectors extracted from the administrative law enforcement documents in step B and the labels, i.e., the short texts, are used as labels, and the sentence vectors generated from the entire text of the administrative law enforcement documents are used as input to train the extraction model; E. Short text generation based on UniLM: The key sentences of administrative law enforcement documents extracted by the trained extraction model are used as input, and the manually annotated short texts are used as labels to train the generation model; F. Input the law enforcement document text for prediction: After the administrative law enforcement documents to be converted are processed through steps A to C, the processing results are input into the trained extraction model to generate and save the extraction results. The extraction results are further input into the trained generation model to obtain short texts suitable for analysis of other downstream tasks, and finally the extraction results and short texts suitable for analysis of other downstream tasks are obtained; The specific implementation process of step D is as follows: The sentence vector corresponding to the administrative law enforcement document text obtained in step C is used as the input of the extraction model. The features are extracted through a layer of DGCNN. The extracted features are input into the Attention layer to complete the integration of sequence information, including encoding the sentence vector sequence of the administrative law enforcement document into a total administrative law enforcement document text vector, and encoding the sentence vector of the annotated short text into a total annotated short text vector. The Attention layer is shown in formula (2): In formula (2), α, W are both trainable parameters, and Act() is the activation function, taking tanch; x i is the sequence before encoding, x is the vector after encoding, λ i is the weight during calculation, softmax i () is the activation function, also known as the normalized exponential function; Afterwards, the sentence vector X corresponding to the total administrative law enforcement document text obtained in step C is used as input, and the most similar sentence vector Y extracted is used as the corresponding label, which is input into a 5-layer DGCNN, and then connected to a fully connected layer; after activation with the Sigmoid activation function, it is compared with the specified threshold, and the sentence vectors greater than the threshold are saved as a list, and the vectors less than the threshold are discarded, and then the sentence vector is decoded into the original text to finally obtain the information extraction result; DGCNN is a dilated gate convolutional neural network, which is constructed as follows: First, add a gate to the ordinary one-dimensional convolution, and the formula is shown in formula (3): In formula (3), X represents the input sequence, Y represents the output sequence, Conv1D1 and Conv1D2 are two one-dimensional convolutions; Next, use dilated convolution; specifically, the input X is first passed through a fully connected layer to extract features, then input into a dilated gate convolution layer with a dilation rate of 1, and then the output is input into a dilated gate convolution layer with a dilation rate of 2; then connect a dilated gate convolution layer with a dilation rate of 4 and a dilated gate convolution layer with a dilation rate of 8; finally, after connecting two dilated gate convolution layers with a dilation rate of 1, the result Y is output after passing through a sigmoid activated fully connected layer.
2. According to claim 1, the intelligent administrative law enforcement case information extraction and case identification method is characterized in that: In step A, marking the text content means: extracting all required key fields from the administrative law enforcement documents, and recombining them into a new short text that only includes key purpose information based on semantics; and marking these short texts in the administrative penalty documents.
3. According to claim 1, the intelligent administrative law enforcement case information extraction and case identification method is characterized in that: In step B, Split all long sentences in administrative law enforcement documents into short sentences according to punctuation marks, specifically: use jieba word segmentation to split the text of administrative law enforcement documents into sentence formats according to punctuation marks, and save them as elements in a list; The short text annotated in the data set of step A is divided into sentences according to punctuation marks, specifically: the annotated short text is divided into short sentence formats using jieba word segmentation and saved in array format; Extract the longest short sentence in the array, match it with the administrative law enforcement documents in the list, extract the sentence in the administrative law enforcement documents in the list that is most similar to the longest short sentence and save it, and perform this step recursively until all the sentences in the array are matched.
4. According to claim 3, the intelligent administrative law enforcement case information extraction and case identification method is characterized in that: Extract the sentences in the administrative law enforcement documents in the list that are most similar to the longest short sentence. The specific implementation process is as follows: Assume x1 is a labeled short text sequence X = [x1, x2, ... x , ] in a sentence, y . It is the administrative law enforcement document sequence Y = [y1, y2, ...y . ], then Flcs is as shown in formula (1): In formula (1), R 012 It refers to the recall rate, which is the ratio of the number of identical characters in the extracted text and the short text to the number of characters in the manual label, i.e. the short text; P 012 It refers to the accuracy, which is the ratio of the number of identical characters in the extracted text and the short text to the total number of extracted characters; F 012 It is ROUGE-L; For a sentence x in a short text sequence X to be labeled , All sentences in the sequence of administrative law enforcement documents y C , j = 1, 2...m; calculate F 012 , take the value that makes F 012 The sentence y in the administrative law enforcement document sequence with the highest score D It is saved as the most similar sentence, and this step is performed recursively until all sentences in the labeled short text sequence X are matched. The extraction result is saved as R, which is the most similar sentence.
5. According to claim 1, the intelligent administrative law enforcement case information extraction and case identification method is characterized in that: The specific implementation process of step C is as follows: First, use the Tokenizer toolkit in tensorflow to remove punctuation and line breaks from the text for text preprocessing; Then, use the Fit_on_texts method of the Tokenizer toolkit to learn the dictionary of the text. Word_index is the mapping relationship between the corresponding word and the number Dict. Through this Dict, each word in each sentence is converted into a number, that is, Texts_to_sequences; Again, pad them to the same length using the Padding method; then use the embedding layer in Keras to perform a vectorization; Finally, the BERT pre-trained Chinese model with full-word mask, namely the Chinese_roberta_wwm_ext_L-12_H-768_A-12 pre-trained model, is introduced to supplement the average pooling.
6. A method for intelligent administrative law enforcement case information extraction and case identification according to any one of claims 1 to 5, characterized in that: The specific implementation process of step E is as follows: a. Group the administrative law enforcement document data Y and the corresponding annotated short text X in step B into n pieces. The grouping standard is: n-1 pieces of data are used as training sets to train the extraction model in step D. In the prediction stage, the untrained data is used as input to predict and generate short texts. This step is repeated n times until all data are trained and used as input for prediction. In this way, the extracted short texts of all administrative law enforcement documents are obtained, which are recorded as C. b. Input the grouped data into the UniLM network for training: First, the extracted short text sequence C is packaged, and the result is P E =[x1,x2,...x H ]; Then, the above packaging result P E Input a K-layer Transformer, that is, P J =Transformer D (P JL1 ),k∈[1,K]; In this way, P passes through K layers of Transformer, E Encoded at different levels of abstraction J =[x1 J ,x2 J ,...x H J ] on the context vector representation; For the k-th layer Transformer, its self-attention head A D The output calculation method is shown in formula (4): In formula (4), Q, K, and V are three vectors. is the trained weight matrix, k∈[1,K]; M is the mask matrix, M iC is the element of the mask matrix, indicating whether it is being paid attention to, 0 means being paid attention to, and negative infinity means being masked; d D represents the dimension of matrix K, A D and V D The subscript k in represents the kth layer.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the intelligent administrative law enforcement case information extraction and case cause identification method described in any one of claims 1-6.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the intelligent administrative law enforcement case information extraction and case cause identification method described in any one of claims 1-6 are implemented.
