Legal core elements identification method and system based on BERT-S6 model

Abstract

The present invention discloses a method and system for identifying the core elements of law based on a BERT-S6 model, and belongs to the technical field of text classification. The method comprises: performing word segmentation on a text to be detected; inputting the word segmentation result into a BERT-S6 stacking block to obtain the legal core element recognition result of the text to be detected; wherein the BERT-S6 stacking block is composed of a plurality of BERT-S6 block stacks, and the processing process of each BERT-S6 block comprises: obtaining a first content representation of input information based on an S6 model, wherein the input information is the output information of a previous BERT-S6 block, and the input information of the first BERT-S6 block is the word segmentation result; obtaining a second content representation of the input information based on a BERT model; and obtaining the output information of the BERT-S6 block based on the first content representation and the second content representation. The present invention can effectively extract and identify the contextual logical relationship of a long sequence of texts in the multi-label classification task of legal element recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of text classification and proposes a legal core element recognition method and system based on the BERT-S6 model. Background Art

[0002] Text-based multi-label classification has gone through three stages. The first stage is the rule reasoning method of event relationship research, the second stage is the research of machine learning methods, and the third stage is the research of the popular neural network model. Traditional multi-label classification methods include binary relevance classifiers, label power sets, and classifier chains. These methods have their own advantages and disadvantages when dealing with multi-label tasks, but they generally have problems such as high computational complexity and difficulty in capturing label correlation. The bottleneck of the current research on neural network models is concentrated on the weak extraction of long-distance dependency relationships of case description information. Summary of the invention

[0003] In response to the above problems, the present invention proposes a method and system for identifying legal core elements based on the BERT-S6 model. By combining the BERT model with the S6 model, the contextual logical relationship of long sequence texts can be effectively extracted and identified in the multi-label classification task of legal element identification.

[0004] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0005] A method for identifying core legal elements based on a BERT-S6 model, the method comprising:

[0006] Perform word segmentation on the text to be detected;

[0007] The word segmentation result is input into the BERT-S6 stacking block to obtain the legal core element recognition result of the text to be detected; wherein the BERT-S6 stacking block is composed of multiple BERT-S6 blocks, and the processing process of each BERT-S6 block includes:

[0008] Obtaining a first content representation of input information based on the S6 model, where the input information is output information of a previous BERT-S6 block, and the input information of the first BERT-S6 block is the word segmentation result;

[0009] Obtain a second content representation of the input information based on the BERT model;

[0010] According to the first content representation and the second content representation, output information of the BERT-S6 block is obtained.

[0011] Furthermore, the obtaining the first content representation of the input information based on the S6 model includes:

[0012] By linearly embedding the input information, a first feature representation of the input information is obtained;

[0013] The first feature representation is subjected to convolution, activation function Silu and S6 block processing to obtain a first content representation of the input information.

[0014] Furthermore, the step of obtaining a second content representation of the input information based on the BERT model includes:

[0015] By linearly embedding the input information, a second feature representation of the input information is obtained;

[0016] Encoding the second feature representation as a low-dimensional dense vector;

[0017] Encoding the low-dimensional dense vector based on an encoder in a Transformer architecture;

[0018] After pre-fine-tuning the encoding result, a second content representation of the input information is obtained.

[0019] Furthermore, encoding the second feature representation into a low-dimensional dense vector includes:

[0020] Calculate a word embedding tensor represented by the second feature;

[0021] Calculate a sentence segment embedding tensor represented by the second feature;

[0022] Calculating a position encoding tensor of the second feature representation;

[0023] A low-dimensional dense vector is obtained based on the word embedding tensor, the sentence segment embedding tensor and the position encoding tensor represented by the second feature.

[0024] Further, the calculating the position encoding tensor of the second feature representation includes:

[0025] Learn the corresponding network parameters based on the training data;

[0026] According to the corresponding network parameters, a position encoding tensor of the second feature representation is obtained.

[0027] Further, obtaining output information of the BERT-S6 block according to the first content representation and the second content representation includes:

[0028] multiplying the first content representation by the second content representation;

[0029] After performing a linear operation on the multiplication result, the linear operation result is embedded and dimensionally reduced to obtain the output information of the BERT-S6 block.

[0030] A legal core element recognition system based on a BERT-S6 model, the system comprising:

[0031] The word segmentation module is used to segment the text to be detected;

[0032] The recognition module is used to input the word segmentation result into the BERT-S6 stacking block to obtain the legal core element recognition result of the text to be detected; wherein the BERT-S6 stacking block is composed of a plurality of BERT-S6 blocks, and the processing process of each BERT-S6 block includes:

[0033] Obtaining a first content representation of input information based on the S6 model, where the input information is output information of a previous BERT-S6 block, and the input information of the first BERT-S6 block is the word segmentation result;

[0034] Obtain a second content representation of the input information based on the BERT model;

[0035] According to the first content representation and the second content representation, output information of the BERT-S6 block is obtained.

[0036] An electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the legal core element identification method based on the BERT-S6 model described in any one of the above items is implemented.

[0037] A computer-readable storage medium, characterized in that computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, any of the above-mentioned legal core element identification methods based on the BERT-S6 model is implemented.

[0038] Compared with the prior art, the present invention has the notable feature that the factual statements of cases are of varying and longer lengths. The BERT-S6 model better integrates the characteristics of BERT and S6, and effectively captures the label characteristics based on the complex contextual logic in long texts, thereby improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Architecture diagram of the BERT-S6 model.

[0040] Figure 2 Schematic diagram of Bert's fine-tuning network for downstream classification tasks. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below in conjunction with specific embodiments. The given examples are only for illustrating the present invention, but not for limiting the scope of the present invention.

[0042] 1. Model framework

[0043] The model structure adopted in this technical solution is as follows Figure 1 As shown:

[0044] Input: Contains case fact description and its corresponding label content; the number and value of labels are provided in advance based on the case fact description, where 1 represents a hit and 0 represents a miss.

[0045] Word segmentation: The input content is segmented according to the Chinese word segmenter to facilitate understanding and representation in subsequent models.

[0046] BERT-S6 stacking block: Multiple BERT-S6 blocks are stacked here, and the output of the previous BERT-S6 block is used as the input of the next BERT-S6 block.

[0047] BERT-S6 block: The input information in this block is divided into two parts:

[0048] 1) The left side will first be linearly embedded, that is, upgraded to a high-dimensional space, used as a feature representation, and then in S6. After convolution, activation function Silu, and S6 blocks, the matrix result will eventually output a content representation in matrix form.

[0049] 2) The right side will first be linearly embedded, that is, upgraded to a high-dimensional space to be used as a feature representation. After that, it will pass through the Embedding module, Transformer module, and pre-fine-tuning module in BERT, and finally output a content representation in the form of a matrix.

[0050] (1) Embedding module: It is composed of three types of Embeddings, and the output tensor is the direct sum of these three tensors.

[0051] Token Embeddings is a word embedding tensor. The first word is the CLS flag, which can be used for subsequent classification tasks.

[0052] Segment Embeddings is a sentence segment embedding tensor, which is used to serve the subsequent pre-training task with two sentences as input.

[0053] Position Embeddings is a position encoding tensor. Note that it is different from the traditional Transformer in that it is not a fixed position encoding calculated by trigonometric functions, but is obtained through learning.

[0054] (2) Transformer module: uses the Encoder part of the classic Transformer architecture.

[0055] (3) Pre-fine-tuning module: The pre-trained token embedding knowledge and Bert’s selfAttention structure are used to directly convert text into corresponding feature vectors, where the first position [CLS] of the vector is used solely for downstream classification tasks. The diagram of Bert’s fine-tuning network for downstream classification tasks is shown in the figure below. Figure 2 As shown in the figure, the input [CLS] + sentence + [SEP] is passed into BERT, and then the vector at the [CLS] position is taken out for classification. The parameters that need to be learned for the fine-tuning task include the classifier of the last layer, the token embedding in Bert, the position encoding, the linear mapping in Self Attention, and various full connections.

[0056] 3) Fusion module: Finally, the structure on the left is multiplied by the structure on the right to avoid overfitting by performing random inactivation to introduce randomness, and then perform linear operations, and finally embedding dimensionality reduction (reducing the dimension back to the same dimension as the input) to obtain the final output.

[0057] Output: Train according to the previous steps, divide the fitting results according to the specified threshold, and assign a value of 1 to each label if it is greater than the threshold, and a value of 0 to each label if it is less than the threshold. The output content is consistent with the input label content structure.

[0058] (II) Based on BERT and Selective State Space Method

[0059] 1. Transformer Architecture

[0060] The core of the Transformer model is the self-attention mechanism, and its overall structure consists of two parts: the encoder and the decoder. The encoder is used to encode the input sequence into a high-dimensional feature vector representation, while the decoder is responsible for decoding the vector representation into the target sequence. Both the encoder and the decoder contain multiple identical layers, each of which consists of multiple attention mechanism modules and feedforward neural network modules. In traditional RNN (recurrent neural network) and LSTM, information is transmitted through time steps, resulting in computational bottlenecks and difficulty in parallelization. Transformer abandons the traditional processing method and introduces the self-attention mechanism to achieve better and more efficient information transmission.

[0061] Transformer has the following three characteristics:

[0062] 1) Parallel performance: Since the Transformer model relies entirely on the self-attention mechanism to model the global dependencies between input and output, it can achieve higher parallelization. This means that the model can process multiple elements in a sequence at the same time, greatly speeding up training.

[0063] 2) Long-distance dependencies: Traditional recurrent neural network models are prone to gradient vanishing or gradient exploding problems when processing long sequences, making it difficult for the model to capture long-distance dependencies. The Transformer model can directly calculate the correlation between any two elements in the sequence through the self-attention mechanism, thereby effectively capturing long-distance dependencies.

[0064] 3) Context-aware: The Transformer model uses a self-attention mechanism to globally model each element in the sequence and establish connections between the elements. This enables the model to consider the information on both sides of the sequence at the same time, thereby capturing the contextual information of the text more accurately.

[0065] 2. BERT Model

[0066] The BERT model is mainly based on the encoder part of the Transformer, but with some special designs. Specifically, the BERT model consists of multiple stacked Transformer encoder layers, each of which contains a self-attention mechanism and a feedforward neural network.

[0067] The input of the BERT model includes text sequences and corresponding position information. The text sequence is first converted into a high-dimensional vector representation through word embedding and position embedding. Then, these vectors are fed into multiple Transformer encoder layers for processing, and each encoder layer updates the vector representation based on the contextual information of the text.

[0068] An important feature of the BERT model is its bidirectionality. Traditional language models (such as RNN and LSTM) usually process text in a unidirectional manner, that is, from left to right or from right to left. The BERT model uses the Transformer encoder structure to achieve bidirectional processing of text sequences. This means that when generating text representations, the BERT model can consider the information on both the left and right sides of the text sequence at the same time, thereby more accurately capturing the contextual information of the text.

[0069] 3. Selective State Space Model

[0070] Dynamically adjust the state space through input-dependent parameters. It has linear time complexity, can process discrete data, selectively process information, has high computational efficiency, and is suitable for large-scale sequence data. It uses one-dimensional convolution to capture local temporal dependencies and has unique advantages in long sequence modeling.

[0071] In summary, the present invention is based on BERT and uses a state space model for multi-label recognition of legal core elements. Among them, the BERT model can capture the rich semantic information and contextual relationships of the text, greatly improving the ability to understand the text. The core idea of ​​S6 is to achieve sequence modeling with linear time complexity by selectively updating the state space parameters, thereby greatly reducing the computational complexity while maintaining the model performance, and being able to efficiently capture long-distance dependencies in the text, making it more suitable for processing long text sequences. Combining the two will capture the correlation between labels while efficiently processing long text sequences to obtain high-quality text representations, and apply them to multi-label binary classification tasks.

[0072] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology in this field to understand the content of the present invention and implement it, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying core legal elements based on the BERT-S6 model, characterized in that: The method comprises: Perform word segmentation on the text to be detected; The word segmentation result is input into the BERT-S6 stacking block to obtain the legal core element recognition result of the text to be detected; wherein the BERT-S6 stacking block is composed of multiple BERT-S6 blocks, and the processing process of each BERT-S6 block includes: Acquiring a first content representation of input information based on the S6 model, wherein the input information is output information of a previous BERT-S6 block, and the input information of the first BERT-S6 block is the word segmentation result; the acquiring the first content representation of the input information based on the S6 model includes: By linearly embedding the input information, a first feature representation of the input information is obtained; Perform convolution, activation function Silu and S6 block processing on the first feature representation to obtain a first content representation of the input information; Obtain a second content representation of the input information based on the BERT model; According to the first content representation and the second content representation, output information of the BERT-S6 block is obtained.

2. The method according to claim 1, characterized in that The obtaining a second content representation of the input information based on the BERT model includes: By linearly embedding the input information, a second feature representation of the input information is obtained; Encoding the second feature representation as a low-dimensional dense vector; Encoding the low-dimensional dense vector based on an encoder in a Transformer architecture; After pre-fine-tuning the encoding result, a second content representation of the input information is obtained.

3. The method according to claim 2, characterized in that Encoding the second feature representation into a low-dimensional dense vector includes: Calculate a word embedding tensor represented by the second feature; Calculate a sentence segment embedding tensor represented by the second feature; Calculating a position encoding tensor of the second feature representation; A low-dimensional dense vector is obtained based on the word embedding tensor, the sentence segment embedding tensor and the position encoding tensor represented by the second feature.

4. The method according to claim 3, characterized in that The calculating the position encoding tensor of the second feature representation includes: Learn the corresponding network parameters based on the training data; According to the corresponding network parameters, a position encoding tensor of the second feature representation is obtained.

5. The method according to claim 1, characterized in that The obtaining output information of the BERT-S6 block according to the first content representation and the second content representation includes: multiplying the first content representation by the second content representation; After performing a linear operation on the multiplication result, the linear operation result is embedded and dimensionally reduced to obtain the output information of the BERT-S6 block.

6. A legal core element recognition system based on the BERT-S6 model, characterized in that: The system comprises: The word segmentation module is used to segment the text to be detected; The recognition module is used to input the word segmentation result into the BERT-S6 stacking block to obtain the legal core element recognition result of the text to be detected; wherein the BERT-S6 stacking block is composed of a plurality of BERT-S6 blocks, and the processing process of each BERT-S6 block includes: Acquiring a first content representation of input information based on the S6 model, wherein the input information is output information of a previous BERT-S6 block, and the input information of the first BERT-S6 block is the word segmentation result; the acquiring the first content representation of the input information based on the S6 model includes: By linearly embedding the input information, a first feature representation of the input information is obtained; Perform convolution, activation function Silu and S6 block processing on the first feature representation to obtain a first content representation of the input information; Obtain a second content representation of the input information based on the BERT model; According to the first content representation and the second content representation, output information of the BERT-S6 block is obtained.

7. An electronic device, characterized in that: The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for identifying core legal elements based on the BERT-S6 model as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the legal core element identification method based on the BERT-S6 model as described in any one of claims 1 to 5.

Images