Method and system for intelligent analysis of bills based on semantic graph model
The GPT graph neural network method addresses limitations of Chinese pre-trained models in legal text analysis by constructing a legal knowledge graph, enabling efficient and accurate processing of legal precedents with enhanced interaction and knowledge integration.
Patent Information
- Application Number
- JP2023197632
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2023-11-03
- Filing Date
- 2023-11-21
- Publication Date
- 2024-11-07
- Estimated Expiration
- 2043-11-21
AI Technical Summary
Existing Chinese pre-trained language models struggle with handling longer legal documents, adapting to specialized legal terminology, and integrating external knowledge bases, leading to inefficiencies in legal precedent analysis.
A legal knowledge graph constructed using a GPT graph neural network method, incorporating a deep learning model with a self-attention mechanism, graph neural network feature optimization, and downstream task utilization to process legal texts, enabling feature extraction, summarization, and compliance analysis.
Enhances legal precedent analysis by allowing multi-round interactions, adapting to legal terminology, and integrating external knowledge, improving efficiency and accuracy in legal text processing.
Smart Images

Figure 0007579022000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to the technical fields of computer software and big data processing, and in particular to a method and system for intelligent analysis of bills based on a semantic graph model. [Background technology]
[0002] At present, with the increasing complexity and volume of legal affairs and the evolution of technology, there are many cases where automated and intelligent solutions are required to solve legal problems. Existing methods mainly rely on legal personnel to analyze case law. Traditional computer technologies, such as text search and synonym analysis, can only realize simple matching, auxiliary search, and other functions, and the operation steps are complicated, making the efficiency of case law analysis very limited. At present, ChatGPT and other language large models have opened a new era of technological paradigm and brought new possibilities to legal case law analysis. Large models can understand the content of case law from the cognitive level and provide legal personnel with a new man-machine interaction mode, which can lower the barrier to AI use in the legal industry and greatly improve the processing efficiency of legal personnel.
[0003] In recent years, pre-trained models have become popular in the field of natural language processing, but most of the mainstream pre-trained models are aimed at the English domain, and the Chinese domain is lagging behind. Chinese and English are the two most frequently used languages in the world, and in view of the importance of Chinese in natural language processing, academia and industry have conducted extensive research and submitted many Chinese pre-trained models. Unlike English pre-trained models, due to the inherent differences between Chinese and English languages, Chinese texts often consist of multiple consecutive characters and have no clear delimiters between words, so pre-trained models trained on a single corpus English dataset, such as BERT, tend to be unsatisfactory when processing general Chinese tasks. To solve the above problems, the research team proposed schemes such as modified word segmentation and masking methods, pre-training with external information, and pre-training for specific domains, and proposed Chinese pre-trained models, such as MacBERT, ERNIE, and RoBERTa, which are mainly based on two basic models, Transformer and BERT, and have achieved good results in many natural language processing tasks.
[0004] With the ever-increasing demand for legal consulting and the development of natural language processing technology, using Chinese natural language pre-trained models to perform text classification and similarity analysis of legal cases has become a problem to be solved. Most of the existing Chinese pre-trained models are targeted at large-scale corpora that are general-purpose, but pre-trained models targeted at specific fields are lagging behind in development and have not been fully used in actual operations. Legal cases have many extractable features and a large data scale. By analyzing existing case corpora, it is possible to summarize common points between similar cases and recommend similar cases.
[0005] The legal field is a field involving various types, long texts, and specialized knowledge, which is challenging and valuable for natural language processing tasks. For example, tasks such as legal case search, judgment prediction, and legal question answering require models to understand and analyze legal case texts and recommend similar cases. Current pre-trained language models have achieved good results in general-purpose fields, but the following three problems remain in the legal field.
[0006] (1) There is a length limit. Because mainstream PLMs are limited in input length to 512 tokens, longer documents must be truncated or spliced together, which can result in lost or confused information. For example, if a case law text contains multifaceted content such as facts, reasons, and judgments, truncation or splicing can destroy the structure and logic of the text, preventing the model from correctly understanding and analyzing the text. Therefore, models that can handle longer sequences are needed to maintain the completeness and consistency of the text.
[0007] (2) Field of application Because the amount of technical terminology, citations of legal provisions, logical reasoning, etc., differs greatly between legal texts and general texts, they may rarely or never appear in general corpora. Therefore, using a model pre-trained on a general corpus may lead to the model lacking knowledge and capabilities in the legal field and being unable to accurately handle tasks in the legal field. Therefore, a model that can adapt to the characteristics and needs of the legal field is needed to improve the generalization ability and effectiveness of the model.
[0008] (3) Knowledge fusion Legal texts not only contain their own text information, but also have close relationships with external knowledge bases. For example, one case text may involve multiple related legal provisions, precedents, principles, etc. These external knowledge can provide more background and references to the text to help the model better understand and analyze the text. However, current PLMs mainly rely on the information in the text itself and rarely utilize external knowledge to enhance the model's capabilities. Therefore, a model that can acquire and fuse related knowledge from external knowledge bases is needed to improve the model's text understanding and analysis capabilities. Summary of the Invention [Problem to be solved by the invention]
[0009] In response to the problems existing in the prior art, the objective of the present invention is to provide a method and system for intelligent analysis of bills based on a semantic graph model, which uses a new GPT graph neural network method to construct a legal knowledge graph, and uses a large amount of case law and reasoning data to train the legal knowledge graph, thereby solving the problem of integrating specific case law texts and knowledge of related legal provisions and regulations, and achieving the definite effect of efficiently providing users with services such as legal case law feature extraction and marching. [Means for solving the problem]
[0010] In order to achieve the above object, the present invention provides A bill intelligent analysis method based on a semantic graph model, comprising: It includes three steps: a GPT language processing step, a graph neural network feature optimization step, and a downstream task step. The GPT language processing step uses a deep learning model Transformer network with a self-attention mechanism to convert the input legal text into a high-dimensional feature vector that can be processed by a computer. In the graph neural network feature optimization step, the high-dimensional feature vector by the GPT language processing module is further optimized and improved, and the connection between nodes is realized by building the relationship between semantics and knowledge, thereby generating bill semantic features, and converting the text data into a graphic representation; In the downstream task utilization step, the graphic representation in the graph neural network feature optimization step is received as input, and then the corresponding result is output, thereby providing a bill intelligent analysis method for realizing understanding and processing of legal text and analysis of legal precedents.
[0011] Further, the downstream tasks include bill comprehension, bill summarization, bill preparation, bill compliance, bill simulation, and bill management.
[0012] In addition, the GPT language processing module performs supervised tasks based on the GPT-2 framework by using unsupervised pre-trained models, and its core lies in language modeling, which is a form of unsupervised distribution estimation. Natural language text is sequence data composed of a set of words or phrases of variable length, and since language is naturally sequenced, the joint probability of symbols is decomposed into a product of conditional probabilities;
number
[0013] In addition, for the sub-blocks of the standard Transformer model, we place a layer regularizer for each layer before each sub-block, add a layer regularizer layer after the last self-attention region, use a corrected initialization method to take into account the influence of model depth on the current layer, multiply the weights of the remaining layers by √N (where N is the number of residual layers) when initializing the weights, expand the context size to 1024 tokens, and also correct it to use a batchsize of 512.
[0014] Furthermore, in the GPT language processing step, before inputting the text into the model, a quantization process is performed, and after word segmentation is performed on the text, a thesaurus is constructed, and then the token is mapped as an ID. In the quantization process step, the input Chinese text is converted to Unicode, unrepresentable characters are deleted, and space characters are converted to displayable space characters. Then, spaces are added before and after the Chinese characters, and words are split by spaces.
[0015] In addition, we use a self-attention mechanism in the Transformer to process long sequences of sentences at the input, modeling dependencies, and serially connect various positions in a sequence so that each word can establish weight dependencies on itself and other words at the time of reconstruction. The weights are calculated as follows: We divide each word vector into one query vector q i and one key vector k i and q i And calculate the similarity or correlation of all key vectors, normalize it with probability estimate through Softmax,
number
number
number
number
number
number
number
number
number
[0016] Furthermore, the graph neural network feature optimization involves classifying texts based on the text features extracted in the text analysis part and calculating the similarity between texts. Specifically, first, a linear transformation is performed on the input embedding using a shared weight matrix, and then a self-attention mechanism is used for each adjacent node, where the self-attention coefficient e for each adjacent node is calculated. ij is as follows: e ij =(Wv i ) T (Wv j ) where W is the trainable linear layer weight and v i is the vector representation of the i-th word output by the Transformer in the first stage, and v j is the vector representation of the jth word output by the Transformer in the first stage. This coefficient represents the importance of node j relative to node i, and the self-attention mechanism allows for the fusion of information from each of the nodes with other neighboring nodes. Finally, we normalize all the neighbors of a node using a softmax function, thereby giving a stable distribution to the output of each of the nodes.
[0017] Furthermore, a multi-head attention mechanism with multiple trainable attention vectors is used to calculate attention coefficients and node feature convolutions for each vector separately, and after calculating the output structure, a nonlinear transformation of the mean function and activation function is performed;
number
[0018] Further, downstream task utilization steps include feature extraction and task adaptation; Feature extraction involves three steps: feature selection, which selects optimal features from the preprocessed data; feature extraction, which extracts the most representative features from the selected features; and feature representation, which represents the extracted features as vectors. Task adaptation involves two steps: task classification, which divides downstream tasks into different categories and processes them using different models for each category, and feature tuning, which adjusts the extracted features according to the downstream task.
[0019] In another aspect, the present invention provides a system for intelligent analysis of bills based on a semantic graph model, which is used to implement the method according to the present invention, A GPT language processing module that converts input legal text into a high-dimensional feature vector that can be processed by a computer; A graph neural network feature optimization module that further optimizes and improves the high-dimensional feature vectors produced by the GPT language processing module, and converts the text data into a graphical representation; and a downstream task utilization module, which receives the graphic representation in the graph neural network feature optimization step as input, and then outputs corresponding results, thereby realizing legal text understanding and processing, and legal case analysis. Effect of the Invention
[0020] <Beneficial Effects> 1) Realize application innovation of GPT general-purpose model in the legal field. By combining the GPT general-purpose model with the consulting and analysis work of specific legal cases in the legal field, we can solve the problem that the traditional legal case analysis can only perform single-round interaction through keyword search, realize the new application effect of multi-round interaction between human and machine questions and answers, and reconstruct the cognitive mode and thinking of case analysis. 2) Realizing technological innovation of adaptive modification of the LawGPT dedicated model. In response to the unique demands of highly specialized legal terminology and complex logical inference, adaptive technological innovation has been made to modules such as language processing of general-purpose large-scale models, the characteristics of graph neural networks, and downstream task utilization, to construct an accurate LawGPT model, which has passed the test verification of the technical validity of the LawGPT model. 3) Realize innovation in the way knowledge is integrated between legal precedents and legal regulations. [Brief description of the drawings]
[0021] [Figure 1] FIG. 1 is an architecture schematic diagram of a bill intelligent analysis system based on a semantic graph model according to the present invention; [Diagram 2] FIG. 2 is a schematic diagram of the Transformers network structure in the bill intelligent analysis method based on the semantic graph model of the present invention; [Diagram 3] 1 illustrates a process of using the attention mechanism of the bill intelligent analysis method and system based on the semantic graph model of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0022] The technical solutions of the present invention will be described below clearly and completely with reference to the drawings, but it is clear that the described embodiments are only some of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present invention.
[0023] In the description of the present invention, the orientations or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the orientations or positional relationships shown in the drawings, and are merely intended to facilitate and simplify the description of the present invention, and do not indicate or imply that the device must have a specific orientation, be configured, or operate in a specific orientation, and are not to be understood as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for the purpose of description only, and should not be understood as indicating or implying relative importance.
[0024] In the description of the present invention, unless otherwise expressly specified or limited, the terms "attach", "connect" and "connection" should be understood in a broad sense, and may be, for example, a fixed connection, a detachable connection, or an integral connection, a mechanical connection, an electrical connection, a direct connection, an indirect connection via an intermediate medium, or an internal communication between two components. A person skilled in the art can understand the specific meaning of the above terms in the present invention depending on the situation.
[0025] In response to the fact that models based on text analysis and classification of legal case law have not yet been widely adopted in the field of Chinese natural language processing, the present invention proposes a pre-trained model that can perform analytical processing on legal case law texts based on Chinese large-scale models, facilitates users' use of a large amount of legal case law corpus, and can also be applied to downstream tasks, such as similarity analysis between different cases or other tasks.
[0026] As shown in Fig. 1, the intelligent analysis system for legal case law based on language graph model (LawGPT) according to the present invention includes three core technology modules, namely, a GPT language processing module, a graph neural network feature optimization module, and a downstream task utilization module, as shown in Fig. 2. The three modules work in close cooperation to deeply understand and process legal texts, thereby improving the efficiency and accuracy of legal case analysis.
[0027] 1. GPT Language Processing Module: This module is the foundation of the entire system, and its main task is to convert the input legal text into a computer-processable high-dimensional feature vector. The high-dimensional feature vector in this case refers to the mapping of words or phrases from a lexicon to a real-valued space of vectors. These high-dimensional vectors can capture the semantic and grammatical relationships between words, for example, the vectors of synonyms will be closer in the space and antonyms will be farther away. They can also capture other characteristics of words, such as gender, plural, tense, etc. To achieve this goal, the present invention employs an advanced deep learning model called the Transformer network. The Transformer network has a self-attention mechanism, which can effectively capture the complex contextual relationships in the legal text and generate high-quality text representations. The Transformer network also has excellent parallel computing capabilities, which can greatly improve the training efficiency of the model. Through deep learning techniques, this module encodes the semantics of complex legal text into a computer-understandable form, which lays the foundation for subsequent processing steps.
[0028] 2. Graph Neural Network Feature Optimization Module: The main function of this module is to further optimize and improve the high-dimensional feature vectors by the GPT language processing module. To achieve this, the present invention introduces an advanced machine learning technology called graph neural network (GNN). Graph neural network can convert text data into a graphical representation, strengthen the connections between nodes by building semantics and knowledge relationships, and generate enhanced bill semantic features. This method can help the system to deeply understand the detailed information of legal precedents, and improve the accuracy and efficiency of downstream tasks. The introduction of graph neural network also enables the system to efficiently process large-scale legal text data, further improving the processing power and scalability of the system.
[0029] 3. Downstream Task Utilization Module: This module is the application side of the system, and is mainly responsible for solving actual legal problems. The present invention designs six different downstream tasks, including bill comprehension, bill summary, bill preparation, bill compliance, bill simulation, and bill management. These tasks cover the main parts of legal work, from understanding and summarizing legal precedents, to preparing legal documents, checking document compliance, and simulating and managing bills, and this module provides appropriate solutions for any of them. To achieve these tasks, the present invention designs separate models for each task, which are trained by many legal texts, thus enabling deep understanding and efficient processing of various legal problems. When processing tasks, this module receives the result representation from the graph neural network module as input, and the data after graph neural network processing includes the relationship between sentences and the corresponding predicted index relationship, but the specific form is still represented as a high-dimensional feature vector, and these high-dimensional feature vectors are input into a dedicated downstream task utilization module, and the corresponding results are output, and these network output results are decoded to output the corresponding legal answers, thereby realizing the automated processing of legal problems.
[0030] The present invention rigorously optimizes and debugs the design of each module so that the three modules work together efficiently. An adaptive learning rate adjustment strategy is introduced into the GPT language processing module, which enables the model to automatically adjust the learning rate during training and improve the learning efficiency. In the graph neural network module, a separate attention mechanism is designed to enable the module to more accurately grasp the important information in the legal text. In the downstream task utilization module, different model structures and parameters are designed based on the characteristics of each task, so that each model can achieve the optimal effect in its specific task. In short, the technical support of LawGPT is jointly constructed by three main technical modules, namely, the GPT language processing module, the graph neural network, and the downstream task processing, and the three cooperate with each other to jointly realize the efficient processing of legal text. This not only reflects the present invention's understanding of the state-of-the-art of deep learning and natural language processing technology, but also shows the present invention's unique strategy and method for solving practical problems.
[0031] The bill intelligent analysis method based on the semantic graph model according to the present invention includes three steps: a GPT language processing step, a graph neural network feature optimization step, and a downstream task utilization step.
[0032] The GPT language processing step is performed by the GPT language processing module, which uses a deep learning model Transformer network with a self-attention mechanism to convert the input legal text into a high-dimensional feature vector that can be processed by a computer.
[0033] The graph neural network feature optimization step is carried out by the graph neural network feature optimization module, which further optimizes and improves the high-dimensional feature vector by the GPT language processing module, and realizes the connection between nodes by building the relationship between semantics and knowledge, thereby generating bill semantic features and converting the text data into a graphic representation.
[0034] The downstream task utilizing step is performed by a downstream task utilizing module, which receives the graphic representation in the graph neural network feature optimization step as input, and then outputs the corresponding result, thereby realizing the understanding and processing of legal text and the analysis of legal precedents.
[0035] The implementation details of each step are described below. The bill intelligent analysis method based on the semantic graph model according to the present invention includes the following steps S1 to S3. S1. GPT language processing step: A deep learning model Transformer network with a self-attention mechanism is used to convert the input legal text into a high-dimensional feature vector that can be processed by a computer. S2. Graph neural network feature optimization step, further optimizing and improving the high-dimensional feature vector by the GPT language processing module, and realizing the connection between nodes by building the relationship between semantics and knowledge, thereby generating bill semantic features and converting the text data into a graphic representation. S3. A downstream task utilization step receives the graphic representation in the graph neural network feature optimization step as input, and then outputs the corresponding result, thereby realizing the understanding and processing of legal text and the analysis of legal precedents.
[0036] Specifically, in the GPT language processing step, first, the natural language text is mapped to a corresponding word vector, and the word vector is usually a concatenation of an embedding vector (usually obtained by mapping a learnable linear layer) that encodes semantics and a position vector that encodes a position. Specifically, the present invention performs a quantization process before inputting the text into the model. Before inputting into the Transformer network, the text is word-split and mapped, and then input into the encoder and decoder structure. The present invention adopts the BERT tokenizer, which is different from the English tokenizer, in Chinese, it is necessary to add an understanding of the definition of a "word", and in Chinese, the basic unit is a character, and the English "word" can be split into a prefix or suffix form, for example, postgraduate can be split into ["post", "##graduate"]. After word splitting is performed on the text, a thesaurus is constructed, and then the token is mapped as an ID, and the GPT language processing step is specifically as follows:
[0037] S101. Convert the input Chinese text to Unicode, remove unrepresentable characters, convert space characters to displayable space characters, then add spaces before and after Chinese characters, and split words by spaces.
[0038] S102.Then, the unsupervised pre-trained model performs a supervised task based on the GPT-2 framework. In this process, language modeling is the core of model training. Language modeling is usually a form of unsupervised distribution estimation. Natural language text is sequence data consisting of a set of words or phrases of variable length. Since language is naturally sequenced, we usually decompose the joint probability of symbols into a product of conditional probabilities.
number
[0039] S103. Finally, the processed text is input into the encoder and decoder architecture to obtain the final output. As shown in FIG. 2, the language model of the present invention is realized based on standard Transformers, and the overall architecture is composed of two parts, an encoder and a decoder, and both the encoder and the decoder are composed of multiple identical sub-blocks stacked together. Each sub-block contains two sub-layers: a self-attention layer and a fully-connected layer. The self-attention layer learns the dependencies between elements by modeling the relationship between each element and other elements in the input sequence. The fully-connected layer further processes the output of the self-attention layer by a multi-layer perceptron. Here, the working principle of the self-attention layer is as follows.
[0040] S1031. Map the input sequence to a query vector, a key vector, and a value vector, respectively.
[0041] S1032. By calculating the dot product of the query vector and the key vector obtained in step S1031, an attention score matrix indicating which query vector is highly correlated with which key vector is obtained.
[0042] S1033. Multiply the attention score matrix obtained in step S1032 by the value vector to obtain a weighted sum vector indicating the output corresponding to each query vector.
[0043] S1034. The output of the self-attention layer is obtained by splicing the outputs of all query vectors (corresponding to each input participle) obtained in S1033.
[0044] In the fully connected layer, the output of the self-attention layer is input to a multi-layer perceptron to obtain the final output through the combination of multiple nonlinear transformations and activation functions. Specifically, the fully connected layer includes a linear transformation and an activation function ReLU. Then, a further linear transformation is performed to obtain the final output of each subblock.
[0045] In the encoder and decoder, the order of the self-attention layers and fully-connected layers is the same, but their inputs and outputs are different. In the encoder, the input to each self-attention layer is an embedding representation of the input sequence, and the output is the encoded representation after the self-attention and fully-connected layers are calculated. In the decoder, the same is true for each self-attention layer and fully-connected layer, but the key vectors and value vectors of the self-attention layer are the output of the encoder and are used to decode the input sequence.
[0046] The language model of the present invention is implemented based on standard Transformers and is modified as follows based on the standard Transformers.
[0047] 1) The standard Transformer subblocks are changed, that is, a layer normalization for each layer is placed before each subblock, a layer regularization layer is added after the last self-attention region, and a layer regularization is placed before the subblock. The main idea is as follows: The text information of the input network can maintain relative stability, and the number of layers of a specific subblock is determined by the downstream task of the model. The more complex the task, the more layers of the subblocks that are stacked, and the deeper the network model becomes. Thus, the more information the network can learn, and the more features of the representation become. With the increase in the depth of the model, especially when multiple Transformer layers are stacked, the network may encounter problems of gradient vanishing or gradient explosion, which makes training difficult and in serious cases, unable to converge. In response, the present invention introduces a layer regularization layer after the self-attention layer. Layer regularization is a normalization technique for normalizing each feature within a layer so that the distribution of each feature is stable. This stability contributes to the stability of gradient propagation and training.
[0048] 2) A modified initialization method is adopted to multiply the weights of the remaining layers by √N (N is the number of residual connection layers) when initializing the weights, taking into account the impact of the model depth on the current layer. The purpose of this modified initialization method is to ensure that the gradient propagation in the network is smoother by taking into account the number of residual connection layers, and the key point is to adjust the scale of the initialization weights based on the depth of the network, thereby better adapting to the existence of residual connections and contributing to the effective propagation of gradients in deep networks.
[0049] 3) We expand the context size to 1024 tokens and use a batch size of 512. Here, the context size represents the length of the input sequence that the model can see when processing the text. This length is usually measured in tokens, and in traditional natural language processing tasks, the context size is usually set small. For example, the context of a conversation system may include the previous sentence of the conversation, while the context of a text generation task may include the previous sentence or paragraph.
[0050] Self-Attention is a key component of the Transformer, which can handle long sequences of sentences at the input, allowing us to model dependencies without considering their distance in the input-output sequence, and also allows us to serially connect different positions of a sequence, allowing each word to establish weighted dependencies on itself and other words at the reconstruction time. We feed each word vector into a single query vector q i and one key vector k i and q i Then, the similarity or correlation of all the key vectors is calculated and normalized to a probability estimate by Softmax.
number
number
number
[0051] In the third stage, we perform attention operations in the Transformer.
number
number
number
number
number
number
[0052] During the training phase, the present invention uses multi-task learning to learn multiple related tasks in parallel, and simultaneously backward-propagates gradients, so that multiple tasks can complement each other's information about the learned domains through basic shared representations, help each other learn tasks, and improve the generalization effect.
[0053] In the validation phase, zero-shot learning is adopted, which mainly solves the following problem: the model is trained using the training set data, and the model can classify the objects in the test set, but there is no intersection between the training set categories and the test set categories, and in the meantime, it is necessary to use the category description to establish the association between the training set and the test set to validate the model. GPT-2 uses a huge corpus to train the model, and can directly connect many downstream tasks during training. Then, when performing a specific task, it is sufficient to input it directly into the pre-trained model without training, and the effect is expected to exceed the effect of other models on small data.
[0054] S2: Graph Neural Network Feature Optimization Step mainly aims to classify texts based on the text features extracted in the text analysis part and calculate the similarity between texts. Specifically, it includes the following sub-steps:
[0055] S201. In order to obtain sufficient expressiveness to transform the input features into higher-level features, the GNN part first performs a linear transformation on the input embedding using a shared weight matrix. Then, for each neighbor node, a self-attention coefficient e ij Calculate. e ij =(Wv i ) T (Wv j )
[0056] where W is the trainable linear layer weight and v i is the vector representation of the i-th word output by the Transformer in the first stage, and v jis the vector representation of the jth word output by the Transformer in the first stage. This coefficient represents the importance of node j to node i, that is, the influence of the two vectors in classifying the text. The self-attention mechanism allows each vector to be fused with other adjacent vector information. In this formula, the attention coefficient is calculated between all nodes so as not to add structural information between vectors. Therefore, in the present invention, when the similarity between two vectors is small, a masking mechanism is incorporated that sets the attention mechanism to 0, so that vectors with small similarity do not affect each other.
[0057] S202. In order to avoid value vanishing or explosion due to different numbers of adjacent nodes of different nodes, we normalize every adjacent node of a node using softmax function, thereby giving a stable distribution to each output of the node. The normalization formula is shown below.
number
[0058] S203. For each node, k points that are closest to the Euclidean distance are connected to the node to generate a graph structure between the nodes, enabling information fusion between different nodes. In the present invention, a part of the node classification is realized based on the GAT, but the input of the GAT does not include an input containing one graph structure feature. In order to classify nodes even if information corresponding to the graph structure is not input, a graph structure generation method has been devised in which k points that are closest to the Euclidean distance are connected to the node to generate a graph structure between the nodes, enabling information fusion between different nodes. This method has an effect similar to clustering, and can obtain more similar features from nodes with similar features by an attention mechanism, thereby accelerating the training speed of the neural network and improving the classification effect.
[0059] In S204. input data processing, the input data is normalized column-wise to a distribution with a mean value of 1 and a variance of 1, and then scaled by a factor of 1 / the number of nodes so that the data to be subjected to high-dimensional input feature generation has a distribution suitable for network learning.
[0060] After calculating the attention coefficient, the normalized coefficient can be used to calculate the node, and the corresponding features can be used to output the corresponding nodes. In order to improve the classification effect of the attention mechanism and improve the efficiency of information utilization, a multi-head attention mechanism with multiple trainable attention vectors is used in the present invention, specifically, the attention coefficient calculation and node feature convolution are performed separately for each vector, and the output structure is calculated, and then the nonlinear transformation of the average function and the activation function is performed.
number
[0061] Here, W k is the linear layer weight of the kth head, the superscript k denotes the index of the head in the multi-head attention mechanism (K heads in total), σ is a nonlinear activation function, and h i is the input word vector of the i-th word, and h i ’ is the output word vector of the i-th word.
[0062] S205.The network structure is designed to use two attention layers for feature processing, and after passing through the ELU activation function, to perform classification using one fully connected layer.
[0063] In Fig. 3, the connecting lines between the nodes represent different heads, and K=3 heads are illustrated in the figure. The meaning of the letters in Fig. 3 is the same as in the formula above.
[0064] S3: Downstream task utilization step,The downstream task utilization module is one of the key modules of the LawGPT system, which passes the optimized features to each downstream task module, so that the system as a whole can perform multiple tasks, such as bill understanding, bill summarization, bill preparation, bill compliance, bill simulation, bill management, etc. The downstream task utilization module mainly includes two parts: feature extraction and task adaptation.
[0065] Feature extraction refers to obtaining feature representations corresponding to words in natural language text in the LawGPT system, which usually involves representing the fonts in the text as high-dimensional vectors and passing them to downstream task modules to perform downstream tasks.
[0066] Task adaptation is to pass the extracted features to downstream task modules and adjust the features for each downstream task. This process mainly includes two steps: task classification and feature adjustment. Task classification is to classify downstream tasks into different categories and use different models to process each category. For example, a text classification model can be used for the bill comprehension task, and a generative model can be used for the bill summarization task. Feature adjustment is to adjust the extracted features for each downstream task to improve the accuracy and effectiveness of the model. For example, in the bill compliance task, the order of extracted features needs to be changed and evaluated to ensure that the model complies with regulations.
[0067] The overall testing procedure for the method and system is as follows. 1. Feature extraction and classification of legal texts. 2. Use the GPT language processing module to extract features from the test data and obtain a vector representation for each piece of text. 3. Using the classification task of the "downstream task module", evaluate the classification results with a confusion matrix and calculate metrics such as precision, recall, and F1 score. 4. Record the execution time and memory usage of the classification algorithm. 5. Perform similarity analysis on the input text and recommend similar cases. 6. The "GPT language processing module" uses a pre-trained model to extract features from the input text and obtain a vector representation of the input text. Then, the "graph neural network feature optimization module" optimizes the features to obtain the word with the highest probability, and obtains the predicted output text. 7. Using the similarity measurement task of the "downstream task module", calculate the similarity between the output text and each text in the test data by a method such as cosine similarity, and obtain a similarity list. 8. Record the correlation score and user satisfaction score of the recommended precedent. 9. Record the response time and memory usage of the recommended processes. The system of the present invention can perform bill comprehension, bill summary, bill creation, bill compliance, bill simulation, and bill management.
[0068] Understand the bill: LawGPT is able to understand the content of a bill at a semantic level, and after users provide suggestions to the system, they can pose questions about the bill and the model can answer these questions. Users can have LawGPT analyze the proposal and provide a short summary for a quick understanding of the relevant bill. For example, in one specific embodiment, a user enters the following text: JPEG0007579022000027.jpg103167The bill intelligent analysis system based on the semantic graph model according to the present invention processes the above input content, and the bill understanding module outputs the following text: JPEG0007579022000028.jpg67168
[0069] Legislation: The user can ask the system to automatically generate similar bills based on the content of the current bill and to make certain modifications to the case law based on the user's instructions.
[0070] Compliance with the bill: Users can ask the intelligent assistant to analyze which specific laws and regulations are covered under the current draft and perform a compliance analysis based on the corresponding legislation.
[0071] Bill Simulation One of the most important applications of bill analysis is determining the outcome of upcoming litigation cases. LawGPT provides professional and intelligent legal assistance and analysis through learning from a large number of bills. Users submit the content of the lawsuit they are planning to file to LawGPT intelligent assistant, and LawGPT will analyze the probability of winning and provide specific legal suggestions to users.
[0072] Bill management: Users can manage legal precedents that have been uploaded in the past within the system. LawGPT performs systematic intelligent analysis on the bills uploaded by users and displays them in intelligent visualization ways, including bill-related parallel coordinate graphs, bill semantic radial tree graphs, bill dependency graphs, bill hotspot horizontal trend graphs, etc.
[0073] The system also has a moot court module. One of the most important applications of bill analysis is to judge the outcome of upcoming litigation cases. LawGPT provides professional and intelligent legal assistance and analysis through learning from a large number of bills. Users submit the content of the lawsuit they are planning to file to LawGPT intelligent assistant, and LawGPT will analyze the probability of winning and provide specific legal suggestions to users.
[0074] The technical advantages of the present invention are as follows: 1) Realize application innovation of GPT general-purpose model in the legal field. By combining the GPT general-purpose model with the consulting and analysis work of specific legal cases in the legal field, we can solve the problem that the traditional legal case analysis can only perform single-round interaction through keyword search, realize the new application effect of multi-round interaction between human and machine questions and answers, and reconstruct the cognitive mode and thinking of case analysis. 2) Realize technological innovation of adaptive modification of LawGPT dedicated model. In response to the unique demands of highly specialized legal terminology and complex logical inference, adaptive technological innovation has been made in modules such as general-purpose large-scale model language processing, graph neural network characteristics, and downstream task utilization to build an accurate LawGPT model. Based on the 2,000 case data of the intelligent legal database of Beihai University Law Treasure and actual case precedents from legal offices, the technical validity of the LawGPT model has been tested and verified. 3) Realize innovation in the method of integrating knowledge between legal precedents and legal regulations. From the perspective of the logic of the entire work, a new GPT graph neural network method is used to construct a legal knowledge graph, and a large amount of case and reasoning data is used to train the legal knowledge graph, thereby solving the problem of integrating specific case texts and related legal provisions and regulations, and achieving the tangible effect of efficiently providing users with services such as legal precedent feature extraction and marching.
[0075] From the above, LawGPT is characterized by applied innovation, technological innovation and methodological innovation, and has demonstrated its remarkable innovativeness and practicality in understanding legal texts, digging deeper and applying legal knowledge, and improving the accuracy and efficiency of the system.
[0076] Any process or method description in the flow charts of the present invention or otherwise described herein may be understood as a module, fragment, or portion representing a code of executable instructions comprising one or more steps for implementing a particular logical function or process, and may be implemented in any computer-readable medium for an instruction execution system, apparatus, or device, said computer-readable medium being any medium containing a program for storing, communicating, propagating, or transmitting for use with the execution system, apparatus, or device, including read-only memory, magnetic disks, optical disks, etc.
[0077] In the present description, reference to terms such as "embodiment", "example", etc. means that the particular feature, structure, material, or characteristic described with reference to the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, general expressions of the above terms do not necessarily have to be directed to the same embodiment or example. Furthermore, a person skilled in the art may combine or combine different embodiments or examples described herein and their features without creating a contradiction.
[0078] While the embodiments of the present invention have been shown and described above, the above examples are illustrative and should not be understood as limiting the present invention. It will be understood that a person skilled in the art can make updates such as changes, modifications, substitutions, and variations to the above examples within the scope of the present invention.
Claims
1. A bill intelligent analysis method based on a semantic graph model, comprising: a GPT language processing module that converts input legal text into a high-dimensional feature vector that can be processed by a computer; a graph neural network feature optimization module that further optimizes and improves the high-dimensional feature vector converted by the GPT language processing module, and converts legal text data into a graphic representation; A downstream task utilization module receives the graphic representation in the graph neural network feature optimization step as input, and outputs a corresponding result, thereby realizing the understanding and processing of the input legal text and the analysis of legal precedents; The bill intelligent analysis method includes three steps: a GPT language processing step realized by the GPT language processing module; a graph neural network feature optimization step realized by a graph neural network feature optimization module; and a downstream task utilization step realized by a downstream task utilization module; In the GPT language processing step, the GPT language processing module converts the input legal text into a high-dimensional feature vector that can be processed by a computer using a deep learning model Transformer network with a self-attention mechanism, specifically, In the GPT language processing step, before the legal text is input into the unsupervised pre-trained model, a quantization process is performed, and the text is subjected to word segmentation, after which a thesaurus is constructed, and then the token is mapped as an ID; In the quantization step, the input legal text is converted into Unicode, the unrepresentable characters are deleted, and the space characters are converted into displayable space characters. Then, spaces are added before and after the Chinese characters, and words are divided by the spaces; Next, the quantized data is input to an unsupervised pre-trained model, and the unsupervised pre-trained model executes a supervised task based on the GPT-2 framework, predicting the character or word with the highest conditional probability calculated by the following (Equation 1) as the next sentence or word, and repeating this process up to the stop character of the sentence to generate a complete sentence. where, s is the probability that the unsupervised pre-trained model generates phrase s, [0010] where n is the length of phrase s after word segmentation, and the conditional probability is the probability of the first i-1 (2≦i≦n) words s of the unsupervised pre-trained model. 1 , …, s i-1 Given a word, the unsupervised pre-trained model estimates the probability p(s i |s 1 , …, s i-1 ) Then, the generated complete sentence text is input to an encoder-decoder architecture to obtain a high-dimensional feature vector as the final output; In the graph neural network feature optimization step, the graph neural network feature optimization module further optimizes and improves the high-dimensional feature vector obtained by the GPT language processing module, and realizes the connection between nodes by building the relationship between semantics and knowledge, thereby generating bill semantic features, and converting the text data into a graphic representation, specifically: The text analysis part classifies texts based on the extracted text features and calculates the similarity between texts. Specifically, first, a linear transformation is performed on the input high-dimensional feature vector using a shared weight matrix, and then a self-attention coefficient e is calculated for each adjacent node, which is a node provided in each layer from the input layer to the output layer of the neural network and means a node that is adjacent to the input layer in a data transmission relationship. ij is calculated using the following formula, e ij =(Wv i ) T (1) j ) where W is the trainable linear layer weight and v i is the vector representation of the i-th word output by the Transformer in the GPT language processing step, and v j is the vector representation of the jth word output by the Transformer in the GPT language processing step, and the self-attention coefficient e ij represents the importance of node j, in which a vector corresponding to a jth word is located, relative to node i, in which a vector corresponding to an ith word is located, i.e., the influence of two vectors located at nodes i and j on the classification of text; Finally, we normalize all the neighbors of a node using the softmax function, thereby giving a stable distribution to the output of each of the nodes. In the downstream task utilizing step, the downstream task utilizing module receives the graphic representation in the graph neural network feature optimizing step as input, and then executes the downstream task to output a corresponding result, thereby realizing the downstream task; A bill intelligent analysis method comprising:
2. The method for intelligent bill analysis based on a semantic graph model according to claim 1 , wherein the downstream tasks include bill comprehension, bill summarization, bill preparation, bill compliance, bill simulation, and bill management.
3. The method for intelligent analysis of bills based on semantic graph models as described in claim 1, characterized in that for sub-blocks of the standard Transformer model, a layer regularization is placed before each sub-block, a layer regularization layer is added after the last self-attention region, the size of the context is expanded to 1024 tokens, and a batch size of 512 is used.
4. The Transformer uses a self-attention mechanism to process long sequences of sentences at the input, modeling dependencies and serially connecting various positions of a sequence so that each word can establish weight dependencies on itself and other words at the time of reconstruction. The weights are calculated as follows: Each word vector is treated as a single query vector q i and one key vector k i and q i And calculate the similarity or correlation of all key vectors, and normalize it with probability estimation through Softmax; [0025] Here, sim ij =q i T k j and a ij is the i-th query vector q i and the j-th key vector k j is the attention coefficient of , e is a natural number, T denotes matrix transpose, In the third step, the Attention operation in the Transformer is performed. [0030] The query, key, and value vector sums of each word in the sentence are spliced into a matrix and parallel operations are performed, where [0045] is a matrix representation of the query vector, [0050] is a matrix representation of the key vector, [006] is a matrix representation of the value vector, [0070] is the matrix representation of the output vector, and d k The method for intelligent analysis of bills based on a semantic graph model according to claim 1, wherein is the number of dimensions of the query, key and value vectors.
5. Using the self-attention mechanism means using a multi-head attention mechanism with multiple trainable attention vectors to calculate attention coefficients and node feature convolution for each vector separately, and then calculating the output structure, and then performing nonlinear transformation of the mean function and the activation function; [0080] Here, W k is the linear layer weight of the kth head, the superscript k denotes the index of the head in the multi-head attention mechanism, σ is a nonlinear activation function, and h i is the input word vector of the i-th word, and h i ’ The method for intelligent analysis of bills based on semantic graph model according to claim 1 , wherein i is the output word vector of the i-th word.
Citation Information
Patent Citations
Document association auditing method based on semantic analysis
CN111091009A
Cited By
Structure-aware large model graph reasoning method and system
CN119150996A
Key work completeness evaluation method and device based on generative AI large model
CN119180566A
Humanoid large model pregnant and lying-in woman accompanying digital human system and accompanying method thereof
CN119227724A
Power grid fault diagnosis method and system based on text vectorization
CN119622483A
Power grid fault diagnosis method and system based on text vectorization
CN119622483B
