A bill text multi-label classification method and system based on contrast learning and graph learning

Abstract

The application discloses a bill text multi-label classification method and system based on contrast learning and graph learning, and comprises the following steps: taking a roBerta model as a basis, and continuously training on a bill text data set to adapt to the characteristics of legal texts; based on an aprior algorithm, frequent item sets in labels are mined to provide a similarity measurement index for constructing positive and negative samples; based on the fine-tuned roBerta model and the similarity measurement index, sample pairs are constructed to perform contrast learning; based on the characteristics of bill data, a heterogeneous graph is constructed in combination with external data, and the fine-tuned roBerta model is used for bill text representation learning to serve as the initial representation of related nodes in the heterogeneous graph; graph neural network technology is used to complete node representation learning; and on this basis, edge prediction between bill and label nodes is realized, and bill text multi-classification is completed. The application effectively alleviates the model training problem caused by a small sample corpus environment and has strong expansibility.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and system for multi-label classification of legal texts based on contrastive learning and graph learning. Background Technology

[0002] Multi-label text classification aims to provide a set of labels for a text instance to identify different dimensional features of the text. For example, a report on the successful launch of Shenzhou-14 can be classified as social news, but it can also be labeled as science and technology news. However, solving multi-label classification tasks faces several challenges: first, the number of classes is uncertain; second, there are interdependent relationships between class labels. Especially in the field of topic classification of legal texts, the dataset size is small and there is a serious class imbalance problem, making it even more difficult to support the training and application of current mainstream natural language processing methods—pre-trained language models.

[0003] Currently, multi-label classification techniques are often improvements on traditional machine learning methods, such as ML-KNN and Rank-SVM. For deep learning methods, to avoid the exponential expansion of the prediction space, multi-label classification tasks are generally converted into sequence prediction tasks. These methods suffer from problems such as weak feature mining capabilities, low performance, and poor scalability. Summary of the Invention

[0004] This invention addresses the shortcomings of existing multi-label classification technologies, such as weak feature mining capabilities, low performance, and poor scalability. It provides a method and system for multi-label classification of bill texts based on contrastive learning and graph learning, specifically designed to solve the task of classifying bill text topics with multiple labels. The aim is to capture features related to topic labels in the text through contrastive learning, thereby obtaining a better semantic representation. Based on this, heterogeneous graph modeling is used for the multi-label bill text topic classification task, fusing multiple types of information. Then, graph neural network technology is used to learn the representation of nodes, followed by edge prediction. In the design of positive and negative samples for contrastive learning and the graph modeling process, the characteristics of the multi-label classification task are fully considered, and the potential dependencies between multiple labels are explored.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] This invention proposes a multi-label classification method for legal texts based on contrastive learning and graph learning, comprising:

[0007] Step 1: Based on the trained language model roBerta, continue training on the legal text dataset to adapt to the characteristics of legal texts, thus completing a fine-tuning of the roBerta model.

[0008] Step 2: Based on the aprior algorithm, frequent itemsets in the tags are mined to provide a similarity metric, Sim, for constructing positive and negative examples;

[0009] Step 3: Based on the fine-tuned roBerta model and the similarity metric Sim, construct sample pairs for comparative learning to complete the second fine-tuning of the roBerta model.

[0010] Step 4: Based on the characteristics of the bill data, construct a heterogeneous graph by combining external data, and use the second-fine-tuned roBerta model for learning the bill text representation as the initial representation of the relevant nodes in the heterogeneous graph;

[0011] Step 5: Use graph neural network technology to learn node representations; based on this, predict the edges between bill and label nodes to complete the multi-classification of bill text.

[0012] Further, step 1 includes:

[0013] Step 1.1: Select the pre-trained language model roBerta as the base. The language version of this model is adjusted according to the language of the task.

[0014] Step 1.2: Use keyword extraction technology to identify keywords from the samples in the bill text dataset;

[0015] Step 1.3: Replace the keywords above with a mask [mask] and input it into the roBerta model as the training dataset to complete the prediction task of the [mask] label.

[0016] Further, step 2 includes:

[0017] Step 2.1: Based on the Aprior algorithm, mine frequently occurring label pairs in the labels and obtain the confidence level σ of the frequently occurring label pairs. On this basis, set a threshold Δ to filter out label pairs with high confidence.

[0018] Step 2.2, for a given pair of bill texts<T1,T2> and its labels<L1,L2> Define and calculate the similarity between the two:

[0019] Sim(T1, T2)=[len(L1∩L2) / len(L1∪L2)]·(∑ i，j σ ij / N i，j (2)

[0020] Where i and j are any two labels in the intersection of the two bill label sets, {i, j | i, j ∈ L1 ∩ L2, σ i，jSim(T1, T2) represents the similarity between T1 and T2; len(L1∩L2) / len(L1∪L2) is the similarity benchmark, which is the ratio of the size of the intersection and the size of the union of the two labels.

[0021] Further, step 3 includes:

[0022] Step 3.1: Based on the calculated similarity index of the bill text, design the loss function for contrastive learning:

[0023] loss=||Cosine(E1, E2)-λ·Sim(T1, T2)|| (3)

[0024] Where Cosine(·) represents the cosine similarity calculation; E1 and E2 represent the two text representations obtained by inputting a pair of legal texts into the fine-tuned roBerta model; λ represents the hyperparameter.

[0025] Step 3.2: Randomly select a pair of legal texts and input them into the fine-tuned roBerta model to obtain the representations of the two texts; then, calculate the loss function according to formula (3) and backpropagate the loss to continue training the model.

[0026] Further, step 4 includes:

[0027] Step 4.1: Construct a heterogeneous graph, using the tags, bill initiators, and the bill text itself as nodes in the graph. Design association rules based on the real-world context to connect the nodes and complete the graph construction process.

[0028] Step 4.2 assigns different feature representations to different nodes, where the labels and bill initiators are encoded using one-hot encoding, and the bill text nodes are encoded using the roBerta model after secondary fine-tuning.

[0029] Further, step 5 includes:

[0030] Step 5.1, Divide the training set and test set: First, divide the bill nodes to obtain training nodes and test nodes; then, randomly occlude the edges between the bill and different labels according to a certain probability, so as to use the content predicted by the model.

[0031] Step 5.2, train the graph neural network: use the heterogeneous graph neural network algorithm to predict the edges of the random masking mentioned above, and complete the model training;

[0032] Step 5.3: Based on the trained graph neural network model, predict whether there is a correlation between bills and label nodes in the test dataset.

[0033] Another aspect of this invention proposes a multi-label classification system for legal texts based on contrastive learning and graph learning, comprising:

[0034] The adaptive training module is used to further train the roBerta model on the legal text dataset to adapt to the characteristics of legal texts, thus completing a fine-tuning of the roBerta model.

[0035] The similarity calculation module is used to mine frequent itemsets in tags based on the aprior algorithm, providing a similarity metric, Sim, for constructing positive and negative examples.

[0036] The contrastive learning module is used to construct sample pairs based on the fine-tuned roBerta model and the similarity metric Sim for contrastive learning, thereby completing the second fine-tuning of the roBerta model.

[0037] The heterogeneous graph construction module is used to construct a heterogeneous graph based on the characteristics of the bill data and combined with external data, and to use the roBerta model after secondary fine-tuning for learning the bill text representation as the initial representation of the relevant nodes in the heterogeneous graph.

[0038] The edge prediction module uses graph neural network technology to learn node representations; based on this, it performs edge prediction between bill and label nodes to complete multi-classification of bill text.

[0039] Furthermore, the adaptive training module is specifically used for:

[0040] The pre-trained language model roBerta was chosen as the base, and the language version of the model was adjusted according to the language of the task.

[0041] Keyword extraction techniques were used to identify keywords from samples in the bill text dataset.

[0042] Replace the keywords above with a mask [mask] and input it into the roBerta model as the training dataset to complete the prediction task of the [mask] label.

[0043] Furthermore, the similarity calculation module is specifically used for:

[0044] Based on the Aprior algorithm, we mine frequently occurring tag pairs and obtain the confidence level σ of the frequently occurring tag pairs. Based on this, we set a threshold Δ to filter out tag pairs with high confidence.

[0045] For a given pair of bill texts<T1,T2> and its labels<L1,L2> Define and calculate the similarity between the two:

[0046] Sim(T1, T2)=[len(L1∩L2) / len(L1UL2)]·(∑ i，j σ ij / N i，j (2)

[0047] Where i and j are any two labels in the intersection of the two bill label sets, {i, j | i, j ∈ L1 ∩ L2, σ i，j Sim(T1, T2) represents the similarity between T1 and T2; len(L1∩L2) / len(L1UL2) is the similarity benchmark, which is the ratio of the size of the intersection and the size of the union of the two labels.

[0048] Furthermore, the contrastive learning module is specifically used for:

[0049] Based on the calculated similarity index of the bill text, a loss function for contrastive learning is designed:

[0050] loss=||Cosine(E1, E2)-λ·Sim(T1, T2)|| (3)

[0051] Where Cosine(·) represents the cosine similarity calculation; E1 and E2 represent the two text representations obtained by inputting a pair of legal texts into the fine-tuned roBerta model; λ represents the hyperparameter.

[0052] A pair of legal texts is randomly selected and input into the fine-tuned roBerta model to obtain the representations of the two texts; then, the loss function is calculated according to formula (3) and the loss is backpropagated to continue training the model.

[0053] Furthermore, the heterogeneous graph construction module is specifically used for:

[0054] Construct a heterogeneous graph, using labels, bill initiators, and the bill text itself as nodes in the graph. Design association rules based on the real-world context to connect the nodes and complete the graph construction process.

[0055] Different features are assigned to different nodes. Labels and bill initiators are encoded using one-hot encoding, while bill text nodes are encoded using a roBerta model that has been finely tuned twice.

[0056] Furthermore, the edge prediction module is specifically used for:

[0057] Divide the data into training and testing sets: First, divide the bill nodes into training and testing nodes; then, randomly mask the edges between the bill and different labels with a certain probability to serve as the content for model prediction.

[0058] Training the graph neural network: The heterogeneous graph neural network algorithm is used to predict the edges of the random masking mentioned above to complete the model training;

[0059] Based on a trained graph neural network model, the system predicts whether there is a correlation between bills and label nodes in the test dataset.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] (1) This invention employs a contrastive learning method, effectively alleviating the model training challenges posed by small sample corpora. Simultaneously, it better achieves semantic embedding representation of sentences, enabling the feature vectors learned by the model to effectively integrate multi-label semantics and the inherent relationships within the labels themselves. The frequent itemset mining method can achieve label dependency mining in a relatively efficient manner.

[0062] (2) This invention achieves multi-label bill text classification based on the edge prediction task of heterogeneous graphs, which can effectively avoid the label space explosion caused by multi-label combinations. The construction process of heterogeneous graphs can incorporate other effective information, such as the bill proposer, to help improve the accuracy of multi-label text classification tasks.

[0063] (3) This invention has strong scalability. First, it can be used in cross-modal tasks such as text and images to help overcome modal limitations. Second, it is not limited to bill text classification tasks. For text classification tasks with few samples and multiple labels, domain knowledge can be combined to construct appropriate heterogeneous graphs to complete the bill text classification task. Attached Figure Description

[0064] Figure 1 This is a general framework diagram of a multi-label classification method for bill text based on contrastive learning and edge prediction tasks according to an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram of the architecture of a multi-label classification system for legal text based on contrastive learning and edge prediction tasks, according to an embodiment of the present invention. Detailed Implementation

[0066] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0067] like Figure 1 As shown, a multi-label classification method for bill text based on contrastive learning and graph learning includes the following steps:

[0068] Step 1) Select the roBerta model and use this pre-trained language model as a base to continue training on the legal text dataset to adapt to the characteristics of legal texts, thus completing a fine-tuning of the roBerta model.

[0069] Step 2) Based on the aprior algorithm, frequent itemsets in the tags are mined to provide a similarity metric, Sim, for constructing positive and negative examples;

[0070] Step 3) Based on the fine-tuned roBerta model, construct sample pairs to complete comparative learning, which is essentially a secondary fine-tuning of the roBerta model;

[0071] Step 4) Based on the characteristics of the bill data, construct a heterogeneous graph by combining external data, and use the second-fine-tuned roBerta model for learning the bill text representation as the initial representation of the relevant nodes in the heterogeneous graph;

[0072] Step 5) Use graph neural network technology to learn node representations; on this basis, perform edge prediction between bill and label nodes to complete multi-classification of bill text.

[0073] Further, step 1 includes:

[0074] Step 1.1) Select the pre-trained language model roBerta as the base and download the language model that matches the language of the task data. The roBerta model samples mask prediction and next-sentence prediction tasks during training, consistent with the BERT model. However, unlike the BERT model, roBerta employs a dynamic masking mechanism, which greatly enhances data diversity.

[0075] Step 1.2) For the samples in the bill text dataset, use TF-IDF or other keyword extraction techniques to identify keywords. After filtering the bill text for stop words, input it into the TF-IDF algorithm model, calculate the TF-IDF value of each word in the text, sort them from high to low, and take 10% of the length of the top-ranked bill text as the objects to be masked;

[0076] Step 1.3) For each sample, replace one of the keywords mentioned above with a mask [mask]. The same sample constitutes different training data because the positions of the replaced keywords differ. The training dataset is input into the Roberta model, and the output of the last hidden layer of the Roberta model is used to complete the prediction task of the [mask] replacement word. The loss is calculated as follows:

[0077] Loss(p, q) = -∑ x p(x)·logq(x) (1)

[0078] For domain-specific text classification tasks with few samples, it is essential to perform domain-transfer retraining on the language model. Furthermore, to avoid insufficient model training, the keyword truncation ratio in step 1.2 can be appropriately increased.

[0079] Further, step 2 includes:

[0080] Step 2.1) In the existing bill text dataset, each bill text has at least two labels, mainly related to the content of the bill, such as economy and medical care. It is considered beneficial to extract frequent 2-itemsets for subsequent contrastive learning. Therefore, based on the Aprior algorithm, frequently occurring label pairs are mined, and the confidence σ of each label pair is calculated. Based on this, a threshold Δ is set to filter out high-confidence label pairs. The threshold is set based on 0.25, 0.5, and 0.75 bits, and the experimental results under different parameter settings are compared.

[0081] Step 2.2) For a given pair of bill texts<T1,T2> and its tag set pairs<L1,L2> Define the correlation (similarity) between the two:

[0082] Sim(T1, T2)=[len(L1∩L2) / len(L1UL2)]·(∑ i，j σ ij / N i，j (2)

[0083] Among them, {i,j|i,j∈L1∩L2,σ i，j Let len(L1∩L2) / len(L1UL2) be the similarity benchmark, representing the ratio of the intersection to the union of the combined labels of the two bills, measuring the similarity between the two bill texts. Based on this, considering the impact of potential dependencies within the label sets on similarity calculation, label pair confidence is introduced into the similarity calculation: where i and j are any two labels in the intersection of the two bill label sets, participating in the calculation when their confidence is greater than a set threshold. All possible i and j are iterated, and the confidence values ​​are accumulated and summed.

[0084] Furthermore, step 3 includes:

[0085] Step 3.1) Design the contrastive learning loss function. It is assumed that the semantic representation of the bill obtained through contrastive learning should have the following characteristics: the higher the label overlap, the higher the similarity of the text semantic representation; conversely, the lower the overlap, the lower the similarity. Based on this, using the bill text relevance (similarity) calculated in equation (2), the contrastive learning loss function is designed:

[0086] loss=||Cosine(E1, E2)-λ·Sim(T1, T2)|| (3)

[0087] Where Cosine(·) represents the cosine similarity calculation, and λ represents the hyperparameter, which is a variable coefficient slightly greater than 1;

[0088] Step 3.2) Select a pair of legal texts from the bill text dataset using a purely random method. Input them into the fine-tuned roBerta model to obtain the representations E1 and E2 of the two texts; then, calculate the loss according to the method of formula (3) and backpropagate the loss based on the BP algorithm to correct the roBerta model parameters and complete the secondary training of the model.

[0089] Furthermore, step 4 includes:

[0090] Step 4.1) Construct a heterogeneous graph. Examine the characteristics of the bill data and conclude that bills proposed by the same author or organization share similar themes. Therefore, label (node) l ), bill sponsors or organizations (node) a ) and the text of the bill itself (node t The nodes are represented as nodes in the graph. Based on the real-world context, association rules are designed, assuming the basic graph pattern of this heterogeneous graph is that a bill is proposed by a bill initiator or organization, and the bill has multiple topic labels. Furthermore, there are edge connections between label pairs with confidence levels higher than a threshold. Based on this, the nodes are associated to complete the heterogeneous graph construction process.

[0091] Step 4.2) Node representation initialization. Different feature representations are assigned to different nodes, which can be divided into two categories: for nodes without complex semantics, such as tags, bill initiators, etc., one-hot encoding can be used; for bill text nodes with complex semantics, the roBerta model obtained in step 3) with secondary fine-tuning is used for semantic encoding.

[0092] Furthermore, step 5 includes:

[0093] Step 5.1) Divide the training and testing sets. To prevent the data used for testing the model from being leaked during model training, the bill nodes are divided in an 8:2 ratio to obtain the training node set U and the testing node set T. First, for the testing node T, all edges between it and the label node are deleted; then, the edges between bills and labels in the training node set U are randomly occluded with a certain probability to construct the training dataset.

[0094] Step 5.2) Train the graph neural network. A heterogeneous graph neural network algorithm, such as HGAT, is used to predict the random masking edges mentioned above. Specifically, the HGAT algorithm is first used to perform full-graph information transfer on the heterogeneous graph after dividing it into training and test sets. After multiple iterations, the representation E of each node is obtained. i Therefore, for edge prediction tasks, a feedforward layer is used to complete the binary classification task:

[0095]

[0096] Step 5.3) Based on the trained graph neural network model, predict whether there is a correlation between bills and label nodes in the test dataset.

[0097] Based on the above embodiments, such as Figure 2 As shown, this invention also proposes a multi-label classification system for legal texts based on contrastive learning and graph learning, comprising:

[0098] The adaptive training module is used to further train the roBerta model on the legal text dataset to adapt to the characteristics of legal texts, thus completing a fine-tuning of the roBerta model.

[0099] The similarity calculation module is used to mine frequent itemsets in tags based on the aprior algorithm, providing a similarity metric, Sim, for constructing positive and negative examples.

[0100] The contrastive learning module is used to construct sample pairs based on the fine-tuned roBerta model and the similarity metric Sim for contrastive learning, thereby completing the second fine-tuning of the roBerta model.

[0101] The heterogeneous graph construction module is used to construct a heterogeneous graph based on the characteristics of the bill data and combined with external data, and to use the roBerta model after secondary fine-tuning for learning the bill text representation as the initial representation of the relevant nodes in the heterogeneous graph.

[0102] The edge prediction module uses graph neural network technology to learn node representations; based on this, it performs edge prediction between bill and label nodes to complete multi-classification of bill text.

[0103] Furthermore, the adaptive training module is specifically used for:

[0104] The pre-trained language model roBerta was chosen as the base, and the language version of the model was adjusted according to the language of the task.

[0105] Keyword extraction techniques were used to identify keywords from samples in the bill text dataset.

[0106] Replace the keywords above with a mask [mask] and input it into the roBerta model as the training dataset to complete the prediction task of the [mask] label.

[0107] Furthermore, the similarity calculation module is specifically used for:

[0108] Based on the Aprior algorithm, we mine frequently occurring tag pairs and obtain the confidence level σ of the frequently occurring tag pairs. Based on this, we set a threshold Δ to filter out tag pairs with high confidence.

[0109] For a given pair of bill texts<T1,T2> and its labels<L1,L2> Define and calculate the similarity between the two:

[0110] Sim(T1, T2)=[len(L1∩L2) / len(L1UL2)]·(∑ i，j σ ij / N i，j (2)

[0111] Where i and j are any two labels in the intersection of the two bill label sets, {i, j | i, j ∈ L1 ∩ L2, σ i，j Sim(T1, T2) represents the similarity between T1 and T2; len(L1∩L2) / len(L1UL2) is the similarity benchmark, which is the ratio of the size of the intersection and the size of the union of the two labels.

[0112] Furthermore, the contrastive learning module is specifically used for:

[0113] Based on the calculated similarity index of the bill text, a loss function for contrastive learning is designed:

[0114] loss=||Cosine(E1, E2)-λ·Sim(T1, T2)|| (3)

[0115] Where Cosine(·) represents the cosine similarity calculation; E1 and E2 represent the two text representations obtained by inputting a pair of legal texts into the fine-tuned roBerta model; λ represents the hyperparameter.

[0116] A pair of legal texts is randomly selected and input into the fine-tuned roBerta model to obtain the representations of the two texts; then, the loss function is calculated according to formula (3) and the loss is backpropagated to continue training the model.

[0117] Furthermore, the heterogeneous graph construction module is specifically used for:

[0118] Construct a heterogeneous graph, using labels, bill initiators, and the bill text itself as nodes in the graph. Design association rules based on the real-world context to connect the nodes and complete the graph construction process.

[0119] Different features are assigned to different nodes. Labels and bill initiators are encoded using one-hot encoding, while bill text nodes are encoded using a roBerta model that has been finely tuned twice.

[0120] Furthermore, the edge prediction module is specifically used for:

[0121] Divide the data into training and testing sets: First, divide the bill nodes into training and testing nodes; then, randomly mask the edges between the bill and different labels with a certain probability to serve as the content for model prediction.

[0122] Training the graph neural network: The heterogeneous graph neural network algorithm is used to predict the edges of the random masking mentioned above to complete the model training;

[0123] Based on a trained graph neural network model, the system predicts whether there is a correlation between bills and label nodes in the test dataset.

[0124] In summary, this invention employs a contrastive learning method to effectively alleviate the model training challenges posed by small sample corpora. It also better achieves semantic embedding representation of sentences, enabling the feature vectors learned by the model to effectively integrate multi-label semantics and the inherent relationships within the labels themselves. The frequent itemset mining method can efficiently mine label dependencies. This invention achieves multi-label bill text classification based on the edge prediction task of heterogeneous graphs, effectively avoiding the label space explosion caused by multi-label combinations. The construction process of heterogeneous graphs can incorporate other effective information, such as information about bill proposers, to help improve the accuracy of multi-label text classification tasks. This invention has strong scalability. First, contrastive learning can be used in cross-modal tasks such as text and images to help overcome modal limitations; second, it is not limited to bill text classification tasks, but can be combined with domain knowledge to construct appropriate heterogeneous graphs for text classification tasks with few samples and multiple labels to complete the bill text classification task.

[0125] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-label classification method for legal texts based on contrastive learning and graph learning, characterized in that, include: Step 1: Based on the trained language model roBerta, continue training on the legal text dataset to adapt to the characteristics of legal texts, thus completing a fine-tuning of the roBerta model. Step 2: Based on the Aprior algorithm, frequent itemsets in the labels are mined to provide a similarity metric for constructing positive and negative examples. ; Step 3: Based on the fine-tuned roBerta model and similarity metrics Example pairs are constructed for comparative learning to complete the secondary fine-tuning of the roBerta model; Step 4: Based on the characteristics of the bill data, construct a heterogeneous graph by combining external data, and use the second-fine-tuned roBerta model for learning the bill text representation as the initial representation of the relevant nodes in the heterogeneous graph; Step 5: Use graph neural network technology to learn node representations; based on this, perform edge prediction between bill and label nodes to complete multi-classification of bill text.

2. The method for multi-label classification of legal texts based on contrastive learning and graph learning according to claim 1, characterized in that, Step 1 includes: Step 1.1: Select the pre-trained language model roBerta as the base. The language version of this model is adjusted according to the language of the task. Step 1.2: Use keyword extraction technology to identify keywords from the samples in the bill text dataset; Step 1.3: Replace the keywords above with a mask [mask] and input it into the roBerta model as the training dataset to complete the prediction task of the [mask] label.

3. The method for multi-label classification of legal texts based on contrastive learning and graph learning according to claim 1, characterized in that, Step 2 includes: Step 2.1: Based on the Aprior algorithm, mine frequently occurring label pairs in the labels and obtain the confidence level σ of the frequently occurring label pairs. On this basis, set a threshold Δ to filter out label pairs with high confidence. Step 2.2, for a given pair of bill texts and its labels Define and calculate the similarity between the two: （2） in and It is any two tags in the intersection of the tag sets of the two bills. , express and The similarity between them; The similarity benchmark is the ratio of the size of the intersection to the size of the union of the two labels.

4. The method for multi-label classification of legal texts based on contrastive learning and graph learning according to claim 3, characterized in that, Step 3 includes: Step 3.1: Based on the calculated similarity index of the bill texts, design the loss function for contrastive learning. : （3） in This indicates the calculation of cosine similarity. This represents the two texts obtained when a pair of legal texts are input into a fine-tuned roBerta model. Indicates hyperparameters; Step 3.2: Randomly select a pair of legal texts and input them into the fine-tuned roBerta model to obtain the representations of the two texts; then, calculate the loss function according to formula (3) and backpropagate the loss to continue training the model.

5. The method for multi-label classification of legal texts based on contrastive learning and graph learning according to claim 1, characterized in that, Step 4 includes: Step 4.1: Construct a heterogeneous graph, using the tags, bill initiators, and the bill text itself as nodes in the graph. Design association rules based on the real-world context to connect the nodes and complete the graph construction process. Step 4.2 assigns different feature representations to different nodes, where the labels and bill initiators are encoded using one-hot encoding, and the bill text nodes are encoded using the roBerta model after secondary fine-tuning.

6. The method for multi-label classification of legal texts based on contrastive learning and graph learning according to claim 5, characterized in that, Step 5 includes: Step 5.1, Divide the training set and test set: First, divide the bill nodes to obtain training nodes and test nodes; then, randomly mask the edges between the bill and different labels according to a certain probability, so as to use the content predicted by the model. Step 5.2, train the graph neural network: use the heterogeneous graph neural network algorithm to predict the edges of the random masking mentioned above, and complete the model training; Step 5.3: Based on the trained graph neural network model, predict whether there is a correlation between bills and label nodes in the test dataset.

7. A multi-label classification system for legal texts based on contrastive learning and graph learning, characterized in that, include: The adaptive training module is used to further train the roBerta model on the legal text dataset to adapt to the characteristics of legal texts, thus completing a fine-tuning of the roBerta model. The similarity calculation module is used to mine frequent itemsets in tags based on the Aprior algorithm, providing a similarity metric for constructing positive and negative examples. ; The contrastive learning module is used to learn based on the fine-tuned roBerta model and similarity metrics. Example pairs are constructed for comparative learning to complete the secondary fine-tuning of the roBerta model; The heterogeneous graph construction module is used to construct a heterogeneous graph based on the characteristics of the bill data and combined with external data, and to use the roBerta model after secondary fine-tuning for learning the bill text representation as the initial representation of the relevant nodes in the heterogeneous graph. The edge prediction module uses graph neural network technology to learn node representations; based on this, it performs edge prediction between bill and label nodes to complete multi-classification of bill text.

8. A multi-label classification system for legal texts based on contrastive learning and graph learning according to claim 7, characterized in that, The adaptive training module is specifically used for: The pre-trained language model roBerta was chosen as the base, and the language version of the model was adjusted according to the language of the task. Keyword extraction techniques were used to identify keywords from samples in the bill text dataset. Replace the keywords above with a mask [mask] and input it into the roBerta model as the training dataset to complete the prediction task of the [mask] label.

9. A multi-label classification system for legal texts based on contrastive learning and graph learning according to claim 7, characterized in that, The similarity calculation module is specifically used for: Based on the Aprior algorithm, we mine frequently occurring tag pairs and obtain the confidence level σ of the frequently occurring tag pairs. Based on this, we set a threshold Δ to filter out tag pairs with high confidence. For a given pair of bill texts and its labels Define and calculate the similarity between the two: （2） in and It is any two tags in the intersection of the tag sets of the two bills. , express and The similarity between them; The similarity benchmark is the ratio of the size of the intersection to the size of the union of the two labels.

10. A multi-label classification system for legal texts based on contrastive learning and graph learning according to claim 9, characterized in that, The contrastive learning module is specifically used for: Based on the calculated similarity index of the bill text, a loss function for contrastive learning is designed. : （3） in This indicates the calculation of cosine similarity. This represents the two texts obtained when a pair of legal texts are input into a fine-tuned roBerta model. Indicates hyperparameters; A pair of legal texts is randomly selected and input into the fine-tuned roBerta model to obtain the representations of the two texts; then, the loss function is calculated according to formula (3) and the loss is backpropagated to continue training the model; Preferably, the heterogeneous graph construction module is specifically used for: Construct a heterogeneous graph, using labels, bill initiators, and the bill text itself as nodes in the graph. Design association rules based on the real-world context to connect the nodes and complete the graph construction process. Different feature representations are assigned to different nodes, with labels and bill initiators using one-hot encoding, and bill text nodes using a roBerta model that has been fine-tuned twice; Preferably, the edge prediction module is specifically used for: Divide the data into training and testing sets: First, divide the bill nodes to obtain training and testing nodes; then, randomly mask the edges between the bill and different labels with a certain probability to serve as the content for model prediction. Training the graph neural network: The heterogeneous graph neural network algorithm is used to predict the edges of the random masking mentioned above to complete the model training; Based on a trained graph neural network model, the system predicts whether there is a correlation between bills and label nodes in the test dataset.

Images