Legal event detection method based on big language model and active learning cooperation

Abstract

The invention discloses a legal event detection method based on cooperation of a large language model and active learning, and belongs to the technical field of natural language processing and artificial intelligence. According to the method, on the basis of a small amount of manual annotation data, an active learning mechanism is introduced, samples with high information amount are dynamically selected from unannotated legal texts, and the selected samples are automatically annotated by using a large language model subjected to efficient fine adjustment of parameters. A multi-candidate generation and data quality evaluation screening mechanism is designed for the problems of uncertainty and noise existing in an automatic labeling result, and high-reliability labeling data is screened out to participate in model training by carrying out joint constraint on the confidence coefficient and the distinction degree of the labeling result. By means of the mode, iterative collaborative optimization of the event detection model and the annotation data is achieved, the manual annotation cost is remarkably reduced under the low-resource condition, meanwhile, the accuracy and stability of legal event detection are improved, and high practical value is achieved.

Description

A Legal Event Detection Method Based on the Collaboration of Large Language Model and Active Learning Technical Field

[0001] This invention belongs to the field of natural language processing and artificial intelligence technology, specifically relating to a legal event detection method based on the collaboration of a large language model and active learning. Background Technology

[0002] Legal event detection is a crucial foundational task in intelligent legal text analysis. Its goal is to automatically identify event trigger words and determine the corresponding event type from texts such as judgments and legal documents. This task plays a vital supporting role in downstream legal fact-finding, judicial decision support, and legal knowledge modeling. However, in practical applications, legal event types are complex and diverse, and high-quality labeled data heavily relies on human experts, resulting in high labeling costs and long cycles, making it difficult to meet the needs of rapid model iteration and large-scale application.

[0003] Existing methods primarily rely on supervised learning or traditional active learning frameworks. Supervised learning is heavily dependent on large-scale manually labeled data, while traditional active learning, although it can reduce the number of labeled samples to some extent, still requires manual annotation in its core stages, resulting in high overall costs. With the development of large language models, they have demonstrated strong capabilities in semantic understanding and text generation, providing new technical approaches for automated data annotation. However, directly using large language models to generate labeled data still faces problems such as insufficient annotation consistency and high noise levels, limiting their reliable application in legal event detection tasks. Therefore, there is an urgent need for a legal event detection method that can fully leverage the capabilities of large language models while ensuring annotation quality and further reducing the cost of manual annotation. Summary of the Invention

[0004] The purpose of this invention is to address the aforementioned deficiencies or problems by proposing a legal event detection method based on the collaboration of a large language model and active learning. By introducing a large language model as an automatic annotator into the active learning framework, and combining it with a data quality assessment and screening mechanism, the method effectively reduces the need for manual annotation under low-resource conditions, while improving the overall performance and stability of the legal event detection model.

[0005] Technical Solution: To address the aforementioned technical problems, this invention proposes a legal event detection method based on the collaboration of a large language model and active learning. This method includes the following:

[0006] Step 1) Construct the initial labeled dataset: Manually label event detection data samples from legal text corpora, identify keywords in legal texts that can directly represent the occurrence of events as event trigger words, and assign them corresponding event type labels as initial training data;

[0007] Step 2) Training the large language model labeler: Use the initial training data to adjust the parameters of the large language model and train it to automatically label legal event trigger words and event type classification;

[0008] Step 3) Sample ranking based on active learning: The event detection model is initially trained using the initial training data. In the unlabeled legal text data pool, the event detection model is used to rank the unlabeled legal texts using a query strategy.

[0009] Step 4) Automatic labeling and multiple candidate generation: The large language model labeler is used to automatically label the sorted unlabeled samples, and multiple candidate labeling results are obtained through a multi-path generation mechanism;

[0010] Step 5) Data quality assessment and screening: Based on the confidence and discrimination of the candidate annotation results, the automatically labeled data is screened, and samples that meet the quality constraints are retained;

[0011] Step 6) Model update and iterative training: Add the selected automatically labeled samples to the training set, update and train the event detection model, and repeat steps 3) to 6) until the preset iteration conditions are met.

[0012] Furthermore, the specific method for step 1) is as follows:

[0013] The task involves labeling event-triggered words and their event types within a given text describing facts. The event types to be labeled are from a pre-defined, finite set. This event detection task can be viewed as a sequence labeling task, given a sequence of input text. The output is , that is, each element Marked as ,in, The values ​​are “BT”, “IT”, or “O”; where “BT” indicates that the element belongs to event type T and is at the beginning of the segment, “IT” indicates that the element belongs to event type T and is in the middle of the segment, and “O” indicates that it does not belong to any event type.

[0014] Furthermore, the specific method for step 2) is as follows:

[0015] The parameter fine-tuning mechanism adopts a low-rank adaptive adjustment method. Under the premise of freezing the main parameters of the large language model, the model weights are updated through the low-rank matrix to reduce training costs and improve domain adaptation efficiency. It is assumed that the update of the model weight matrix is ​​performed through low-rank decomposition, that is, the weight update is represented as the product of two low-rank matrices, which effectively reduces the number of trainable parameters.

[0016]

[0017] in, It is an update of the model weight matrix. and Is the rank as The matrix, It is a constant scaling factor, applied only to [specific parameters] during training. Optimization is performed, and the model weight matrix... Remain unchanged; among which, It is randomly initialized, and Then it is initialized to zero, and at the start of training, the model state satisfies The parameters are consistent with those of the original model.

[0018] Furthermore, the specific method for step 3) is as follows:

[0019] The event detection model is built based on the pre-trained bidirectional coding representation model BERT. The initial manually labeled data constructed in step 1) is used as a supervision signal for initial training. It is used to identify event trigger words and predict event types in unlabeled legal texts. The output prediction probability distribution serves as the basis for active learning query strategy calculation and uncertainty measurement.

[0020] The query strategies include uncertainty-based strategies and randomized strategies. Uncertainty-based strategies measure the uncertainty of samples by predicting probability distributions using a model; the degree of uncertainty represents the priority of model learning. For unlabeled legal text sequences... and the output of the event detection model The specific query strategies are as follows:

[0021] Random (RD): Ranking samples in unannotated legal texts in a random manner, without considering model predictions;

[0022] Breaking Ties (BT): Based on the predicted probability distribution output by the event detection model, it characterizes the prediction uncertainty by measuring the difference between the two labels with the highest predicted probabilities. Its uncertainty metric function is defined as follows:

[0023]

[0024] in, and These are the most likely label and the second most likely label, respectively. The smaller the difference, the higher the uncertainty of the model for that prediction unit.

[0025] Least Confidence (LC): Based on the predicted probability distribution output by the event detection model, it characterizes uncertainty by measuring the model's confidence in predicting the most likely label. Its uncertainty metric function is defined as follows:

[0026]

[0027] in, It is the most likely label; the lower the prediction confidence, the higher the uncertainty.

[0028] Prediction Entropy (PE): Based on the predicted probability distribution output by the event detection model, the uncertainty of the model is measured by calculating the information entropy of the predicted label distribution. Its uncertainty metric function is defined as follows:

[0029]

[0030] in, This represents the total number of categories for the predicted label. The higher the entropy value, the greater the uncertainty of the model for that prediction unit.

[0031] For legal text sentences Multiple prediction units included The corresponding uncertainty measurement results are aggregated to obtain sentence-level uncertainty scores, preferably using the maximum value aggregation method:

[0032]

[0033] in, for , and Based on any of these methods, all unannotated legal texts are sorted according to sentence-level uncertainty scores, forming a set of sentences arranged from highest to lowest uncertainty. .

[0034] Furthermore, the specific method for step 5) is as follows:

[0035] For the set of sentences selected by the query strategy in step 4 ,against Each text sequence in A candidate annotation set is generated using a large language model annotator and a beam search algorithm. , Represented as a large language model annotator in The first generated above There are 10 candidate labels, and each candidate label is a complete corresponding output. , Candidate labels The score, based on the score The scores, The annotation results are arranged by score Sort in descending order of size, i.e. Among them, candidate annotation score The calculation is as follows:

[0036]

[0037]

[0038] in, It is the conditional probability output by the generative model;

[0039] When the highest confidence score of a candidate label Exceeding the predefined threshold And the highest score With the second highest score The difference between them exceeds another threshold When this condition is met, the annotation is considered valid, and the specific filtering rules are as follows:

[0040]

[0041] After screening It is ultimately from The selected sentence set, the number of labeled sentences required for the current training iteration is equal to The number of sentences in the data; if the currently selected labeled data does not meet the filtering rules, continue to select samples from the remaining data for labeling until the training data required for the next iteration is obtained.

[0042] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0043] (1) By introducing large language models into the active learning framework, the reliance on manual annotation is significantly reduced, and the cost of constructing legal event detection data is lowered;

[0044] (2) Through the multi-candidate generation and quality screening mechanism, the automatic annotation noise is effectively suppressed, and the reliability and consistency of the training data are improved;

[0045] (3) In the scenario of detecting legal events with low resources, it can achieve detection performance close to supervised learning under the condition of no human supervision, and has good practical value and promotion prospects. Attached Figure Description

[0046] Figure 1 is a flowchart of the method.

[0047] Figure 2 shows the learning curves of different query strategies and evaluation metrics on the LEVEN dataset.

[0048] Figure 3 shows the learning curves of different query strategies and evaluation metrics on the ACE05-C dataset. Detailed Implementation

[0049] The ALLED legal event detection model, based on a large language model and active learning collaboration, was validated using the following experimental and results analysis process.

[0050] As shown in Figure 1, this invention proposes a legal event detection method based on the collaboration of a large language model and active learning. The method includes the following:

[0051] Step 1) Construct the initial labeled dataset: Manually label event detection data samples from legal text corpora, identify keywords in legal texts that can directly represent the occurrence of events as event trigger words, and assign them corresponding event type labels as initial training data;

[0052] Step 2) Training the large language model labeler: Use the initial training data to adjust the parameters of the large language model and train it to automatically label legal event trigger words and event type classification;

[0053] Step 3) Sample ranking based on active learning: The event detection model is initially trained using the initial training data. In the unlabeled legal text data pool, the event detection model is used to rank the unlabeled legal texts using a query strategy.

[0054] Step 4) Automatic labeling and multiple candidate generation: The large language model labeler is used to automatically label the sorted unlabeled samples, and multiple candidate labeling results are obtained through a multi-path generation mechanism;

[0055] Step 5) Data quality assessment and screening: Based on the confidence and discrimination of the candidate annotation results, the automatically labeled data is screened, and samples that meet the quality constraints are retained;

[0056] Step 6) Model update and iterative training: Add the selected automatically labeled samples to the training set, update and train the event detection model, and repeat steps 3) to 6) until the preset iteration conditions are met.

[0057] Furthermore, the specific method for step 1) is as follows:

[0058] The task involves labeling event-triggered words and their event types within a given text describing facts. The event types to be labeled are from a pre-defined, finite set. This event detection task can be viewed as a sequence labeling task, given a sequence of input text. The output is , that is, each element Marked as ,in, The values ​​are “BT”, “IT”, or “O”; where “BT” indicates that the element belongs to event type T and is at the beginning of the segment, “IT” indicates that the element belongs to event type T and is in the middle of the segment, and “O” indicates that it does not belong to any event type.

[0059] Furthermore, the specific method for step 2) is as follows:

[0060] The parameter fine-tuning mechanism adopts a low-rank adaptive adjustment method. Under the premise of freezing the main parameters of the large language model, the model weights are updated through the low-rank matrix to reduce training costs and improve domain adaptation efficiency. It is assumed that the update of the model weight matrix is ​​performed through low-rank decomposition, that is, the weight update is represented as the product of two low-rank matrices, which effectively reduces the number of trainable parameters.

[0061]

[0062] in, It is an update of the model weight matrix. and Is the rank as The matrix, It is a constant scaling factor, applied only to [specific parameters] during training. Optimization is performed, and the model weight matrix... Remain unchanged; among which, It is randomly initialized, and Then it is initialized to zero, and at the start of training, the model state satisfies The parameters are consistent with those of the original model.

[0063] Furthermore, the specific method for step 3) is as follows:

[0064] The event detection model is built based on the pre-trained bidirectional coding representation model BERT. The initial manually labeled data constructed in step 1) is used as a supervision signal for initial training. It is used to identify event trigger words and predict event types in unlabeled legal texts. The output prediction probability distribution serves as the basis for active learning query strategy calculation and uncertainty measurement.

[0065] The query strategies include uncertainty-based strategies and randomized strategies. Uncertainty-based strategies measure the uncertainty of samples by predicting probability distributions using a model; the degree of uncertainty represents the priority of model learning. For unlabeled legal text sequences... and the output of the event detection model The specific query strategies are as follows:

[0066] Random (RD): Ranking samples in unannotated legal texts in a random manner, without considering model predictions;

[0067] Breaking Ties (BT): Based on the predicted probability distribution output by the event detection model, it characterizes the prediction uncertainty by measuring the difference between the two labels with the highest predicted probabilities. Its uncertainty metric function is defined as follows:

[0068]

[0069] in, and These are the most likely label and the second most likely label, respectively. The smaller the difference, the higher the uncertainty of the model for that prediction unit.

[0070] Least Confidence (LC): Based on the predicted probability distribution output by the event detection model, it characterizes uncertainty by measuring the model's confidence in predicting the most likely label. Its uncertainty metric function is defined as follows:

[0071]

[0072] in, It is the most likely label; the lower the prediction confidence, the higher the uncertainty.

[0073] Prediction Entropy (PE): Based on the predicted probability distribution output by the event detection model, the uncertainty of the model is measured by calculating the information entropy of the predicted label distribution. Its uncertainty metric function is defined as follows:

[0074]

[0075] in, This represents the total number of categories for the predicted label. The higher the entropy value, the greater the uncertainty of the model for that prediction unit.

[0076] For legal text sentences Multiple prediction units included The corresponding uncertainty measurement results are aggregated to obtain sentence-level uncertainty scores, preferably using the maximum value aggregation method:

[0077]

[0078] in, for , and Based on any of these methods, all unannotated legal texts are sorted according to sentence-level uncertainty scores, forming a set of sentences arranged from highest to lowest uncertainty. .

[0079] Furthermore, the specific method for step 5) is as follows:

[0080] For the set of sentences selected by the query strategy in step 4 ,against Each text sequence in A candidate annotation set is generated using a large language model annotator and a beam search algorithm. , Represented as a large language model annotator in The first generated above There are 10 candidate labels, and each candidate label is a complete corresponding output. , Candidate labels The score, based on the score The scores, The annotation results are arranged by score Sort in descending order of size, i.e. Among them, candidate annotation score The calculation is as follows:

[0081]

[0082]

[0083] in, It is the conditional probability output by the generative model;

[0084] When the highest confidence score of a candidate label Exceeding the predefined threshold And the highest score With the second highest score The difference between them exceeds another threshold When this condition is met, the annotation is considered valid, and the specific filtering rules are as follows:

[0085]

[0086] After screening It is ultimately from The selected sentence set, the number of labeled sentences required for the current training iteration is equal to The number of sentences in the data; if the currently selected labeled data does not meet the filtering rules, continue to select samples from the remaining data for labeling until the training data required for the next iteration is obtained.

[0087] The experiments were conducted using the widely used event detection datasets ACE05-C and LEVEN. The statistical results of the datasets are shown in Table 1.

[0088] Table 1. Statistical information of the event detection dataset

[0089]

[0090] To verify the effectiveness of the legal event detection model based on the collaboration of a large language model and active learning, it was compared with the following methods on the above dataset. The comparison results are shown in Table 2. The model is described in detail below:

[0091] (1) Context Learning: Context learning was performed using large models with a small number of samples, with examples randomly selected from the training set. Since the examples did not cover all possible event types, the results of applying the LoRA parameter fine-tuning mechanism to these large language models were also compared in the experiment. For model selection, the following mainstream large models were chosen for testing:

[0092] ChatGPT: A closed-source large language model released by OpenAI, possessing powerful language understanding and generation capabilities, and widely used in various natural language processing tasks.

[0093] Qwen2.5-32B and Qwen2.5-7B: Large-scale open-source language models launched by Alibaba Cloud, with 32 billion and 7 billion parameters respectively. They perform well in multiple language tasks, combining performance and resource efficiency.

[0094] InternLM2-7B: Developed by the Shusheng Puyu team, this is an open-source second-generation large language model designed and optimized specifically for domestic enterprise-level use cases. It has 7 billion parameters and aims to improve language understanding and generation capabilities.

[0095] Baichuan2-7B is Baichuan Intelligence's second-generation open-source large language model, boasting 7 billion parameters. Trained on a high-quality corpus of 2.6 trillion tokens, it achieves leading performance for its size across multiple Chinese and English baselines.

[0096] (2) Supervised fine-tuning: The model provided in LEVEN is used for fine-tuning. The proportion of training data used corresponds to the proportion of data used in ALLED, i.e., 30% of the training data. In addition, the results of supervised fine-tuning using the complete data are provided as reference values.

[0097] Classification: The input sentence is encoded using deep neural networks, including pre-trained language models BERT and RoBERTa. After encoding, the hidden representation of each candidate trigger word is extracted and fed into the classification layer to predict its corresponding event type. This process is essentially a token-level multi-class classification task designed to identify all potential event trigger words and their types in the text.

[0098] Sequence labeling: Sequence labeling methods are introduced to more effectively model the contextual dependencies and relationships between different event-triggered words in a sentence. By overlaying a conditional random field layer on top of the encoder, the model can capture structural information between label sequences, thereby improving overall labeling performance.

[0099] (3) Use a random strategy: The random strategy is the simplest query strategy in active learning.

[0100] In the evaluation process, we adopted the evaluation criteria commonly used in event detection tasks: if the offset of the trigger word matches that of the reference trigger word (Trig-I), the trigger word is correctly identified. If the event type of the trigger word matches that of the reference trigger word (Trig-C), the trigger word is correctly classified. We report the F1 scores for Trig-I and Trig-C. All experiments were repeated using different random seeds, and the results are presented as the final mean and standard deviation.

[0101] Table 2 Experimental results for different baselines and ALLED

[0102]

[0103] The results show that the ALLED method achieves excellent results across multiple settings, especially on the LEVEN dataset, where the BT strategy outperforms most baseline models. On the ACE05-C dataset, the BT strategy also achieves the highest performance in these comparisons, further validating the positive impact of active learning strategies on model performance. However, when using all training data, ALLED still lags behind, indicating that active learning has a significant advantage in scenarios with limited high-quality data, while traditional supervised learning remains highly competitive when data is abundant.

[0104] Figures 2 and 3 show the learning curves for different datasets, query strategies, and evaluation metrics, respectively. Overall, all three uncertainty-based query strategies are more effective than randomly selecting samples to train the model. This result verifies that in the active learning framework, prioritizing samples with higher information content for training can indeed improve model performance more efficiently.

[0105] Those skilled in the art will readily understand the embodiments of this invention. The above descriptions are merely examples of basic implementations of this invention and are not intended to limit the scope of the invention.

Claims

1. A legal event detection method based on the collaboration of a large language model and active learning, characterized in that, The method includes the following steps: Step 1) Constructing an initial labeled dataset: Manually labeling event detection data samples from legal text corpora, identifying keywords that directly represent the occurrence of events in the legal text as event trigger words, and assigning them corresponding event type labels as initial training data; Step 2) Training a large language model labeler: Using the initial training data, adjusting the parameters of the large language model, and automatically labeling and training it for legal event trigger word recognition and event type classification; Step 3) Sample ranking based on active learning: Initially training the event detection model using the initial training data, and using the unlabeled legal text data pool to sort the event... The event detection model sorts unlabeled legal texts using a query strategy; Step 4) Automatic labeling and multi-candidate generation: The sorted unlabeled samples are automatically labeled using a large language model labeler, and multiple candidate labeling results are obtained through a multi-path generation mechanism; Step 5) Data quality assessment and screening: Based on the confidence and discriminative power of the candidate labeling results, the automatically labeled data is screened, and samples that meet the quality constraints are retained; Step 6) Model update and iterative training: The automatically labeled samples after screening are added to the training set, the event detection model is updated and trained, and steps 3) to 6) are repeated until the preset iteration conditions are met.

2. The legal event detection method based on the collaboration of a large language model and active learning as described in claim 1, characterized in that, Step 1) is specifically described as follows: In a given factual description text, identify the event trigger words and their event types. The event types to be identified are from a pre-defined, finite set. Treat the event detection task as a sequence labeling task; for a given input text sequence… The output is , that is, each element Marked as ,in, The values ​​are "BT", "IT", or "O". "BT" indicates that the element belongs to event type T and is at the beginning of the segment, "IT" indicates that the element belongs to event type T and is in the middle of the segment, and "O" indicates that it does not belong to any event type.

3. The legal event detection method based on the collaboration of a large language model and active learning as described in claim 2, characterized in that, The specific method for step 2) is as follows: The parameter fine-tuning mechanism adopts a low-rank adaptive adjustment method. Under the premise of freezing the main parameters of the large language model, the model weights are updated through the low-rank matrix. It is assumed that the update of the model weight matrix is ​​carried out through low-rank decomposition, that is, the weight update is represented as the product of two low-rank matrices. ;in, It is an update of the model weight matrix. and Is the rank as The matrix, It is a constant scaling factor, applied only to [specific parameters] during training. Optimize the model weight matrix Remain unchanged; among which, It is randomly initialized, and Then it is initialized to zero, and at the start of training, the model state satisfies The parameters are consistent with those of the original model.

4. The legal event detection method based on the collaboration of a large language model and active learning as described in claim 3, characterized in that, The specific method for step 3) is as follows: The event detection model is built based on the pre-trained bidirectional encoding representation model BERT. The initial manually labeled data constructed in step 1) is used as the supervision signal for initial training. This data is used to identify event trigger words and predict event types in unlabeled legal text. The output prediction probability distribution serves as the basis for calculating the active learning query strategy and measuring uncertainty. The query strategy includes uncertainty-based strategies and random strategies. The uncertainty-based strategy measures the uncertainty of the sample through the model's predicted probability distribution, and the degree of uncertainty represents the priority of model learning. For unlabeled legal text sequences... and the output of the event detection model The specific query strategies are as follows: Random (RD): Randomly sorts samples in unlabeled legal text without considering model prediction results; Breaking Ties (BT): Based on the predicted probability distribution output by the event detection model, it characterizes the prediction uncertainty by measuring the difference between the two labels with the highest predicted probabilities. Its uncertainty measurement function is defined as: ;in, and These are the most likely label and the second most likely label, respectively. The smaller the difference, the higher the uncertainty of the model for that prediction unit. Least Confidence (LC): Based on the predicted probability distribution output by the event detection model, uncertainty is characterized by measuring the model's confidence in predicting the most likely label. Its uncertainty metric function is defined as: ;in, The most likely label is the prediction confidence level; the lower the prediction confidence level, the higher the uncertainty. Prediction Entropy (PE): Based on the predicted probability distribution output by the event detection model, the uncertainty of the model is measured by calculating the information entropy of the predicted label distribution. Its uncertainty measurement function is defined as: ;in, This represents the total number of categories for the predicted label. A higher entropy value indicates greater uncertainty in the model regarding that predicted unit. For legal text sentences... Multiple prediction units included The corresponding uncertainty measurement results are then aggregated to obtain sentence-level uncertainty scores. The maximum aggregation method used is as follows: ;in, for 、 and Based on any of these methods, all unannotated legal texts are sorted according to sentence-level uncertainty scores, forming a set of sentences arranged from highest to lowest uncertainty. 。 5. The legal event detection method based on the collaboration of a large language model and active learning as described in claim 4, characterized in that, The specific method for step 5) is as follows: For the set of sentences selected in step 4 using the query strategy... ,against Each text sequence in A candidate annotation set is generated using a large language model annotator and a beam search algorithm. ， Represented as a large language model annotator in The first generated above There are 10 candidate labels, and each candidate label is a complete corresponding output. ， Candidate labels The score, based on the score The scores, The annotation results are arranged by score Sort in descending order of size, i.e. Among them, candidate annotation score The calculation is as follows: ； ;in, It is the conditional probability output by the generative model; the highest confidence score of a candidate label. Exceeding the predefined threshold And the highest score With the second highest score The difference between them exceeds another threshold When this condition is met, the annotation is considered valid, and the specific filtering rules are as follows: After screening, It is ultimately from The selected sentence set, the number of labeled sentences required for the current training iteration is equal to The number of sentences in the data; if the currently selected labeled data does not meet the above filtering rules, then continue to select samples from the remaining data for labeling until the training data required for the next iteration is obtained.

Images