Script event prediction method based on multi-level joint pre-training and prompt fine tuning
By building a BART-large-based encoder-decoder architecture, combining event-level and parameter-level occlusion strategies and multi-level pre-training and fine-tuning of continuous prompt embedding and double-contrast loss functions, the problems of limited knowledge base coverage and insufficient fine-tuning strategies in the existing technology are solved, and the generalization and accuracy of script event prediction are improved.
Patent Information
- Application Number
- CN202510480543.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-17
- Publication Date
- 2025-08-01
AI Technical Summary
The existing script event prediction methods are limited in the knowledge base coverage, poor diversification and adaptability, and the fine-tuning strategy fails to effectively activate common sense knowledge of implicit events in the model, resulting in insufficient generalization and adaptability.
The encoder-decoder architecture based on BART-large is adopted, and pre-trained with event-level and parameter-level occlusion strategies are combined for fine-tuning, and a continuous prompt embedding mechanism and dual contrast loss function are introduced for fine-tuning, to build a prediction model of multi-level joint pre-training and cues fine-tuning.
It significantly improves the model's understanding and reasoning ability of script event chains, reduces external knowledge dependence, and improves generalization and prediction accuracy in complex script scenarios.
Smart Images
Figure CN120409672A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of script event prediction, and particularly to a script event prediction method based on multi-level joint pre-training and prompt fine-tuning. Background Art
[0002] The script event prediction task aims to infer possible subsequent events by analyzing the evolution law of event sequences, which is of great significance for narrative understanding and generation. Traditional methods mainly rely on shallow semantic matching or simple sequence modeling: early studies (such as Event-Comp) learn event vectors through Word2Vec and calculate similarities, but ignore the temporal dependencies between events; subsequent improved methods (such as Pair-LSTM) introduce LSTM to capture the order of event chains, while methods based on narrative event evolution graphs (such as SGNN) model the co-occurrence relationship of events through graph structures, still difficult to deeply mine the internal semantic associations of events.
[0003] In recent years, researchers have tried to enhance the reasoning ability through external knowledge, such as combining the ASER knowledge base or the ATOMIC causal relationship map (such as RoBERTa+Know.Model). However, such methods face two major bottlenecks: (1) the coverage of the knowledge base is limited, making it difficult to adapt to diverse script scenarios, especially when the original text lacks explicit discourse relations; (2) the knowledge injection process usually relies on manually designed alignment rules, which may introduce noise and reduce the model generalization.
[0004] With the rise of pre-trained language models, methods based on event-centered pre-training (such as ClarET, Event-centric+Contrastive) learn event correlations through event-level masking strategies, but there are problems with insufficient fine-grained modeling: existing methods mostly focus on event-level semantic associations and ignore the implicit connections between internal arguments of events (such as subjects, objects).
[0005] In addition, existing fine-tuning strategies mostly adopt discrete prompts with fixed templates (such as "What will happen next?") or traditional classifiers (such as GraphBERT), which fail to effectively activate the implicit event common sense knowledge in the model. Discrete prompts have poor adaptability to script semantic changes, while linear layer-based classifiers are limited by the target differences between pre-training and fine-tuning tasks, restricting the knowledge transfer efficiency. Therefore, there is an urgent need for a multi-level knowledge fusion and dynamic knowledge mining mechanism to overcome the limitations of existing technologies. Summary of the Invention
[0006] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide a script event prediction method based on multi-level joint pre-training and prompt fine-tuning.
[0007] To achieve the above object, the present invention provides the following solutions:
[0008] A script event prediction method based on multi-level joint pre-training and prompt fine-tuning, comprising:
[0009] Constructing a prediction model;
[0010] Obtaining data to be predicted and inputting it into the prediction model to obtain a prediction result;
[0011] Among them, the construction method of the prediction model is:
[0012] Based on BART-large, constructing a model framework;
[0013] Using event-level masking and parameter-level masking strategies to pre-train the model framework to obtain an initial prediction model;
[0014] Fine-tuning the initial prediction model to obtain a final prediction model.
[0015] Preferably, the model framework is an encoder-decoder architecture.
[0016] Preferably, the using event-level masking and parameter-level masking strategies to pre-train the model framework to obtain an initial prediction model includes:
[0017] Constructing an event sequence;
[0018] Based on the event sequence, performing random event masking and conducting the first training to obtain an event masking prediction model;
[0019] Determining unmasked events and performing random parameter masking and using the event masking prediction model for the second training to obtain an initial prediction model.
[0020] Preferably, the fine-tuning the initial prediction model to obtain a final prediction model includes:
[0021] Introducing a continuous prompt embedding mechanism;
[0022] Generating corresponding Gaussian distribution parameters according to the continuous prompt embedding mechanism;
[0023] Determining the mean and variance of the Gaussian distribution parameters and conducting the first model adjustment;
[0024] Performing a reparameterization method on the adjusted model to convert the non-differentiable Gaussian sampling process into a differentiable process for the second model adjustment;
[0025] Defining a dual contrast loss function and optimizing the loss function for the final adjustment to obtain a final prediction model.
[0026] Preferably, defining a dual contrast loss function and optimizing the loss function for final adjustment to obtain a final prediction model, including:
[0027] Defining a scoring function for calculating the overall score of each candidate event;
[0028] Converting the overall score into a probability;
[0029] Defining a dual contrast loss function according to the probability, wherein the dual contrast loss function includes a cross-entropy term and a complementary entropy term;
[0030] Using the cross-entropy term and the complementary entropy term to perform final adjustment of the model to obtain a final prediction model.
[0031] Preferably, the expression of the scoring function is:
[0032]
[0033] where N yi represents the number of tokens in the candidate event y i and S [[ID=2,7]] m represents adding a [mask] token at the end of the context event chain.
[0034] Preferably, the expression of the dual contrast loss function is:
[0035]
[0036] where s t [[ID=3,9]]represents the final score of the correct candidate event, M represents the total number of candidate events, i represents the index of all candidate events, and s i represents the final scores of all candidate events.
[0037] The present invention discloses the following technical effects:
[0038] The present invention provides a script event prediction method based on multi-level joint pre-training and prompt fine-tuning, including: constructing a prediction model; obtaining data to be predicted and inputting it into the prediction model to obtain a prediction result; wherein, the construction method of the prediction model is: based on BART-large, constructing a model framework; using an event-level masking and a parameter-level masking strategy to pre-train the model framework to obtain an initial prediction model; fine-tuning the initial prediction model to obtain a final prediction model. The present invention significantly reduces the dependence on external resources through event-level and parameter-level collaborative modeling, adaptive knowledge mining of continuous prompt embedding, and efficient inference without external knowledge dependence, improves the model's understanding and reasoning ability of script event chains, and provides a more reliable basis for downstream tasks such as content generation and dialogue systems. Brief Description of the Drawings
[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.
[0040] Figure 1 It is a flowchart of a script event prediction method based on multi-level joint pre-training and prompt fine-tuning provided by an embodiment of the present invention;
[0041] Figure 2 It is an architecture diagram of a script event prediction method based on multi-level joint pre-training and prompt fine-tuning provided by an embodiment of the present invention. Detailed Embodiments
[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.
[0043] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the drawings and specific embodiments.
[0044] As Figure 1 shown, the present invention provides a script event prediction method based on multi-level joint pre-training and prompt fine-tuning, including:
[0045] Step 100: Construct a prediction model;
[0046] Step 200: Obtain the data to be predicted and input it into the prediction model to obtain a prediction result;
[0047] Among them, the construction method of the prediction model is:
[0048] Step 201: Based on BART-large, construct a model framework;
[0049] Step 202: Use the event-level masking and parameter-level masking strategies to pre-train the model framework to obtain an initial prediction model;
[0050] Step 203: Fine-tune the initial prediction model to obtain a final prediction model.
[0051] Further, the model framework is an encoder-decoder architecture.
[0052] Specifically, as Figure 2 shown, we use BART-large as the base model to construct an encoder-decoder architecture to process the event sequence. The encoder receives the concatenated input of the context event chain and the candidate event, and the decoder predicts the optimal subsequent event through the generation probability.
[0053] Further, the model framework is pre-trained using event-level masking and parameter-level masking strategies to obtain an initial prediction model, including:
[0054] Construct an event sequence;
[0055] Perform random event masking on the basis of the event sequence and conduct the first training to obtain an event masking prediction model;
[0056] Determine the unmasked events, perform random parameter masking, and use the event masking prediction model to conduct the second training to obtain the initial prediction model.
[0057] Specifically, the model adopts two pre-training strategies in pre-training: one is event-level masking. First, construct an event sequence. For example, concatenate the original script event chain X = {x1, x2,..., x n}, with the correct candidate event y t to form an extended sequence:
[0058] S ori = x1, x2,..., x n , y t
[0059] After that, randomly select several complete events (including the first and last events) in the sequence S ori and replace them with <mask>Mark, and finally require the model to generate masked events. For example, after masking "Clinton attended the rally", the model generates "gave a speech"; the second is parameter-level masking: randomly select its parameters (such as the subject, object) for the unmasked event to be masked and replaced with <arg-mask>(If only the predicate "participate" is retained), then force the model to complete the subject or object. Subsequently, we use this masked context as the encoder input of the BART model, and use the masked part of the original sequence as the decoder input to calculate the cross-entropy loss between the generated tokens and the original labels. The conditional probability distribution P(E|S) is defined as shown in Equation 1:
[0060]
[0061] The combined use of event-level and argument-level masking enables the model to master both the global logic of the event chain and deeply understand the fine-grained semantics within the event, significantly enhancing the model's reasoning ability and generalization in complex script scenarios, while reducing the dependence on external knowledge.
[0062] Furthermore, the fine-tuning of the initial prediction model to obtain the final prediction model includes:
[0063] Introduce a continuous prompt embedding mechanism;
[0064] Generate corresponding Gaussian distribution parameters according to the continuous prompt embedding mechanism;
[0065] Determine the mean and variance of the Gaussian distribution parameters and perform the first model adjustment;
[0066] Perform a reparameterization method on the adjusted model to convert the non-differentiable Gaussian sampling process into a differentiable process for the second model adjustment;
[0067] Define a dual contrast loss function and perform loss function optimization for the final adjustment to obtain the final prediction model.
[0068] Furthermore, the defining of the dual contrast loss function and performing loss function optimization for the final adjustment to obtain the final prediction model includes:
[0069] Define a scoring function for calculating the overall score of each candidate event;
[0070] Convert the overall score into a probability;
[0071] Define a dual contrast loss function according to the probability, where the dual contrast loss function includes a cross-entropy term and a complementary entropy term;
[0072] Use the cross-entropy term and the complementary entropy term for the final adjustment of the model to obtain the final prediction model.
[0073] Specifically, to address the semantic ambiguity caused by diverse script scenarios, we introduce a continuous prompt embedding mechanism parameterized by Gaussian distribution. Gaussian distribution is a symmetric continuous distribution, suitable for representing the continuous variation characteristics of semantic uncertainty. In prompt embedding, the semantic associations of parameters (such as subject and predicate) are usually continuous and symmetric, and the mean and variance of Gaussian distribution can intuitively represent the "semantic center" and "confidence range". The continuous prompt embedding mechanism maps the parameters (such as subject and predicate) of candidate events to a Gaussian distribution with mean μ and variance σ 2 to characterize their semantic uncertainty, allowing the model to dynamically adjust the prompt representation within the soft region. This design enables the model to autonomously capture the potential semantic associations of different scripts, breaking through the fixed pattern limitations of discrete prompts. The four arguments (subject, verb, object) of each candidate event are used to generate the Gaussian parameters corresponding to the prompt tokens:
[0074] The mean μ of the subject prompt token [t j,1 is calculated through scaled dot - product attention, as shown in Equation 2: j,1 where
[0075]
[0076] is the embedding of the candidate event subject, and E is the embedding of the context event chain. The variance σ
[0077] is ensured to be non - negative through an exponential function, as shown in Equation 3: 2
[0078]
[0079] The verb, object, and indirect object prompt tokens are generated in the same way.
[0080] On the other hand, using discrete Gaussian sampling as prompt embedding inevitably leads to non - differentiable parameters, making it impossible to optimize the prompt embedding through backpropagation. To solve this problem, we introduce the re - parameterization trick to convert the non - differentiable Gaussian sampling process into a differentiable one. Specifically, we introduce standard normal noise and allow the sampling position x from the Gaussian distribution to be equivalently represented by the related function of this standard normal noise. For example, Equation 4: [[ID=XXX]]
[0081]
[0082] [t] j,1 = x j,1 = μ j,1 + σ j,1 × ∈ #(4)
[0082] In addition, we also introduce a dual contrast loss function design. First, calculate the score of each candidate event, as shown in Equation 5:
[0083] Note: There seems to be an "XXX" in the original text where the number should be. I've left it as it is in the translation for now. If it's a formatting error in the original, you may need to correct it for a more accurate translation.
[0084] Among them, N yi represents the number of tags in the candidate event y i , and S m represents adding a [mask] tag at the end of the context event chain. Then, through the Softmax function, the candidate event score is converted into a probability s i , as shown in Equation 6:
[0085]
[0086] Finally, the dual contrast loss function is defined as shown in Equation 7:
[0087]
[0088] The contrast loss function includes (1) the cross-entropy term: L CE = -log(s t ) and (2) the complementary entropy term: Among them, the role of the cross-entropy is to maximize the posterior probability s t of the candidate event y t , forcing the model to learn the mapping relationship from the context event chain S to the target event. In the continuous prompting framework, this gradient can be directly backpropagated to the Gaussian parameters (μ, σ), enhancing the semantic focusing ability of the prompt embedding on the correct argument. However, the traditional cross-entropy loss only focuses on maximizing the probability of the correct candidate and does not explicitly constrain the distribution of the wrong candidates. This may cause the model to assign too high a probability to some wrong candidates. Therefore, we introduce a complementary entropy term in the loss function. The complementary entropy forces its distribution to be uniform by maximizing the normalized entropy of the wrong candidates, thereby preventing the model from overfitting individual wrong patterns due to data sparsity. While optimizing the generation probability of the correct candidate, it can indirectly optimize the distribution of the wrong candidates and prevent the model from generating wrong events that do not belong to the candidate events. The design of the dual contrast loss function that combines cross-entropy and complementary entropy is particularly important in the generative script event prediction task. It improves the accuracy and reliability of the model prediction by balancing the probability distributions of the correct and wrong candidates.
[0089] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same and similar parts among the various embodiments can be referred to each other.
[0090] Specific examples are used in this article to elaborate on the principles and implementation methods of the present invention. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention. < / mask>
Claims
1. A script event prediction method based on multi-level joint pre-training and prompt fine-tuning, characterized in that, Including: Construct a prediction model; Obtain the data to be predicted and input it into the prediction model to obtain a prediction result; Among them, the construction method of the prediction model is: Based on BART-large, construct a model framework; Use the event-level masking and parameter-level masking strategies to pre-train the model framework to obtain an initial prediction model; Fine-tune the initial prediction model to obtain a final prediction model.
2. The script event prediction method based on multi-level joint pre-training and prompt fine-tuning according to claim 1, wherein The model framework is an encoder-decoder architecture.
3. A script event prediction method based on multi-level joint pre-training and prompt fine-tuning according to claim 1, characterized in that The use of the event-level masking and parameter-level masking strategies to pre-train the model framework to obtain an initial prediction model includes: Construct an event sequence; Perform random event masking based on the event sequence and conduct the first training to obtain an event masking prediction model; Determine the unmasked events and perform random parameter masking and use the event masking prediction model to conduct the second training to obtain an initial prediction model.
4. A script event prediction method based on multi-level joint pre-training and prompt fine-tuning according to claim 1, characterized in that, The fine-tuning of the initial prediction model to obtain a final prediction model includes: Introduce a continuous prompt embedding mechanism; Generate corresponding Gaussian distribution parameters according to the continuous prompt embedding mechanism; Determine the mean and variance of the Gaussian distribution parameters and conduct the first model adjustment; Perform a reparameterization method on the adjusted model to convert the non-differentiable Gaussian sampling process into a differentiable process for the second model adjustment; Define a dual contrast loss function and optimize the loss function for the final adjustment to obtain a final prediction model.
5. A script event prediction method based on multi-level joint pre-training and prompt fine-tuning according to claim 4, characterized in that The definition of the dual contrast loss function and the optimization of the loss function for the final adjustment to obtain a final prediction model includes: Define a scoring function for calculating the overall score of each candidate event; Convert the overall score into a probability; Define a dual contrast loss function according to the probability, where the dual contrast loss function includes a cross-entropy term and a complementary entropy term; Use the cross-entropy term and the complementary entropy term for the final adjustment of the model to obtain a final prediction model.
6. The script event prediction method based on multi-level joint pre-training and prompt fine-tuning according to claim 5, characterized in that The expression of the scoring function is: Among them, N yi represents the number of tokens in the candidate event y i , and S m represents adding a [mask] token at the end of the context event chain.
7. A script event prediction method based on multi-level joint pre-training and prompt fine-tuning according to claim 6, characterized in that The expression of the dual contrast loss function is: Among them, s t represents the final score of the correct candidate event, M represents the total number of candidate events, i represents the index of all candidate events, and s i represents the final scores of all candidate events.