A method for generating limited domain restricted text based on meta-learning diffusion model

Abstract

This invention relates to a domain-restricted text generation method based on a meta-learning diffusion model, belonging to the fields of artificial intelligence, big data, and natural language processing. This invention constructs a large number of training samples by extracting keywords and randomly inserting specific placeholder markers [EMP] into the original sentence; it uses local diffusion to fix the keywords, enabling the diffusion model to complete the Hard-CTG task, and introduces a meta-learning strategy to optimize the diffusion model training process; another special marker [SLOT] is introduced to indicate the insertion position of the MDM. During the back-diffusion process, the position marked by [SLOT] is gradually transformed from standard Gaussian noise into an actual word or [EMP], thus obtaining generated text containing keyword input. This invention enables the diffusion model to leverage its powerful generative advantages in the field of conditional text generation.

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence, big data and natural language processing, and specifically relates to a method for generating limited domain restricted text based on a meta-learning diffusion model. Background Art

[0002] Constrained text generation refers to a type of task that generates output text that satisfies some constraints given some input text and some constraints. In real life, constrained text generation tasks have a wide range of application scenarios, such as automatic news release writing, automatic summary generation, intelligent dialogue systems, etc. Considering that in many real-world scenarios, text generation needs to retain pre-given keywords in the generated text, such as retaining specific keywords in meeting minutes, generating product descriptions based on specified products, etc.

[0003] The present invention proposes a diffusion model integrating meta-learning to complete the hard-constrained constrained text generation (Hard-CTG) task, which achieves the control of the consistency of keyword field information while generating text content with high fluency, and can meet various text generation scenarios requiring high accuracy, such as meeting minutes generation and news release writing. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The technical problem to be solved by the present invention is how to provide a method for generating limited domain restricted text based on a meta-learning diffusion model, which mainly solves two problems. One is to study the text generation mechanism to reduce the occurrence of transmission errors in order to address the problem of insufficient fluency of generated text; the other is to study the hard constraint construction method to control the consistency of generated text domain through keyword domain information.

[0006] (II) Technical solution

[0007] In order to solve the above technical problems, the present invention proposes a method for generating limited domain restricted text based on a meta-learning diffusion model, which comprises the following steps:

[0008] Step 1: In the training data construction phase, a large number of training samples are constructed by extracting keywords and randomly inserting specific placeholder tags [EMP] into the original sentences;

[0009] Step 2: In the model training stage, local diffusion is used to fix keywords so that the diffusion model can complete the Hard-CTG task, and a meta-learning strategy is introduced to optimize the diffusion model training process;

[0010] Step 3. Finally, in the model inference phase, a special marker [SLOT] is used to indicate the position where the Meta Diffusion Model (MDM) is inserted. During the reverse diffusion process, the position marked by [SLOT] is gradually converted from standard Gaussian noise to an actual word or [EMP], thereby obtaining a generated text containing the keyword input;

[0011] Given a set of keywords C = {c1, c2, ..., c K}, where K represents the number of keywords; the goal of the Hard-Constrained Text Generation (Hard-CTG) task is to generate a sentence S = {s1, s2..., s n}, so that S contains all the keywords in C, where n, n ≥ K represents the length of the generated sentence.

[0012] (III) Beneficial effects

[0013] The present invention proposes a method for generating limited domain restricted text based on a meta-learning diffusion model. The present invention discloses a method for generating limited domain restricted text based on a meta-learning diffusion model, and its main advantages are reflected in the following aspects:

[0014] (1) We designed a diffusion model that integrates meta-learning to complete the Hard-CTG task, and then migrated the diffusion model from the image field that processes continuous data to the text field that processes discrete data. This allows the diffusion model to play its powerful generative advantages in the field of conditional text generation.

[0015] (2) A local diffusion mechanism is designed to fix keywords during the diffusion process, so that the diffusion model can retain keywords and the domain information they contain in text generation and generate contexts of variable length based on the given keywords.

[0016] (3) The meta-learning idea is introduced to optimize the model training process, and an adaptive denoising mode is designed to smooth the convergence process of denoising and improve the fluency and readability of the generated text. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the overall framework of the model of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0019] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0020] The present invention proposes a method for generating limited domain restricted text based on a meta-diffusion model (MDM), wherein the overall framework of MDM is as follows: Figure 1 As shown, in general, MDM consists of three different stages, namely, training data construction stage, model training stage and model inference stage.

[0021] Step 1: In the training data construction phase, this method constructs a large number of training samples by extracting keywords and randomly inserting specific placeholder tags [EMP] into the original sentences.

[0022] Step 2: During the model training phase, this method uses local diffusion to fix keywords, so that the diffusion model can complete the Hard-CTG task, and introduces a meta-learning strategy to optimize the diffusion model training process.

[0023] Step 3. Finally, in the model inference stage, this method uses a special marker [SLOT] to indicate the location where the MDM is inserted. During the reverse diffusion process, the location marked by [SLOT] will gradually be converted from standard Gaussian noise to an actual word or [EMP], thereby obtaining a generated text containing the keyword input.

[0024] Given a set of keywords C = {c1, c2, ..., c K}, where K represents the number of keywords. The goal of the Hard-Constrained Text Generation (Hard-CTG) task is to generate a sentence S = {s1, s2..., s n}, so that S contains all the keywords in C, where n, n ≥ K represents the length of the generated sentence.

[0025] Step 1: Training data construction

[0026] In order to generate text of variable length in the Hard-CTG task, the present invention modifies the initial diffusion process so that it has the following two functions: 1) retaining pre-given keywords during the diffusion process, and 2) being able to insert a variable number of context words at appropriate positions around the given keywords. For the former, the present invention adopts a local diffusion method in the forward and reverse processes of the diffusion model, that is, the word vectors corresponding to the keywords are fixed, and the remaining words are subjected to standard noise addition and denoising operations. For the latter, the present invention introduces a special tag [EMP], which is a placeholder representing an empty space without actual meaning. That is, when [EMP] is inserted into a sentence, it indicates that no actual word is generated at this position. By training the MDM with samples containing the [EMP] tag, the model can master the ability to adaptively insert empty placeholders, thereby generating variable-length text.

[0027] Therefore, based on the original corpus data You can follow Figure 1 The process shown below constructs the training set for MDM Specifically:

[0028] S11. The present invention first uses AdaptiveUKE to extract keywords in sentence S as hard constraints.

[0029] S12, then, by randomly inserting a certain number of [EMP] in the original sentence, a variable-length representation of S is constructed In order to enable MDM to fully understand the meaning of the special tag [EMP], the present invention performs N rounds of random [EMP] insertion on each sentence to generate a large number of training samples containing [EMP] for MDM learning.

[0030] S13. Finally, the present invention defines a tag vector Used to record The position of the keyword in For Sentences The length of m i =1 indicates For the given keyword constraints, when m i =0 indicates is a context word or a special token [EMP], m i = 1 will be fixed in both the forward and reverse diffusion processes, and m i = 0 will participate in the standard diffusion process.

[0031]

[0032]

[0033] Step 2: Model training

[0034] S21. Diffusion process based on keyword fixation

[0035] Given Any training sample in in Indicates the length of the sentence. First, we need to map discrete words into a continuous vector space. The encoding process is shown in formula (1):

[0036]

[0037] Where EMB(·) is a random embedding function that can train sentence Converted to its continuous embedding representation in Representation training sentence The length of , d represents the dimension of word embedding.

[0038] By continuously sampling the noise vector from a standard Gaussian distribution The forward noise formula can be applied to transform x0 to obtain x t ,in Different from the standard forward diffusion process, the present invention adopts a local diffusion mechanism to retain the preset keywords to ensure that they will not change during the noise addition process. t The calculation is modified to the following expression:

[0039]

[0040] in It is a marker array used to indicate the position of keywords, and its element value is m i ∈(0,1), when m i =1 indicates that the corresponding position is a keyword and does not participate in the diffusion process. The rest of the parts participate in the diffusion process normally. In order to simplify the representation, the present invention represents the unfixed part as in

[0041] Since each dimension of the noise ε is independent and identically distributed during sampling, fixing the keyword in the forward process does not affect the distribution of other dimensions. When t It is still close to a standard Gaussian distribution. Therefore, the forward noise addition process can be further expressed as a conditional probability distribution as shown in formula (3):

[0042]

[0043] where v t and v t-1 They represent the hidden states at time t and time t-1 respectively, represents the standard normal distribution, represents the identity matrix, α t It is an artificially set noise factor.

[0044] Accordingly, the inverse denoising process can be defined as θ (v t-1 |v t ), in order to ensure that this process can be learned, the present invention uses the reparameterization technique to express it as follows:

[0045]

[0046] Where θ represents the parameters of the model, μ θ (v t ,t) and Σ θ (v t ,t) represents the reparameterized v t-1 The mean and variance of .

[0047] After determining the distribution of the forward denoising and reverse denoising processes, the training goal of the diffusion model is to fit the distribution q(v t |v t-1 ) and p θ (v t-1 |V t ), that is, the noise eliminated in the reverse denoising process matches the noise added in the forward denoising process, so that the diffusion model can gradually restore the hidden state that is similar to the noise to the original data sample. Therefore, the training objective can be defined as:

[0048]

[0049] in, is the training loss, represents the expected function, μ θ (v t ,t) is the noise mean predicted by the model, is the posterior probability q(v t-1 |v t ,v0).

[0050] However, As a training loss, it essentially makes the model θ (v t-1 |v t ) is fitted. Under this goal, v can only be guaranteed when t is sufficiently close to 0.t It converges towards the target text embedding v0, which is not enough for the model to fully learn the internal structure of v0. Therefore, the present invention further reparameterizes the loss so that it can directly predict v0 at each time step t, thereby speeding up v t Convergence to v0, the specific loss function is defined as follows:

[0051]

[0052] in is the end-to-end training loss, f θ (v t ,t) is the model prediction output, which is implemented based on the Transformer model in the method of the present invention.

[0053] S22. Adaptive denoising strategy based on meta-learning

[0054] In the diffusion model, the intermediate hidden states at different time steps contain different degrees of Gaussian noise. The end-to-end training method is adopted, that is, v is used at any time step. t Directly predict v0, although it is more helpful to v t Converges to a specific word vector, but destroys the gradual convergence process of diffusion. Moreover, in the inference stage, the intermediate hidden state containing Gaussian noise is gradually denoised, which also leads to inconsistency between training and inference. To this end, the present invention introduces a meta-learning strategy in the training stage, proposes an adaptive denoising method, and optimizes the training objective in formula (6). The method is trained by alternately updating the meta-model and the diffusion model, which can be summarized as a cycle of the following three steps:

[0055] S221. First, the diffusion model loss is calculated according to formula (6) at each time step of the reverse process. In this step, use Calculate the loss on sample data in Then pass The gradient back propagation of the diffusion model parameters θ is pre-updated to obtain the temporary diffusion model parameters

[0056] S222, then, the current loss Input into the meta-model, output an adaptive denoising coefficient λ, and update the meta-model parameters based on the meta-learning weighted loss. The meta-model is defined as a feed-forward neural network with only one hidden layer, which can be formally expressed as:

[0057]

[0058] Where λ is the adaptive denoising coefficient of the diffusion model, is the input of the metamodel, and w is the parameter of the metamodel.

[0059] After obtaining λ, the new loss function with adaptive parameters can be further defined as follows:

[0060]

[0061] in Represents the adaptive denoising loss function.

[0062] In order to ensure that the introduction of parameter λ is beneficial to the diffusion model denoising task, the meta-model needs to be trained. Specifically, the meta-model training data comes from an independent Metadata set in The construction process and Exactly the same, but much smaller The training loss is the adaptive denoising loss in Based on temporary parameters The diffusion model output is obtained by Performing reverse derivation, the parameter w of the metamodel is updated to

[0063] S223. Finally, in the real training set The diffusion model parameters are updated using the trained meta-model and adaptive denoising loss. The specific loss function is defined as follows:

[0064]

[0065] Step 3: Model Inference

[0066] The reasoning of the text diffusion model only includes the reverse process, that is, the Gaussian noise of the input sample is gradually denoised by the trained diffusion model until it is transformed into the target text data. Specifically, the reasoning of MDM follows the following process. Given the keyword hard constraint C, MDM first needs to randomly insert a certain number of [SLOT] in the input keyword sequence to indicate the location of the inserted text. Each [SLOT] will be replaced by standard Gaussian noise at the beginning of the reverse process, that is, at t = T, while the input keyword remains fixed. In the subsequent reverse process, [SLOT] will gradually converge and be mapped to a specific and meaningful word through conditional probability maximization at t = 0, or become an empty placeholder [EMP], as shown in formula (10):

[0067]

[0068] where y is the generated target sentence, y i represents a word in y, x 0,iRepresents the i-th word vector in x0.

[0069] The reverse process of diffusion is a transformation in continuous space, so it cannot guarantee that the intermediate hidden state x t will definitely converge to a specific word vector. To solve this problem, the existing text diffusion model uses a rounding operation to t Constraints can be made, which can represent any vector x in a continuous space t,i Mapping to the word embedding EMB(y) closest to it in the vocabulary V can be formally expressed as:

[0070]

[0071] where x t,i ∈x t is a continuous vector in the intermediate hidden state, EMB(y) represents the word embedding of word y, and V represents the entire vocabulary.

[0072] The current text diffusion model chooses to perform rounding operations at each time step. However, considering that the reverse process is at the beginning when t is close to T, the intermediate hidden state x t It is close to the perfect standard Gaussian noise, and the Euclidean distance between it and the target word embedding is large. Performing rounding operation at this time will introduce a large error. Therefore, the present invention selects a new rounding strategy, that is, when t=2 γ , rounding is performed when γ is a natural number. This strategy has a larger interval at the beginning of the reverse process and a smaller interval near the end of the reverse process, thereby reducing the damage of the initial time step to the denoising process, accelerating the convergence of Gaussian noise to the target word embedding, and helping to generate higher quality text.

[0073] The present invention discloses a method for generating limited domain restricted text based on a meta-learning diffusion model, the main advantages of which are reflected in the following aspects:

[0074] (1) We designed a diffusion model that integrates meta-learning to complete the Hard-CTG task, and then migrated the diffusion model from the image field that processes continuous data to the text field that processes discrete data. This allows the diffusion model to play its powerful generative advantages in the field of conditional text generation.

[0075] (2) A local diffusion mechanism is designed to fix keywords during the diffusion process, so that the diffusion model can retain keywords and the domain information they contain in text generation and generate contexts of variable length based on the given keywords.

[0076] (3) The meta-learning idea is introduced to optimize the model training process, and an adaptive denoising mode is designed to smooth the convergence process of denoising and improve the fluency and readability of the generated text.

[0077] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating limited domain restricted text based on a meta-learning diffusion model, characterized in that: The method comprises the following steps: Step 1: In the training data construction phase, a large number of training samples are constructed by extracting keywords and randomly inserting specific placeholder tags [EMP] into the original sentences; Step 2: In the model training stage, local diffusion is used to fix keywords so that the diffusion model can complete the Hard-CTG task, and a meta-learning strategy is introduced to optimize the diffusion model training process; Step 3. Finally, in the model inference stage, a special marker [SLOT] is used to indicate the position where the MetaDiffusionModel (MDM) is inserted. During the reverse diffusion process, the position marked by [SLOT] is gradually converted from standard Gaussian noise to an actual word or [EMP], thereby obtaining a generated text containing the keyword input; Given a set of keywords C = {c1, c2, ..., c K }, where K represents the number of keywords; the goal of the Hard-Constrained Text Generation (Hard-CTG) task is to generate a sentence S = {s1, s2..., s n }, so that S contains all the keywords in C, where n, n ≥ K represents the length of the generated sentence; in, The first step is based on the original corpus data Constructing the training set of MDM The second step comprises: S21. Diffusion process based on keyword fixation Given Any training sample in in Indicates the length of the sentence; first, the discrete words need to be mapped to a continuous vector space; Continuously sample noise vectors from a standard Gaussian distribution, apply forward denoising, and use a local diffusion mechanism to retain the preset keywords to ensure that they do not change during the denoising process; apply reverse denoising; after determining the distribution of the forward denoising and reverse denoising processes, the diffusion model gradually restores the hidden state that is close to the noise to the original data sample; S22. Adaptive denoising strategy based on meta-learning In the training phase, a meta-learning strategy is introduced, and an adaptive denoising method is proposed to optimize the training objective. The method is trained by alternately updating the meta-model and the diffusion model. S221. First, the diffusion model loss is calculated at each time step of the reverse process; S222, then, input the current loss into the meta-model, and update the meta-model parameters based on the meta-learning weighted loss; S223. Finally, in the real training set Update the diffusion model parameters using the trained meta-model and adaptive denoising loss; The step three comprises: Given a keyword hard constraint C, MDM first needs to randomly insert a certain number of [SLOT] into the input keyword sequence to indicate the location of the inserted text; each [SLOT] will be replaced by standard Gaussian noise at the beginning of the reverse process, that is, at t=T, while the input keyword remains fixed; in the subsequent reverse process, [SLOT] will gradually converge and be mapped to a specific and meaningful word through conditional probability maximization at t=0, or become an empty placeholder [EMP].

2. The method for generating limited domain restricted text based on a meta-learning diffusion model according to claim 1, characterized in that: The first step is based on the original corpus data Constructing the training set of MDM The steps include: S11. First, use AdaptiveUKE to extract keywords from sentence S as hard constraints; S12, then, by randomly inserting a certain number of [EMP] in the original sentence, a variable-length representation of S is constructed S13. Finally, a tag vector is defined Used to record The position of the keyword in For Sentences The length of m i =1 indicates For the given keyword constraints, when m i =0 indicates is a context word or a special token [EMP], m i = 1 will be fixed in both the forward and reverse diffusion processes, and m i = 0 will participate in the standard diffusion process.

3. The method for generating limited domain restricted text based on a meta-learning diffusion model as claimed in claim 2, characterized in that: In S12, N rounds of random [EMP] insertion are performed on each sentence to generate a large number of training samples containing [EMP] for MDM learning.

4. The method for generating limited domain restricted text based on a meta-learning diffusion model as claimed in claim 2, characterized in that: A local diffusion method is used in the forward and backward processes of the diffusion model, that is, the word vectors corresponding to the keywords are fixed, and the remaining words are subjected to standard noise addition and denoising operations; the special token [EMP] introduced is a placeholder that represents an empty space and has no actual meaning. That is to say, when [EMP] is inserted into a sentence, it indicates that no actual word is generated at this position. By training the MDM with samples containing the [EMP] token, the model can master the ability to adaptively insert empty placeholders, thereby generating variable-length text.

5. The method for generating limited domain restricted text based on a meta-learning diffusion model according to any one of claims 2 to 4, characterized in that: The step 2 specifically includes: S21. Diffusion process based on keyword fixation Given Any training sample in in Indicates the length of the sentence; first, the discrete words need to be mapped to a continuous vector space. The encoding process is shown in formula (1): Where EMB(·) is a random embedding function used to train the sentence Converted to its continuous embedding representation in Representation training sentence The length of , d represents the dimension of word embedding; By continuously sampling the noise vector from a standard Gaussian distribution Apply the forward noise formula to transform x0 to obtain x t , where x t , A local diffusion mechanism is used to retain the preset keywords to ensure that they will not change during the noise addition process. Therefore, x t The calculation is modified to the following expression: in It is a marker array used to indicate the position of keywords, and its element value is m i ∈(0,1), when m i =1 indicates that the corresponding position is a keyword and does not participate in the diffusion process; the rest of the parts participate in the diffusion process normally, and the unfixed part is represented as in Furthermore, the forward noise addition process is expressed as a conditional probability distribution as shown in formula (3): where v t and v t-1 They represent the hidden states at time t and time t-1 respectively, represents the standard normal distribution, represents the identity matrix, α t It is the artificially set noise coefficient; The inverse denoising process is defined as θ (v t-1 |v t ), and use the reparameterization technique to express it as follows: Where θ represents the parameters of the model, μ θ (v t ,t) and Σ θ (v t ,t) represents the reparameterized v t-1 The mean and variance of After determining the distribution of the forward denoising and reverse denoising processes, the training goal of the diffusion model is to fit the distribution q(v t |v t-1 ) and p θ (v t-1 |v t ), that is, the noise eliminated in the reverse denoising process is matched with the noise added in the forward denoising process, so that the diffusion model gradually restores the hidden state that is similar to the noise to the original data sample; Therefore, the training objective is defined as follows: in is the end-to-end training loss, f θ (v t ,t) is the model prediction output; S22. Adaptive denoising strategy based on meta-learning In the training phase, a meta-learning strategy is introduced, and an adaptive denoising method is proposed to optimize the training objective in formula (6). This method is trained by alternately updating the meta-model and the diffusion model.

6. The method for generating limited domain restricted text based on a meta-learning diffusion model as claimed in claim 5, characterized in that: The step S21 is implemented based on the Transformer model.

7. The method for generating limited domain restricted text based on a meta-learning diffusion model as claimed in claim 5, characterized in that: The step S22 specifically includes the following steps: S221. First, the diffusion model loss is calculated according to formula (6) at each time step of the reverse process; in this step, use Calculate the loss on sample data in Then pass The gradient back propagation of the diffusion model parameters θ is pre-updated to obtain the temporary diffusion model parameters S222, then, the current loss Input into the meta-model, output an adaptive denoising coefficient λ, and update the meta-model parameters based on the meta-learning weighted loss; where the meta-model is defined as a feedforward neural network with only one hidden layer, formally expressed as: Where λ is the adaptive denoising coefficient of the diffusion model, is the input of the metamodel, and w is the parameter of the metamodel; After obtaining λ, the new loss function with adaptive parameters is further defined as follows: in Represents the adaptive denoising loss function; In order to ensure that the introduction of parameter λ is beneficial to the diffusion model denoising task, the meta-model needs to be trained; the training loss of the meta-model is the adaptive denoising loss in Based on temporary parameters The diffusion model output is obtained by Performing reverse derivation, the parameter w of the metamodel is updated to S223. Finally, in the real training set The diffusion model parameters are updated using the trained meta-model and adaptive denoising loss; the specific loss function is defined as follows:

8. The method for generating limited domain restricted text based on a meta-learning diffusion model according to claim 7, characterized in that: In S222, the metamodel training data comes from an independent Metadata set in The construction process and Exactly the same, but much smaller 9. The method for generating limited domain restricted text based on a meta-learning diffusion model according to claim 7, characterized in that: The step three specifically includes: Given the keyword hard constraint C, MDM first needs to randomly insert a certain number of [SLOT] in the input keyword sequence to indicate the location of the inserted text; each [SLOT] will be replaced by standard Gaussian noise at the beginning of the reverse process, that is, at t = T, while the input keyword remains fixed; in the subsequent reverse process, [SLOT] will gradually converge and be mapped to a specific meaningful word through conditional probability maximization at t = 0, or become an empty placeholder [EMP], as shown in formula (10): where y is the generated target sentence, y i represents a word in y, x 0,i represents the continuous embedding representation of the i-th word vector in x0.

10. The method for generating limited domain restricted text based on a meta-learning diffusion model according to claim 9, characterized in that: The reverse process of diffusion is a transformation in continuous space, using rounding operations on x t Constraints are performed, which converts any vector representation x in a continuous space into t,i Mapped to the word embedding EMB(y) closest to it in the vocabulary V, formally expressed as: where x t,i ∈x t is a continuous vector in the intermediate hidden state, EMB(y) represents the word embedding of word y, and V represents the entire vocabulary; When t=2 γ , when γ is a natural number, a rounding operation is performed. This strategy has a larger interval at the beginning of the reverse process and a smaller interval near the end of the reverse process.

Images