A method for complex dialogue text generation based on diffusion model

Abstract

The present invention discloses a method for generating complex dialogue text based on a diffusion model, which preprocesses a data set containing question text, context text, question type text and answer text. Then, the latent representation of the question text is extracted through the question encoding module, and the features of the context text and the answer text are fused using the matching module. Next, the latent representation of the question text and the fused features are input into a diffusion denoiser to generate a noise latent representation. At the same time, the question type is predicted based on the answer text vector, and the question type classification module is optimized. Finally, the relevant features are input into the decoding module to generate a predicted question, and the optimal text generation model is obtained by constructing a loss function optimization model. The present invention utilizes the gradual denoising characteristics of the diffusion model to improve the diversity and naturalness of question generation; and can more effectively capture the semantic details and context associations in the dialogue, thereby generating more logical and coherent complex questions.

Description

Technical Field

[0001] The present invention belongs to the technical field of text processing, and in particular relates to a method for generating complex dialogue text based on a diffusion model. Background Art

[0002] With the continuous advancement of artificial intelligence technology, particularly in the field of natural language processing (NLP), deep learning-based question generation models have been widely used in various applications, including intelligent customer service, automated question answering, and voice assistants. Although existing technologies have made significant progress in generating simple question content, existing question generation systems still have many limitations when faced with complex scenarios.

[0003] Traditional question generation techniques typically use template-based, sequence-to-sequence (Seq2Seq) models, or Transformer architectures. While these methods can generate standardized question content, they struggle to generate high-quality questions in complex or open-ended scenarios. Existing question generation systems often lack a deep understanding of contextual information, and the questions they generate are relatively simple, lacking in diversity and challenge.

[0004] Although question generation technology has made some progress in recent years, existing question generation methods still have the following major flaws when facing complex dialogue problems:

[0005] 1. Insufficient question generation diversity: Existing dialogue systems primarily rely on template-based or Seq2Seq (sequence-to-sequence) model generation. These methods often generate questions with a single structure and lack of variation, resulting in high repetitiveness and failing to meet the diverse question requirements of different application scenarios.

[0006] 2. Lack of a guidance mechanism for question types: Existing technologies struggle to generate specific question types based on context or user needs. For example, in applications across diverse fields like education, healthcare, and law, generated questions need to be targeted. However, existing systems lack a flexible guidance mechanism, resulting in confusing and inaccurate question types.

[0007] 3. Poor adaptability to complex conversational scenarios: Existing generative models often struggle to maintain contextual consistency when processing complex background information, and are prone to logical confusion, repetition, or meaningless questions. This limitation makes question generation systems perform poorly in tasks requiring reasoning, logical analysis, or long-term information memory.

[0008] 4. Diffusion models are underappreciated in text generation tasks: While diffusion models have achieved breakthroughs in image generation in recent years, their application to natural language generation remains exploratory. Existing text generation models, most of which are based on autoregressive or variational autoencoder (VAE) methods, fail to fully leverage the powerful generation capabilities of diffusion models. This results in limited quality of generated questions, making it difficult to balance accuracy, coherence, and diversity. Summary of the Invention

[0009] In response to the shortcomings of the existing technology, the present invention proposes a method for generating complex dialogue questions based on a diffusion model to improve the diversity, rationality and context adaptability of dialogue questions.

[0010] A method for generating complex conversation text based on a diffusion model, comprising:

[0011] Step S1: Construct a dataset and preprocess the samples in the dataset. The samples include question text, context text, question type text, and answer text.

[0012] Step S2: Construct a text generation model based on the diffusion model to formalize the text generation task;

[0013] The text generation model includes a question encoding module, a diffusion denoiser, a matching module, a question type classification module and a decoding module. The question text in the input dataset is fed into the question encoding module of the text generation model to obtain the potential representation of the question text.

[0014] Step S3: Input the context text and answer text in step S1 into the matching module to obtain the fused feature representation;

[0015] Step S4: Input the latent representation of the question text in step S2 and the fused feature representation in step S3 into the diffusion denoiser to obtain the noise latent representation;

[0016] Step S5: Input the global answer vector of the answer text into the question type classification module, obtain the predicted question type text, and optimize it based on the question type text in step S1;

[0017] Step S6: Input the latent representation of the question text, the noise latent representation and the question type text into the decoding module, and generate a predicted question based on the derived sequence of the question text in step S1;

[0018] Step S7: Construct a total loss function to optimize the text generation model and obtain the optimal text generation model.

[0019] Furthermore, the processing flow of the question coding module in step S2 is specifically as follows:

[0020] Input the question text for encoding and obtain the feature representation of the question text;

[0021] Connect the latent token to the question text feature representation to encode the semantic information of the question text;

[0022] The question text feature representation is formatted based on the latent token to obtain the question text feature representation after format conversion, which is expressed as: ,in, represents a potential marker, Indicates the length of the question text q, Indicates the first word;

[0023] The feature representation of the format-converted question text is processed through the BART encoder to obtain the final hidden state of the potential marker;

[0024] The latent representation of the posterior distribution of the final hidden state of the latent marker is obtained through a multi-layer perceptron , expressed as:

[0025] ;

[0026] ;

[0027] in, represents the final hidden state of the potential marker, represents the set of real numbers, Represents the dimension, Represents a given input The conditional probability distribution of the latent variable z when Indicates the preset parameters. represents the latent representation of the question text, Represents a multilayer perceptron.

[0028] Furthermore, the processing flow of the matching module in step S3 is specifically as follows:

[0029] The input context text and answer text are vectorized and encoded through the BART encoder to obtain the context text vector representation and the answer text vector representation ,in, is the hidden layer dimension, represents the set of real numbers, Indicates the length of the context text, Indicates the length of the answer text;

[0030] Perform cross-attention calculation on the context text vector representation and the answer text vector representation to obtain the matching relationship between the context text vector representation and the answer text vector representation, which is expressed as:

[0031] ;

[0032] in, , represents the attention weight matrix between the context text and the answer text, that is, the matching relationship between the context text vector representation and the answer text vector representation; represents the activation function;

[0033] Obtaining contextual representation after integrating answer text information , expressed as:

[0034] ;

[0035] Will and After splicing, the input is input into the fully connected layer for nonlinear transformation to obtain the fusion feature representation , expressed as:

[0036]

[0037] in, , represents a nonlinear activation function, Indicates the preset offset.

[0038] Furthermore, the processing flow of the diffusion denoiser in step S5 is specifically as follows:

[0039] Sample a time step from the uniform distribution , adding noise to the latent representation of the question text; Indicates the total number of time steps;

[0040] Forming a Markov chain ,in , generating latent variables ; expressed as:

[0041] ;

[0042] in, , is the preset proportional coefficient, used to control the noise ratio in a single step, The latent representation of the question text is represented by the process of adding noise at time step The intermediate state when , that is, the latent variable; The latent representation of the question text is represented by the process of adding noise at time step The intermediate state when The latent representation of the question text is represented by the process of adding noise at time step The final state of represents the normal distribution, represents a normal distribution with mean 0 and covariance as the identity matrix I, Indicates that at time step t, hour The conditional probability distribution of ; express Obey the mean , the covariance is Normal distribution;

[0043] The diffusion denoiser uses a Transformer decoder structure and processes the fusion feature representation through a cross-attention mechanism. and latent variables ; expressed as:

[0044] ;

[0045] in, represents the noise potential representation, Represents the time step Embedded, represents the denoising function.

[0046] Furthermore, step S6 is specifically as follows:

[0047] Represent the answer text Compressed into a fixed-length global answer vector by average pooling operation , expressed as:

[0048] ;

[0049] Will Input into the classifier to predict the question type and obtain the predicted question type text , expressed as:

[0050] ;

[0051] in, , , Represents the weight matrix preset by the classifier, Represents the preset bias of the classifier, Indicates the total number of preset question types;

[0052] The optimization is performed using the cross entropy loss function based on the question type text in step S1.

[0053] Furthermore, the processing of the decoding module is specifically as follows:

[0054] ;

[0055] in, Indicates the generated problem, Denotes the decoding module, A derived sequence representing the question text.

[0056] Furthermore, the decoding module adopts the BART decoder and further adopts the memory mechanism in OPTIMUS to project the latent variables into key-value pairs and concatenate the key-value pairs with the final hidden state of the latent marker.

[0057] Furthermore, the total loss function Expressed as:

[0058] ;

[0059] in, represents the negative log-likelihood loss, represents the bag-of-words loss function, represents the latent variable denoising loss function.

[0060] Furthermore, the formalized text generation task in step S2 is specifically as follows:

[0061] Generate semantically consistent questions based on the given context, answer, and question type;

[0062] Formally formulated as a function of conditional distribution, it is expressed as: ;

[0063] in, Represents contextual text, providing semantic background for question generation; Indicates the answer text, which is the direct target of the question; Indicates the question type text; represents the generated question text, Represents the probability density function of generating question text given context text, answer text and question type text;

[0064] Using continuous latent variables As a high-level representation of the question text; specifically: from the latent variable space Distribution in Sampling continuous latent variables , then, from and Decoding question text .

[0065] The positive progress effect of the present invention is:

[0066] 1. This paper utilizes the gradual denoising properties of the diffusion model to enhance the diversity and naturalness of question generation. The diffusion model can more effectively capture the semantic details and contextual associations in the conversation, thereby generating more logical and coherent complex questions.

[0067] 2. This invention provides a controllable adjustment mechanism for the complexity of the generated problem. Users can introduce specific denoising strategies and random perturbations. The model enhances the diversity of problems while ensuring rationality and is suitable for different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0069] The following describes the embodiments of the present invention through specific examples. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention.

[0070] Reference Figure 1 , a complex dialogue text generation method based on diffusion model, including:

[0071] Step S1: Construct a dataset and preprocess the samples in the dataset. The samples include question text, context text, question type text, and answer text.

[0072] Step S2: Construct a text generation model based on the diffusion model to formalize the text generation task;

[0073] The text generation model includes a question encoding module, a diffusion denoiser, a matching module, a question type classification module and a decoding module. The question text in the input dataset is fed into the question encoding module of the text generation model to obtain the potential representation of the question text.

[0074] Step S3: Input the context text and answer text in step S1 into the matching module to obtain the fused feature representation;

[0075] Step S4: Input the latent representation of the question text in step S2 and the fused feature representation in step S3 into the diffusion denoiser to obtain the noise latent representation;

[0076] Step S5: Input the global answer vector of the answer text into the question type classification module, obtain the predicted question type text, and optimize it based on the question type text in step S1;

[0077] Step S6: Input the latent representation of the question text, the noise latent representation and the question type text into the decoding module, and generate a predicted question based on the derived sequence of the question text in step S1;

[0078] Step S7: Construct a total loss function to optimize the text generation model and obtain the optimal text generation model.

[0079] Furthermore, the processing flow of the question coding module in step S2 is specifically as follows:

[0080] Input the question text for encoding and obtain the feature representation of the question text;

[0081] Connect the latent token to the question text feature representation to encode the semantic information of the question text;

[0082] The question text feature representation is formatted based on the latent token to obtain the question text feature representation after format conversion, which is expressed as: ,in, represents a potential marker, Indicates the length of the question text q, Indicates the first word;

[0083] The feature representation of the format-converted question text is processed through the BART encoder to obtain the final hidden state of the potential marker;

[0084] The latent representation of the posterior distribution of the final hidden state of the latent marker is obtained through a multi-layer perceptron , expressed as:

[0085] ;

[0086] ;

[0087] in, represents the final hidden state of the potential marker, represents the set of real numbers, Represents the dimension, Represents a given input The conditional probability distribution of the latent variable z when Indicates the preset parameters. represents the latent representation of the question text, Represents a multilayer perceptron.

[0088] Furthermore, the processing flow of the matching module in step S3 is specifically as follows:

[0089] The input context text and answer text are vectorized and encoded through the BART encoder to obtain the context text vector representation and the answer text vector representation ,in, is the hidden layer dimension, represents the set of real numbers, Indicates the length of the context text, Indicates the length of the answer text;

[0090] Perform cross-attention calculation on the context text vector representation and the answer text vector representation to obtain the matching relationship between the context text vector representation and the answer text vector representation, which is expressed as:

[0091] ;

[0092] in, , represents the attention weight matrix between the context text and the answer text, that is, the matching relationship between the context text vector representation and the answer text vector representation; represents the activation function;

[0093] Obtaining contextual representation after integrating answer text information , expressed as:

[0094] ;

[0095] Will and After splicing, the input is input into the fully connected layer for nonlinear transformation to obtain the fusion feature representation , expressed as:

[0096]

[0097] in, , represents a nonlinear activation function, Indicates the preset offset.

[0098] Furthermore, the processing flow of the diffusion denoiser in step S5 is specifically as follows:

[0099] Sample a time step from the uniform distribution , adding noise to the latent representation of the question text; Indicates the total number of time steps;

[0100] Forming a Markov chain ,in , generating latent variables ; expressed as:

[0101] ;

[0102] in, , is the preset proportional coefficient, used to control the noise ratio in a single step, The latent representation of the question text is represented by the process of adding noise at time step The intermediate state when , that is, the latent variable; The latent representation of the question text is represented by the process of adding noise at time step The intermediate state when The latent representation of the question text is represented by the process of adding noise at time step The final state of represents the normal distribution, represents a normal distribution with mean 0 and covariance as the identity matrix I, Indicates that at time step t, hour The conditional probability distribution of ; express Obey the mean , the covariance is Normal distribution;

[0103] The diffusion denoiser uses a Transformer decoder structure and processes the fusion feature representation through a cross-attention mechanism. and latent variables ; expressed as:

[0104] ;

[0105] in, represents the noise potential representation, Represents the time step Embedded, represents the denoising function.

[0106] Furthermore, step S6 is specifically as follows:

[0107] Represent the answer text Compressed into a fixed-length global answer vector by average pooling operation , expressed as:

[0108] ;

[0109] Will Input into the classifier to predict the question type and obtain the predicted question type text , expressed as:

[0110] ;

[0111] in, , , Represents the weight matrix preset by the classifier, Represents the preset bias of the classifier, Indicates the total number of preset question types;

[0112] The optimization is performed using the cross entropy loss function based on the question type text in step S1.

[0113] Furthermore, the processing of the decoding module is specifically as follows:

[0114] ;

[0115] in, Indicates the generated problem, Denotes the decoding module, A derived sequence representing the question text.

[0116] Furthermore, the decoding module adopts the BART decoder and further adopts the memory mechanism in OPTIMUS to project the latent variables into key-value pairs and concatenate the key-value pairs with the final hidden state of the latent marker.

[0117] Furthermore, the total loss function Expressed as:

[0118] ;

[0119] in, represents the negative log-likelihood loss, represents the bag-of-words loss function, represents the latent variable denoising loss function.

[0120] Furthermore, the formalized text generation task in step S2 is specifically as follows:

[0121] Generate semantically consistent questions based on the given context, answer, and question type;

[0122] Formally formulated as a function of conditional distribution, it is expressed as: ;

[0123] in, Represents contextual text, providing semantic background for question generation; Indicates the answer text, which is the direct target of the question; Indicates the question type text; represents the generated question text, Represents the probability density function of generating question text given context text, answer text and question type text;

[0124] Using continuous latent variables As a high-level representation of the question text; specifically: from the latent variable space Distribution in Sampling continuous latent variables , then, from and Decoding question text .

[0125] The present invention has been described in detail above with reference to the embodiments of the accompanying drawings. A person skilled in the art can make various modifications to the present invention based on the above description. Therefore, certain details in the embodiments should not be construed as limiting the present invention. The scope of protection of the present invention shall be determined by the scope defined by the appended claims.

Claims

1. A method for generating complex dialogue text based on a diffusion model, characterized in that: include: Step S1: Construct a dataset and preprocess the samples in the dataset. The samples include question text, context text, question type text, and answer text. Step S2: Construct a text generation model based on the diffusion model to formalize the text generation task; The text generation model includes a question encoding module, a diffusion denoiser, a matching module, a question type classification module and a decoding module. The question text in the input dataset is fed into the question encoding module of the text generation model to obtain the potential representation of the question text. Step S3: Input the context text and answer text in step S1 into the matching module to obtain the fused feature representation; Step S4: Input the latent representation of the question text in step S2 and the fused feature representation in step S3 into the diffusion denoiser to obtain the noise latent representation; Step S5: Input the global answer vector of the answer text into the question type classification module, obtain the predicted question type text, and optimize it based on the question type text in step S1; Step S6: Input the latent representation of the question text, the noise latent representation and the question type text into the decoding module, and generate a predicted question based on the derived sequence of the question text in step S1; Step S7: Construct a total loss function to optimize the text generation model and obtain the optimal text generation model.

2. The method for generating complex dialogue text based on a diffusion model according to claim 1, characterized in that: The processing flow of the question coding module in step S2 is specifically as follows: Input the question text for encoding and obtain the feature representation of the question text; Connect the latent token to the question text feature representation to encode the semantic information of the question text; The question text feature representation is formatted based on the latent token to obtain the question text feature representation after format conversion, which is expressed as: ,in, represents a potential marker, Indicates the length of the question text q, Indicates the first word; The feature representation of the format-converted question text is processed through the BART encoder to obtain the final hidden state of the potential marker; The latent representation of the posterior distribution of the final hidden state of the latent marker is obtained through a multi-layer perceptron , expressed as: ； ； in, represents the final hidden state of the potential marker, represents the set of real numbers, Represents dimension, Represents a given input The conditional probability distribution of the latent variable z when Indicates the preset parameters. represents the latent representation of the question text, Represents a multilayer perceptron.

3. The method for generating complex dialogue text based on a diffusion model according to claim 1, characterized in that: The processing flow of the matching module in step S3 is specifically as follows: The input context text and answer text are vectorized and encoded through the BART encoder to obtain the context text vector representation and the answer text vector representation ,in, is the hidden layer dimension, represents the set of real numbers, Indicates the length of the context text, Indicates the length of the answer text; Perform cross-attention calculation on the context text vector representation and the answer text vector representation to obtain the matching relationship between the context text vector representation and the answer text vector representation, which is expressed as: ； in, , represents the attention weight matrix between the context text and the answer text, that is, the matching relationship between the context text vector representation and the answer text vector representation; represents the activation function; Obtaining contextual representation after integrating answer text information , expressed as: ； Will and After splicing, the input is input into the fully connected layer for nonlinear transformation to obtain the fusion feature representation , expressed as: in, , represents a nonlinear activation function, Indicates the preset offset.

4. The method for generating complex dialogue text based on a diffusion model according to claim 3, characterized in that: The specific processing flow of the diffusion denoiser in step S5 is as follows: Sample a time step from the uniform distribution , adding noise to the latent representation of the question text; Indicates the total number of time steps; Forming a Markov chain ,in , generating latent variables ; expressed as: ； in, , is the preset proportional coefficient, used to control the noise ratio in a single step, The latent representation of the question text is represented by the process of adding noise at time step The intermediate state when , that is, the latent variable; The latent representation of the question text is represented by the process of adding noise at time step The intermediate state when The latent representation of the question text is represented by the process of adding noise at time step The final state of represents the normal distribution, represents a normal distribution with mean 0 and covariance as the identity matrix I, Indicates that at time step t, hour The conditional probability distribution of ; express Obey the mean , the covariance is Normal distribution; The diffusion denoiser uses a Transformer decoder structure and processes the fusion feature representation through a cross-attention mechanism. and latent variables ; expressed as: ； in, represents the noise potential representation, Represents the time step Embedded, represents the denoising function.

5. The method for generating complex dialogue text based on a diffusion model according to claim 4, characterized in that: Step S6 is specifically as follows: Represent the answer text Compressed into a fixed-length global answer vector by average pooling operation , expressed as: ； Will Input into the classifier to predict the question type and obtain the predicted question type text , expressed as: ； in, , , Represents the weight matrix preset by the classifier, Represents the preset bias of the classifier, Indicates the total number of preset question types; The optimization is performed using the cross entropy loss function based on the question type text in step S1.

6. The method for generating complex dialogue text based on a diffusion model according to claim 5, characterized in that: The processing of the decoding module is as follows: ； in, Indicates the generated problem, Denotes the decoding module, A derived sequence representing the question text.

7. The method for generating complex dialogue text based on a diffusion model according to claim 6, characterized in that: The decoding module adopts the BART decoder and further adopts the memory mechanism in OPTIMUS to project the latent variables into key-value pairs and concatenate the key-value pairs with the final hidden state of the latent marker.

8. The method for generating complex dialogue text based on a diffusion model according to claim 1, characterized in that: Total loss function Expressed as: ； in, represents the negative log-likelihood loss, represents the bag-of-words loss function, represents the latent variable denoising loss function.

9. The method for generating complex dialogue text based on a diffusion model according to claim 1, characterized in that: The formal text generation task in step S2 is specifically: Generate semantically consistent questions based on the given context, answer, and question type; Formally formulated as a function of conditional distribution, it is expressed as: ; in, Represents contextual text, providing semantic background for question generation; Indicates the answer text, which is the direct target of the question; Indicates the question type text; represents the generated question text, Represents the probability density function of generating question text given context text, answer text and question type text; Using continuous latent variables As a high-level representation of the question text; specifically: from the latent variable space Distribution in Sampling continuous latent variables , then, from and Decoding question text .

Images