Text generation method based on variational auto-encoder and diffusion model

Through the combination of variational autoencoder and diffusion model, the problem of sensitive information leakage in text generation is solved, and the privacy protection and semantic retention of high-quality text generation is achieved, which improves the security and accuracy of text generation.

CN119088948AActive Publication Date: 2024-12-06CHONGQING UNIVERSITY OF SCIENCE AND TECHNOLOGY

Patent Information

Application Number
CN202411207764.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-06
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

Existing text generation methods are difficult to effectively eliminate sensitive information and retain critical information while generating high-quality text, especially when dealing with complex contexts and diverse sensitive information.

Method used

Using text generation method based on variational autoencoder and diffusion model, the text is converted into continuous embedding vectors through BERT encoder, keywords are extracted and vocabulary charts are constructed, and the variational autoencoder and diffusion model are used to add noise to potential vectors. and denoising processing, gradually blur sensitive information and retain key information, and finally generate text through the decoder.

Benefits of technology

It realizes the effective removal of sensitive information in text generation while retaining key information, improves the privacy protection ability and semantic consistency of text generation, and ensures the quality and accuracy of generated text.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088948A_ABST
    Figure CN119088948A_ABST
Patent Text Reader

Abstract

The invention provides a text generation method based on a variational auto-encoder and a diffusion model. The text generation method comprises the following steps: S1, converting discrete characters into continuous embedded vectors by using a BERT encoder; s2, extracting original text keywords and constructing a vocabulary graph; s3, converting the continuous embedded vectors into potential spatial representation by using an encoder in the variational auto-encoder to form potential vectors; s4, gradually iteratively adding Gaussian noise in the forward process of the diffusion model to reach a pure noise state; s5, gradually denoising and reconstructing potential vectors in a reverse process of the diffusion model, so that sensitive information is more effectively fuzzified; s6, mapping the reconstructed potential vector into a text vector by using a decoder in the variational auto-encoder; and S7, restoring the text vector into text data by using a text generator. The method has the advantages that sensitive information can be efficiently removed, important key information is reserved, and the method is suitable for generating safe and high-quality text content.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to text generation technology in the field of deep learning, and specifically to a text generation method based on a variational autoencoder and a diffusion model. Background Art

[0002] With the rapid development of deep learning technology, text generation methods have been widely used in the field of natural language processing. However, the risk of sensitive information leakage in the text generation process also increases, especially in scenarios involving private data and confidential information. Therefore, how to effectively eliminate sensitive information and retain key information while generating high-quality text has become an important research topic in the field of information security.

[0003] Traditional text generation methods, such as rule-based generation methods or statistical models, often have difficulty balancing the confidentiality of information and the coherence of generated text. These methods have certain limitations when dealing with complex contexts and diverse sensitive information. In recent years, with the rise of variational autoencoders (VAE) and diffusion models, text generation methods combined with deep learning technology have provided new ideas for solving this problem. Summary of the invention

[0004] In view of this, the object of the present invention is to provide a text generation method based on a variational autoencoder and a diffusion model, so as to effectively remove sensitive information and retain key information in text generation.

[0005] To achieve the above purpose, the specific technical solutions adopted by the present invention are as follows:

[0006] A text generation method based on a variational autoencoder and a diffusion model, the key of which is to include the following steps:

[0007] S1: Use BERT encoder to convert discrete words into continuous embedding vectors;

[0008] S2: Extract keywords from the original text and construct a vocabulary graph, using each node to represent a word. The edges between nodes are represented by the semantic relationship between words. The importance of each word is evaluated by analyzing the semantic relationship between words.

[0009] S3: Use the encoder in the variational autoencoder to transform the continuous embedding vector into a latent space representation to form a latent vector;

[0010] S4: gradually adding noise to the latent vector, by gradually and iteratively adding Gaussian noise in the forward process of the diffusion model to reach a pure noise state;

[0011] S5: By gradually denoising and reconstructing the latent vector in the reverse process of the diffusion model, it can more effectively blur sensitive information and replace or reconstruct key information when generating text;

[0012] S6: Use the decoder in the variational autoencoder to map the reconstructed latent vector into a text vector;

[0013] S7: Use the text generator to restore the text vector to text data.

[0014] Optionally, in step S2, the embedding vector of each word corresponds to a node in the vocabulary graph, and the relationship between each node and the weight of the edge are calculated through an iterative algorithm to form the importance score of the node, and then the nodes are sorted according to the importance scores, and the original keywords are marked and extracted.

[0015] Optionally, the iterative algorithm is a Text Rank algorithm.

[0016] Optionally, when the encoder in the variational autoencoder is used to convert the continuous embedding vector into a latent space representation in step S3, the embedding vector corresponding to the original text keyword extracted in step S2 is introduced as a conditional input.

[0017] Optionally, the encoder in the variational autoencoder forms the latent vector as follows:

[0018] S31: Set the encoding input layer: take the embedding vector generated by the BERT encoder and the keyword embedding vector of the conditional input as joint input;

[0019] S32: Setting a fully connected layer: sending the joint input of the encoding input layer to the fully connected layer to extract high-dimensional features to form a high-dimensional spatial representation;

[0020] S33: Generate latent space parameters: Input the high-dimensional spatial representation output by the fully connected layer into two parallel sub-networks, one of which is a mean layer for generating a mean μ, and the other is a logarithmic variance layer for generating a logarithmic variance vector σ; the mean layer: generates a mean vector μ = [μ 1 ,μ 2 ,……μ d ], representing the center of the latent space, the logarithmic variance layer: generates the logarithmic variance vector It is used to determine the standard deviation σ of each dimension; d is the dimension represented by the high-dimensional space;

[0021] S34: Set sampling layer: From Gaussian distribution N(μ,σ 2 ),

[0022] z=μ+σ·ε

[0023] Where ε is the noise sampled from the standard normal distribution N(0,I), and σ is the logarithmic variance vector logσ 2 De-exponential standard deviation.

[0024] Optionally, in step S4: from the initial potential vector z 0 First, by gradually adding noise, it is transformed into a series of latent vectors z 0 , z 1 , ... z T , until it ends after T steps of iteration;

[0025] The process of iterative noise addition at each step is defined as:

[0026] Among them, q(z t |z t-1 ) represents the latent vector z t-1 Iterate to the latent vector z t The conditional probability of

[0027] Indicates z t is from a mean The variance is β t is sampled from a Gaussian distribution, I is the identity matrix, β t is a time-dependent noise addition parameter, indicating the amount of noise added at each step. As the number of iterations t increases, the potential vector z t It will gradually approach the standard normal distribution N(0,I) and eventually reach a pure noise state.

[0028] Optionally, in step S5: first, the Gaussian noise z T ~N(0,I) for preliminary sampling, and then the sampling method of Langevin dynamics is used to sample z t-1 ~p θ (z t |z t-1 ), t = T, T-1, ... 0 for iterative sampling, and finally obtain the reconstructed potential vector, the reverse process: Indicates z t-1 is a function whose mean is μ θ (z t ,t), the variance is is sampled from a Gaussian distribution.

[0029] Optionally, to obscure sensitive information, θ Introducing adaptive noise injection:

[0030]

[0031] And in order to enhance the weight of key information, θ Perform attention mechanism weighting:

[0032]

[0033] Therefore, the optimized, reverse process: δ(z t ) represents the amount of adaptive noise injection, γ θ Represents the weight coefficient of the attention mechanism.

[0034] Optionally, the decoder in the variational autoencoder maps the reconstructed latent vector to a text vector by following the steps below:

[0035] S61: Set up the decoding embedding layer: vector z reconstructed by the reverse process 0 Input into the decoding embedding layer, the latent vector is initially processed through a fully connected layer and mapped into a high-dimensional space;

[0036] h 0 =R(W 0 z 0 +b 0 )

[0037] Among them, W 0 is the weight matrix, b 0 is the bias vector, R() is the ReLU activation function, and h 0 It is the high-dimensional space representation after mapping;

[0038] S62: Set the self-attention layer: By introducing a multi-head self-attention layer with a mask mechanism, future information leakage during the decoding process is avoided. The specific formula is as follows:

[0039]

[0040] Among them, Q, K, and V are query matrix, key matrix, and value matrix, respectively, which are obtained by inputting 0 The mask matrix M is obtained by linear transformation. It is used to mask the information of future time steps. The multi-head mechanism enhances the representation ability of the model by combining multiple self-attention heads. k represents the scaling factor;

[0041] S63: Set the first residual connection and normalization layer:

[0042] The output h of the self-attention layer attn With input h 0 Perform residual connection and then normalize:

[0043] h 1 =LayerNorm1(h 0 +hattn )

[0044] LayerNorm1() represents the first residual connection and normalization layer processing function, h 1 It is the output of the first residual connection and normalization layer;

[0045] S64: Setting up the conditional attention layer:

[0046] Introduce conditional information, use the key vector c extracted in step S2 as the query matrix Qc, and use h 1 Linear transformation obtains the key matrix Kc and the value matrix Vc to calculate the conditional attention score:

[0047]

[0048] S65: Set the second residual connection and normalization layer:

[0049] The output h of the conditional attention layer cond With input h 1 Perform residual connection and then normalize:

[0050] h 2 =LayerNorm2(h 1 +h cond )

[0051] LayerNorm2() represents the second residual connection and normalization layer processing function, h 2 Output of the second residual connection and normalization layer;

[0052] S66: Set the feedforward neural network layer: connect the second residual to the normalized layer output h 2 Input to a feedforward neural network:

[0053] h 3 =R(W 1 h 2 +b 1 )

[0054] h 4 =W 2 h 3 +b 2

[0055] Among them, W 1 , W 2 、b 1 、b 2 are the weight and bias parameters of the feedforward network; h 3 is the intermediate variable of the feedforward neural network layer, h 4 is the output of the feedforward neural network layer;

[0056] S67: Setting the third residual connection and normalization layer

[0057] The output h of the feedforward neural network layer 4 With input h 2 Perform residual connection and then normalize:

[0058] h 5 =LayerNorm3(h 4 +h 2 )

[0059] LayerNorm3() represents the third residual connection and normalization layer processing function, h 5 Output of the third residual connection and normalization layer;

[0060] S68: Set the flat layer: Connect the third residual to the high-dimensional vector h output by the normalization layer 5 Converted into a one-dimensional vector; then passed through the fully connected layer and mapped into the final text vector.

[0061] Optionally, step S7 uses a GPT-2 generator to restore the text vector to text data.

[0062] The beneficial effects of the present invention are:

[0063] The present invention proposes a text generation method based on a variational autoencoder and a diffusion model, which focuses on removing sensitive information while retaining key information. By encoding the text using a BERT encoder, the text is converted into a continuous embedding vector, ensuring accurate capture of the original information. Furthermore, TextRank is used to extract key information, and these embedding vectors are input into the variational autoencoder for latent space representation, and regularized in combination with KL divergence to optimize the generation quality. During the denoising process, the model can effectively remove sensitive information while generating content that reflects the emotion of the original text. The introduced jump connection further improves the efficiency of information transfer during the encoding and decoding process, ensuring the quality and accuracy of the final generated text. This method improves the privacy protection capability of text generation while maintaining the semantic consistency and accuracy of the generated text. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0065] Figure 1 It is a flow chart of a text generation method based on a variational autoencoder and a diffusion model provided by the present invention;

[0066] Figure 2 is a VAE encoding flow chart in a specific embodiment of the present invention;

[0067] Figure 3 is a flow chart of diffusion model processing in a specific embodiment of the present invention;

[0068] Figure 4 It is a VAE decoding flow chart in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0069] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, rather than for limiting the protection scope of the present invention.

[0070] The present invention provides a text generation method based on a variational autoencoder and a diffusion model. The variational autoencoder (VAE) can provide greater flexibility for processing sensitive information while retaining the semantics of the original text by encoding the input text into a latent space representation. The diffusion model further blurs the sensitive information through the process of adding noise and denoising, while ensuring the integrity of key information. The combination of these two technologies helps to effectively control the risk of leakage of sensitive information when generating text, and improves the quality and security of the generated text.

[0071] The present invention introduces pre-trained models such as BERT to perform high-dimensional embedding representation of text, then uses TextRank to extract key features, uses conditional VAE to encode these embedded vectors, generates a latent space representation, adds noise and denoises through a diffusion model, blurs sensitive information and retains key features, and finally restores the processed latent vector to a coherent and secure text output through a VAE decoder and a generator such as GPT-2, thereby effectively removing sensitive information and retaining key information in text generation. This text generation method based on variational autoencoders and diffusion models is not only of great significance in the field of deep learning, but also provides an effective solution for the field of information security. With the continuous development of technology, this method is expected to be widely used in various text generation applications, especially in scenarios that require high security and privacy protection.

[0072] When implementing it, Figure 1 As shown, the following steps are included:

[0073] S1: Text embedding, using BERT encoder to convert discrete text into continuous embedding vectors; encoded into Token ID by Tokenizer, represented as X = {x 1 ,x 2 ,......,x n}, where x i is the Token ID of each word. Then, these encoded data are input into the BERT model to generate a high-dimensional vector representation W = {w 1 ,w2 ,......,w n}, these continuous embedding vectors W retain the semantic information of the text;

[0074] S2: Data preprocessing. To improve the model’s focus on key information, the Text Rank algorithm is used to extract keywords from the original text and construct a vocabulary graph. Each node represents a word, and the edges between nodes are represented by the semantic relationship between words. The importance of each word is evaluated by analyzing the semantic relationship between words.

[0075] In specific implementation, the embedding vector of each word corresponds to a node in the vocabulary graph. The relationship between each node and the weight of the edge are calculated through an iterative algorithm to form the importance score of the node. The nodes are then sorted according to their importance scores and the original keywords are marked and extracted.

[0076] The weight iteration process can be expressed as:

[0077]

[0078] Among them, WS(V i ) represents the node V i The weight value, ln(V i ) indicates all points to node V i Node, Out(V i ) represents all nodes pointed to by the node, w ji It indicates the importance of the edge between nodes j and i, d is the damping coefficient, and the nodes are sorted in descending order by weight. The word corresponding to the node with the highest weight is the final keyword;

[0079] S3: Latent space representation, using the encoder in the variational autoencoder to convert the continuous embedding vector into a latent space representation to form a latent vector; and introduce the embedding vector corresponding to the original keyword extracted in step S2 as a conditional input. Figure 2 As shown, the specific steps include:

[0080] S31: Set the encoding input layer: take the embedding vector generated by the BERT encoder and the keyword embedding vector of the conditional input as joint input;

[0081] S32: Setting a fully connected layer: sending the joint input of the encoding input layer to the fully connected layer to extract high-dimensional features to form a high-dimensional spatial representation;

[0082] S33: Generate latent space parameters: Input the high-dimensional spatial representation output by the fully connected layer into two parallel sub-networks, one of which is a mean layer for generating a mean μ, and the other is a logarithmic variance layer for generating a logarithmic variance vector σ; the mean layer: generates a mean vector μ = [μ1 ,μ 2 ,……μ d ], representing the center of the latent space, the logarithmic variance layer: generates the logarithmic variance vector It is used to determine the standard deviation σ of each dimension; d is the dimension represented by the high-dimensional space;

[0083] S34: Set sampling layer: From Gaussian distribution N(μ,σ 2 ),

[0084] z=μ+σ·ε

[0085] Where ε is the noise sampled from the standard normal distribution N(0,I), and σ is the logarithmic variance vector logσ 2 The standard deviation obtained by de-exponential calculation; In order to further ensure the processing of sensitive information in the latent space, a regularization term is introduced in the VAE to calculate the KL divergence to measure the difference between the latent space distribution q(z|W) and the standard normal distribution p(z);

[0086] S4: gradually adding noise to the latent vector, by gradually and iteratively adding Gaussian noise in the forward process of the diffusion model to reach a pure noise state;

[0087] Combination Figure 3 It can be seen that in step S4: from the initial potential vector z 0 First, by gradually adding noise, it is transformed into a series of latent vectors z 0 , z 1 , ... z T , until it ends after T steps of iteration;

[0088] The process of iterative noise addition at each step is defined as:

[0089] Among them, q(z t |z t-1 ) represents the latent vector z t-1 Iterate to the latent vector z t The conditional probability of

[0090] Indicates z t is from a mean The variance is β t is sampled from a Gaussian distribution, I is the identity matrix, β t is a time-dependent noise addition parameter, indicating the amount of noise added at each step. As the number of iterations t increases, the potential vector z t It will gradually approach the standard normal distribution N(0,I) and eventually reach a pure noise state.

[0091] S5: By gradually denoising and reconstructing the latent vector in the reverse process of the diffusion model, it can more effectively blur sensitive information and replace or reconstruct key information when generating text;

[0092] In step S5: first, the Gaussian noise z T ~N(0,I) for preliminary sampling, and then the sampling method of Langevin dynamics is used to sample z t-1 ~p θ (z t |z t-1 ), t = T, T-1, ... 0 for iterative sampling, and finally obtain the reconstructed potential vector, the reverse process: Indicates z t-1 is a function whose mean is μ θ (z t ,t), the variance is is sampled from a Gaussian distribution.

[0093] In order to obscure sensitive information, μ θ Introducing adaptive noise injection:

[0094]

[0095] And in order to enhance the weight of key information, θ Perform attention mechanism weighting:

[0096]

[0097] Therefore, the optimized, reverse process: δ(z t ) represents the amount of adaptive noise injection, γ θ Represents the weight coefficient of the attention mechanism.

[0098] S6: Use the decoder in the variational autoencoder to map the reconstructed latent vector into a text vector;

[0099] Combination Figure 4 It can be seen that the decoder in the variational autoencoder maps the reconstructed latent vector to a text vector according to the following steps:

[0100] S61: Set up the decoding embedding layer: vector z reconstructed by the reverse process 0 Input into the decoding embedding layer, the latent vector is initially processed through a fully connected layer and mapped into a high-dimensional space;

[0101] h 0 =R(W 0 z 0 +b 0 )

[0102] Among them, W 0 is the weight matrix, b 0 is the bias vector, R() is the ReLU activation function, and h 0 It is the high-dimensional space representation after mapping;

[0103] S62: Set the self-attention layer: By introducing a multi-head self-attention layer with a mask mechanism, future information leakage during the decoding process is avoided. The specific formula is as follows:

[0104]

[0105] Among them, Q, K, and V are query matrix, key matrix, and value matrix, respectively, which are obtained by inputting 0 The mask matrix M is obtained by linear transformation. It is used to mask the information of future time steps. The multi-head mechanism enhances the representation ability of the model by combining multiple self-attention heads. k represents the scaling factor;

[0106] S63: Set the first residual connection and normalization layer:

[0107] The output h of the self-attention layer attn With input h 0 Perform residual connection and then normalize:

[0108] h 1 =LayerNorm1(h 0 +h attn )

[0109] LayerNorm1() represents the first residual connection and normalization layer processing function, h 1 It is the output of the first residual connection and normalization layer;

[0110] S64: Setting up the conditional attention layer:

[0111] Introduce conditional information, use the key vector c extracted in step S2 as the query matrix Qc, and use h 1 Linear transformation obtains the key matrix Kc and the value matrix Vc to calculate the conditional attention score:

[0112]

[0113] S65: Set the second residual connection and normalization layer:

[0114] The output h of the conditional attention layer cond With input h 1 Perform residual connection and then normalize:

[0115] h 2 =LayerNorm2(h1 +h cond )

[0116] LayerNorm2() represents the second residual connection and normalization layer processing function, h 2 Output of the second residual connection and normalization layer;

[0117] S66: Set the feedforward neural network layer: connect the second residual to the normalized layer output h 2 Input to a feedforward neural network:

[0118] h 3 =R(W 1 h 2 +b 1 )

[0119] h 4 =W 2 h 3 +b 2

[0120] Among them, W 1 , W 2 、b 1 、b 2 are the weight and bias parameters of the feedforward network; h 3 is the intermediate variable of the feedforward neural network layer, h 4 is the output of the feedforward neural network layer;

[0121] S67: Setting the third residual connection and normalization layer

[0122] The output h of the feedforward neural network layer 4 With input h 2 Perform residual connection and then normalize:

[0123] h 5 =LayerNorm3(h 4 +h 2 )

[0124] LayerNorm3() represents the third residual connection and normalization layer processing function, h 5 Output of the third residual connection and normalization layer;

[0125] S68: Set the flat layer: Connect the third residual to the high-dimensional vector h output by the normalization layer 5 Converted into a one-dimensional vector; then passed through the fully connected layer and mapped into the final text vector.

[0126] S7: Use a text generator to restore the text vector to text data. In this example, the GPT-2 generator is used.

[0127] In order to further understand the technical effect of the above invention, the following experiments are designed to verify it:

[0128] The experimental data comes from foreign social media text comments, including the Twitter Sentiment dataset and the IMDB movie review dataset.

[0129] The data information is shown in Table 1:

[0130] Table 1: Experimental data information

[0131] Corpus Number of texts / items Training set Test Set Target Tags Privacy Label Twitter Sentiment 100000 80000 20000 2 2 IMDB 180000 144000 36000 2 2

[0132] The Twitter corpus is divided into a training set and a test set at a ratio of 8:2, with African American English (AEE) and Standard American English (SAE) as privacy labels, and two emotion labels as target labels, divided into positive and negative emotions. The Amazon corpus is divided into a training set and a test set at a ratio of 8:2, with two emotion labels as target labels, divided into positive and negative emotions; gender labels are privacy labels, divided into male and female.

[0133] During the experiment, a diffusion model based on the Transformer architecture was used, where the noise schedule of the diffusion model was sqrt, the number of inverse diffusion steps was 2000, the maximum sequence length was 128, and the Adam optimizer was used. The word embedding vector dimension was 768 dimensions; the dropout was 0.1, the learning rate was 2e-5, the batch size was 160, and the training epoch was 20.

[0134] Using accuracy as an indicator:

[0135] TP represents positive samples, TN represents negative samples, FP represents false positive samples, and FN represents false negative samples. The higher the accuracy, the better the generated text effect.

[0136] Test results: The experiment used a method based on variational autoencoder and diffusion model to generate desensitized text for Twitter Sentiment and IMDB movie review datasets.

[0137]

[0138] The accuracy rate reached 80.3% in the Twitter Sentiment dataset and 78.9% in the IMDB movie review dataset, which can effectively protect data privacy.

[0139] Through the above embodiments, it can be seen that the text generation method based on variational autoencoder and diffusion model proposed in the present invention can effectively improve the privacy protection capability of text generation while maintaining the semantic consistency and accuracy of the generated text.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A text generation method based on variational autoencoder and diffusion model, It is characterized in that The following steps are involved: S1: Use BERT encoder to convert discrete words into continuous embedding vectors; S2: Extract keywords from the original text and construct a vocabulary graph, using each node to represent a word. The edges between nodes are represented by the semantic relationship between words. The importance of each word is evaluated by analyzing the semantic relationship between words. S3: Use the encoder in the variational autoencoder to transform the continuous embedding vector into a latent space representation to form a latent vector; S4: gradually adding noise to the latent vector, by gradually and iteratively adding Gaussian noise in the forward process of the diffusion model to reach a pure noise state; S5: By gradually denoising and reconstructing the latent vector in the reverse process of the diffusion model, it can more effectively blur sensitive information and replace or reconstruct key information when generating text; S6: Use the decoder in the variational autoencoder to map the reconstructed latent vector into a text vector; S7: Use the text generator to restore the text vector to text data.

2. The text generation method based on variational autoencoder and diffusion model according to claim 1, It is characterized in that In step S2, the embedding vector of each word corresponds to a node in the vocabulary graph. The relationship between each node and the weight of the edge are calculated through an iterative algorithm to form the importance score of the node. The nodes are then sorted according to their importance scores and the original keywords are marked and extracted.

3. The text generation method based on variational autoencoder and diffusion model according to claim 2, It is characterized in that The iterative algorithm is the Text Rank algorithm.

4. The text generation method based on a variational autoencoder and a diffusion model according to any one of claims 1 to 3, It is characterized in that In step S3, when the encoder in the variational autoencoder is used to convert the continuous embedding vector into a latent space representation, the embedding vector corresponding to the original text keyword extracted in step S2 is introduced as a conditional input.

5. The text generation method based on variational autoencoder and diffusion model according to claim 4, It is characterized in that The encoder in a variational autoencoder forms the latent vector by following these steps: S31: Set the encoding input layer: take the embedding vector generated by the BERT encoder and the keyword embedding vector of the conditional input as joint input; S32: Setting a fully connected layer: sending the joint input of the encoding input layer to the fully connected layer to extract high-dimensional features to form a high-dimensional spatial representation; S33: Generate latent space parameters: Input the high-dimensional spatial representation output by the fully connected layer into two parallel sub-networks, one of which is a mean layer for generating a mean μ, and the other is a logarithmic variance layer for generating a logarithmic variance vector σ; the mean layer: generates a mean vector μ = [μ 1 ,μ 2 ,……μ d ], representing the center of the latent space, the logarithmic variance layer: generates the logarithmic variance vector It is used to determine the standard deviation σ of each dimension; d is the dimension represented by the high-dimensional space; S34: Set sampling layer: From Gaussian distribution N(μ,σ 2 ), z=μ+σ·ε Where ε is the noise sampled from the standard normal distribution N(0,I), and σ is the logarithmic variance vector logσ 2 De-exponential standard deviation.

6. The text generation method based on variational autoencoder and diffusion model according to claim 5, It is characterized in that In step S4: From the initial latent vector z 0 First, by gradually adding noise, it is transformed into a series of latent vectors z 0 , z 1 , ... z T , until it ends after T steps of iteration; The process of iterative noise addition at each step is defined as: Among them, q(z t |z t-1 ) represents the latent vector zt -1 Iteration is the conditional probability of the latent vector zt; Indicates z t is from a mean The variance is β t is sampled from a Gaussian distribution, I is the identity matrix, β t is a time-dependent noise addition parameter, indicating the amount of noise added at each step. As the number of iterations t increases, the potential vector z t It will gradually approach the standard normal distribution N(0,I) and eventually reach a pure noise state.

7. The text generation method based on variational autoencoder and diffusion model according to claim 6, It is characterized in that In step S5: first, the Gaussian noise z T ~N(0,I) for preliminary sampling, and then the sampling method of Langevin dynamics is used to sample z t-1 ~p θ (z t |z t-1 ), t=T,T-1,.....0 for iterative sampling, and finally get the reconstructed potential vector, the reverse process: Indicates z t-1 is a function whose mean is μ θ (z t ,t), the variance is is sampled from a Gaussian distribution.

8. The text generation method based on variational autoencoder and diffusion model according to claim 7, It is characterized in that In order to obscure sensitive information, μ θ Introducing adaptive noise injection: And in order to enhance the weight of key information, θ Perform attention mechanism weighting: Therefore, the optimized, reverse process: δ(z t ) represents the amount of adaptive noise injection, γ θ Represents the weight coefficient of the attention mechanism.

9. The text generation method based on a variational autoencoder and a diffusion model according to any one of claims 5 to 8, It is characterized in that The decoder in the variational autoencoder maps the reconstructed latent vector to a text vector by following the steps below: S61: Set up the decoding embedding layer: vector z reconstructed by the reverse process 0 Input into the decoding embedding layer, the latent vector is initially processed through a fully connected layer and mapped into a high-dimensional space; h 0 =R(W 0 z 0 +b 0 ) Among them, W 0 is the weight matrix, b 0 is the bias vector formed by position encoding, R() is the ReLU activation function, and h 0 It is the high-dimensional space representation after mapping; S62: Set the self-attention layer: By introducing a multi-head self-attention layer with a mask mechanism, future information leakage during the decoding process is avoided. The specific formula is as follows: Among them, Q, K, and V are query matrix, key matrix, and value matrix, respectively, which are obtained by inputting h 0 The mask matrix M is obtained by linear transformation. It is used to mask the information of future time steps. The multi-head mechanism enhances the representation ability of the model by combining multiple self-attention heads. k represents the scaling factor; S63: Set the first residual connection and normalization layer: The output h of the self-attention layer attn With input h 0 Perform residual connection and then normalize: h 1 =LayerNorm1(h 0 +h attn ) LayerNorm1() represents the first residual connection and normalization layer processing function, h 1 It is the output of the first residual connection and normalization layer; S64: Setting up the conditional attention layer: Introduce conditional information, use the key vector c extracted in step S2 as the query matrix Qc, and use h 1 Linear transformation obtains the key matrix Kc and the value matrix Vc to calculate the conditional attention score: S65: Set the second residual connection and normalization layer: The output h of the conditional attention layer cond With input h 1 Perform residual connection and then normalize: h 2 =LayerNorm2(h 1 +h cond ) LayerNorm2() represents the second residual connection and normalization layer processing function, h 2 Output of the second residual connection and normalization layer; S66: Set the feedforward neural network layer: connect the second residual to the normalized layer output h 2 Input to a feedforward neural network: h 3 =R(W 1 h 2 +b 1 ) h 4 =W 2 h 3 +b 2 Among them, W 1 , W 2 、b 1 、b 2 are the weight and bias parameters of the feedforward network; h 3 is the intermediate variable of the feedforward neural network layer, h 4 is the output of the feedforward neural network layer; S67: Setting the third residual connection and normalization layer The output h of the feedforward neural network layer 4 With input h 2 Perform residual connection and then normalize: h 5 =LayerNorm3(h 4 +h 2 ) LayerNorm3() represents the third residual connection and normalization layer processing function, h 5 Output of the third residual connection and normalization layer; S68: Set the flat layer: Connect the third residual to the high-dimensional vector h output by the normalization layer 5 Converted into a one-dimensional vector; then passed through the fully connected layer and mapped into the final text vector.

10. The text generation method based on variational autoencoder and diffusion model according to claim 9, It is characterized in that Step S7 uses the GPT-2 generator to restore the text vector to text data.

Citation Information

Patent Citations

  • Method for generating continuous pictures by long text based on diffusion model

    CN117521672A

  • Consumer graphics card-based AIGC multi-mode efficient generation method and device

    CN118115611A

  • Image text generation method, system and equipment and storage medium

    CN118351219A

Cited By

  • Method and device for compressing downtime screenshot image

    CN119854493A

  • Stratosphere wind field simulation method

    CN119940424A

  • Complex dialogue text generation method based on diffusion model

    CN120297258A

  • Model processing method and device for semantic understanding and medium

    CN122088515A