A category text generation system based on feature fusion

Abstract

The present invention relates to a category text generation system based on feature fusion. The system includes: a category text preprocessing module for vectorizing the category text; a multi-way attention module for learning the attention representation of the text using multiple attention mechanisms in the generator; a generator fusion output module for fusing multiple attention representations and outputting the generated text; a local semantic feature extraction module for the discriminator to extract the local dependency features of the text; a global semantic feature extraction module for the discriminator to extract the long-distance dependency features of the text; a discriminator fusion output module for fusing the local and long-distance dependency features of the text and feeding back the discrimination result to the generator.

Description

Technical Field

[0001] The present invention relates to the field of category text generation, and particularly to a category text generation system based on feature fusion. Background Art

[0002] Category text generation aims to automatically generate coherent and meaningful texts with different attributes. The generation of multi-category and multi-sentiment texts can make machines more intelligent, thus achieving better human-computer interaction effects. Category text generation can be widely applied to fields such as dialogue systems, dataset construction for sentiment analysis tasks, review text generation, etc., providing people with convenient and automated tools and reducing the cost of manually writing texts.

[0003] Category text generation is a subtask in the field of text generation. Traditional text generation methods mainly rely on rule templates and complete text generation work through content planning, sentence planning, and language realization. Subsequently, text generation methods based on probability statistics have also been widely applied. This method encodes the relationship between words and context into conditional probabilities and conducts text generation from the perspective of probability statistics. With the rapid development of deep learning, most current category text generation works no longer use traditional text generation methods but use deep learning methods to improve the generalization ability of the model.

[0004] Currently, category text generation models based on deep learning can be mainly divided into the following three categories: those based on the sequence-to-sequence model (Seq2Seq), those based on variational auto-encoders (VAE), and those based on generative adversarial networks (GAN):

[0005] The category text generation model based on the sequence-to-sequence model encodes the input sequence into an intermediate vector through an encoder-decoder model, and then decodes the intermediate vector into an output sequence through the decoder. Currently, some researchers use beam search to generate several alternative sequences and sort them based on certain rules to obtain the best result. Other researchers introduce the LDA model on the basis of the Seq2Seq model to obtain the text theme in order to generate category texts with better diversity. Although this type of model is widely applied, it is difficult to add additional category information, and the category accuracy of the generated texts is limited.

[0006] The category text generation model based on variational autoencoder first encodes the input text into a hidden vector, and then reconstructs the input text according to the hidden variables to train the model parameters. However, the biggest difficulty in training the VAE model is the KL divergence collapse problem. Researchers introduced a weight to control the KL term and gradually increased this weight during training to alleviate the KL divergence collapse problem. In addition, the researchers proposed a planning-based hierarchical variational model, which uses a multi-layer network structure to learn high-level semantic features and local features, achieving good results in long text generation. Although such methods have made some progress in alleviating the KL divergence collapse problem, this problem remains a challenge, and the quality of the generated text is also limited.

[0007] The category text generation model based on generative adversarial network makes the text generated by the generator fit the distribution of real text as much as possible through alternating adversarial training between the generator and the discriminator, so that the discriminator cannot distinguish, thereby improving the quality of the generated text. Researchers proposed a GAN composed of multiple generators and a multi-class discriminator, and passed category information by establishing a penalty-based objective function, enabling each generator to generate text of a specific category. On the basis of the above methods, the researchers proposed to introduce a hierarchical evolutionary learning algorithm during the training of the model, and obtain the optimal direction of the model with the highest quality of the generated text through evolution, evaluation, and screening. However, such methods are relatively one-sided in extracting text semantic features and it is difficult to obtain text that takes into account both quality and diversity. Therefore, people hope to find a more efficient category text generation method to generate more fluent and accurate category text. Summary of the Invention

[0008] The purpose of the present invention is to provide a category text generation system based on feature fusion. For the generator, an improved multi-way attention mechanism is used to extract the semantic feature relationship of sentences from multiple angles, and the multi-way attention features are effectively fused through bidirectional GRU to output the generated text; for the discriminator, a global semantic feature extraction module is introduced on the basis of the local semantic feature extraction module, and the comprehensive discrimination result is fed back to the generator to make it generate high-quality category text.

[0009] To achieve the above purpose, the technical solution of the present invention is: a category text generation system based on feature fusion, including:

[0010] A category text preprocessing module for vectorizing the category text;

[0011] A multi-way attention module that uses multiple attention mechanisms in the generator to learn multiple attention representations of the text;

[0012] A generator fusion output module that fuses multiple attention representations and outputs the generated text;

[0013] A local semantic feature extraction module for the discriminator to extract local dependency features of the text;

[0014] A global semantic feature extraction module for the discriminator to extract long-distance dependency features of the text;

[0015] A discriminator fusion output module that fuses the local and long-distance dependency features of the text and feeds back the discrimination result to the generator.

[0016] In an embodiment of the present invention, the category text preprocessing module converts the tokenized text data into a vector form of One-Hot encoding, introduces additional category information, and forms an input vector through an embedding layer.

[0017] In an embodiment of the present invention, the multi-way attention module uses the multi-head self-attention mechanism, the splicing attention mechanism, and the dot-product attention to extract different types of attention information, and extracts the feature relationship between words in a sentence from different perspectives.

[0018] In an embodiment of the present invention, the generator fusion output module dynamically fuses multi-way attention features based on bidirectional GRU, learns the representation relationship between attentions in the high-dimensional space, forms a feature vector considering multiple perspectives comprehensively, and uses Gumbel-Softmax to generate an approximate sampling of discrete text, solving the problem of gradient disappearance during direct sampling of the generative adversarial network to output the generated text.

[0019] In an embodiment of the present invention, the local semantic feature extraction module uses convolutional kernels of different sizes to extract features from each generated text and real text, and splices the feature maps after max pooling to obtain local context dependency features.

[0020] In an embodiment of the present invention, the global semantic feature extraction module uses bidirectional GRU to extract global semantic features of the sentence, thereby obtaining long-distance context dependency features in the sentence and making up for the deficiency that the convolutional neural network cannot fully capture the global semantic information of the text.

[0021] In an embodiment of the present invention, the discriminator fusion output module fuses the local semantic features and the global semantic features, enables more gradient information to be backpropagated through a high-speed neural network, and feeds the discrimination result back to the generator as a learning signal, enabling the generator to generate high-quality category text that is more likely to deceive the discriminator.

[0022] Compared with the prior art, the present invention has the following beneficial effects: for the generator, an improved multi-path attention mechanism is used to extract the semantic feature relationships of sentences from multiple perspectives, and the multi-path attention features are effectively fused through bidirectional GRU to output the generated text; for the discriminator, a global semantic feature extraction module is introduced on the basis of the local semantic feature extraction module, and the comprehensive discrimination result is fed back to the generator to enable it to generate high-quality category text. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 FIG. is a block diagram of a category text generation system based on feature fusion according to the present invention.

[0024] Figure 2 FIG. is a schematic configuration diagram of a category text generation system based on feature fusion according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] The technical solutions of the present invention will be specifically described below with reference to the drawings.

[0026] As Figure 1 , 2 shown, a category text generation system based on feature fusion according to the present invention includes: a category text preprocessing module 1 for vectorizing the category text; a multi-path attention module 2 for using multiple attention mechanisms in the generator to learn the attention representation of the text; a generator fusion output module 3 for fusing multiple attention representations and solving the problem of gradient disappearance caused by direct sampling in the generative adversarial network to output the generated text; a local semantic feature extraction module 4 for the discriminator to extract the local dependence features of the text; a global semantic feature extraction module 5 for the discriminator to extract the long-distance dependence features of the text; a discriminator fusion output module 6 for fusing the local dependence features and long-distance dependence features of the text and feeding back the discrimination result to the generator. The configurations of each module will be described in detail below.

[0027] 1) Category text preprocessing module 1

[0028] First, describe how the category text preprocessing module 1 obtains the initial text vector appended with category information.

[0029] The input data of the neural network is generally a vector. For the convenience of end-to-end training of the model, it is necessary to vectorize the text data. In the text preprocessing stage, the source text is first tokenized, and then the tokenized text data is converted into a vector form of one-hot encoding for subsequent embedding encoding processing. In the training stage, first, the generated word y at the previous time step is converted into a word embedding t-1 through the embedding layer Then the category information c is encoded into a category embedding E c, and concatenate the two embedding representations, and finally generate the input vector x with class information appended at the current time step through the parameter matrix W x , generating the input vector x with class information appended at the current time step t :

[0030]

[0031] 2) Multi-way Attention Module 2

[0032] The following describes how the multi-way attention module 2 in the generator processes the input vector obtained from the previous module. This module integrates multiple attention mechanisms, including: multi-head self-attention, dot-product attention, and concatenation attention. This new encoding mechanism can solve the problem that the generator of previous models extracts semantic features one-sidedly because it cannot extract semantic features from multiple perspectives, thereby generating more diverse and richer class texts in terms of semantics.

[0033] First, describe the multi-head self-attention mechanism. The generator of the present invention is implemented based on the Relational Memory Core (RMC), which introduces the storage matrix M t to memorize the context information of the text. Specifically, use the storage matrix M obtained at the previous time step t-1 and the current input vector x t to form N groups of query, key, and value, and extract the semantic features of the text itself through the multi-head self-attention mechanism. During the calculation process, and serve as query, key, and value respectively, and are the weight matrices of the i-th group of query, key, and value respectively, d k represents the column dimension of the key, and the N groups of self-attention results are concatenated and finally the multi-head self-attention result at the current time step is generated. The calculation formula is as follows:

[0034]

[0035]

[0036] For the concatenation attention mechanism, and represent the weight matrices of the concatenation attention, is the weight vector, represents the attention of the i-th word at time t, N is the maximum number of words in the sentence, represents the weight of the i-th word at time t, and finally the concatenation attention representation at the current time step is generated. The specific calculation process is as follows:

[0037]

[0038]

[0039]

[0040] The calculation process of the dot - product attention mechanism is similar to that of the concatenation attention mechanism, but the way of calculating the weight of each word is slightly different. In the dot - product attention mechanism, W d represents the weight matrix of the dot - product attention, is the weight vector, represents the attention of the i - th word at time step t, represents the weight of word i at time step t, and finally generates the dot - product attention representation at the current time step

[0041]

[0042]

[0043]

[0044] 3) Generator Fusion Output Module 3

[0045] The following describes how Module 3 in the generator fuses the three attention representations obtained in Module 2 to form a text representation that fully considers the interaction relationship between multiple - path features, and uses the Gumbel - Softmax method to solve the problem of gradient disappearance caused by direct sampling in the generative adversarial network, and finally outputs the generated text.

[0046] Based on the bidirectional gated recurrent unit (GRU), the present invention dynamically fuses each path of semantic features to synthesize the representation relationship of multi - head self - attention, concatenation attention, and dot - product attention in the high - dimensional space, fully learns the internal relationship between each attention, and forms a feature vector that comprehensively considers multiple perspectives. In the update process of the GRU network, x t and h t represent the input and output of the network respectively, W z , W r , W h are trainable parameters, σ is the sigmoid activation function, z t and r t are the update gate and the reset gate respectively, ⊙ is the Hadamard product, and the specific calculation process is as follows:

[0047] z t = σ(W z [h t-1 ,xt )

[0048] r t = σ(W r [h t-1 ,x t )

[0049]

[0050]

[0051] During the process of multi-path feature dynamic fusion, is the representation vector after splicing three attention features. Input into the above bidirectional GRU network to capture the differential information between multi-path features from the forward and backward directions and Splice to obtain the comprehensive attention information at the current time step The specific calculation process is as follows:

[0052]

[0053]

[0054]

[0055]

[0056] Pass the comprehensive attention information through a Multilayer Perceptron (MLP) to obtain an intermediate storage matrix Then, through the gating mechanism of RMC, obtain the storage matrix M at the current time step t and the output vector o t , where ψ1 and ψ2 in the formula represent the gating operations of RMC. The calculation process is as follows:

[0057]

[0058]

[0059]

[0060] Finally, describe how to use the output vector o t Sample and output the generated text. Since the text is discrete data, directly sampling and outputting the text in the generative adversarial network will result in non-differentiable gradient information, leading to the problem of gradient disappearance. The present invention uses Gumbel-Softmax to generate the discrete generated text y t and its differentiable approximation based on the argmax function to solve this problem:

[0061]

[0062]

[0063] where g t represents the Gumbel distribution, d represents the d-th dimension. τ is the temperature control parameter used to control the degree of approximate discrete sampling of y t and at the same time control the balance between the quality and diversity of the generated text.

[0064] 4) Local text feature extraction module 4

[0065] In the discriminator of the model, the present invention uses a convolutional neural network (Convolutional Neural Networks, CNN) with strong local feature capture ability as the local text feature extraction module, and uses multi-size convolutional kernels to perform local feature extraction on real texts and generated texts. Denote the number of words in the longest sentence among all texts as l max , and the word embedding dimension of each word is l e , then the vector representation of the sentence is Next, use convolutional kernels C of different sizes i to perform convolution operations on the sentence matrix to obtain the feature map m under the current convolutional kernel i , where b is the bias vector and ReLU is the activation function:

[0066]

[0067] Perform max-pooling operations on each feature map m i to retain the most prominent discriminative features of the feature map. Finally, concatenate all the feature maps MaxPool(mi) after max-pooling to obtain the local text feature representation m:

[0068] m = [MaxPool(m1); MaxPool(m2); ···]

[0069] 5) Global text feature extraction module 5

[0070] The discriminator of the present invention additionally sets a global text feature extraction module on the basis of module 4, and performs global semantic feature extraction on sentences based on a bidirectional GRU network to obtain long-distance context dependencies in the sentences, so as to make up for the deficiency that the convolutional neural network cannot fully capture the global semantic information of texts. Specifically, extract the classification features of the vector representation E of the sentence through the bidirectional GRU network described in module 3 to obtain the global text feature representation h Y ofn :

[0071] h n = GRU(E Y )

[0072] 6) Discriminator fusion output module 6

[0073] Fuse the local text feature m obtained in module 4 with the global text feature h obtained in module 5 n and obtain the overall sentence feature representation z through ReLU.

[0074] z = ReLU(m + h n )

[0075] With the introduction of the global text feature extraction module, the context information containing dependencies between words increases, while the semantic information of the words themselves decreases. To better utilize the semantic information of the text itself, the present invention introduces Highway Networks, obtains more low-level semantic features based on the overall feature representation z, and at the same time enables more gradient information to flow back. Finally, the text features obtained by the Highway Networks are passed through the feature output layer based on MLP, and the discrimination result of the text is output as a learning signal and fed back to the generator, so that the generator generates high-quality category texts that are more likely to deceive the discriminator.

[0076] The above are the preferred embodiments of the present invention. All changes made according to the technical solutions of the present invention that do not exceed the scope of the technical solutions of the present invention in terms of the functions and effects produced belong to the protection scope of the present invention.

Claims

1. A category text generation system based on feature fusion, characterized in that Including: A category text preprocessing module for vectorizing category text; A multi-way attention module that uses multiple attention mechanisms in the generator to learn multiple attention representations of the text; A generator fusion output module that fuses multiple attention representations and outputs generated text; A local semantic feature extraction module for the discriminator to extract local dependency features of the text; A global semantic feature extraction module for the discriminator to extract long-distance dependency features of the text; A discriminator fusion output module that fuses the local and long-distance dependency features of the text and feeds back the discrimination result to the generator; The generator fusion output module dynamically fuses the semantic features of each path based on bidirectional GRU to synthesize multi-head self-attention Concatenated attention Dot product attention Represents the representation relationship in the high-dimensional space, fully learns the internal relationship between each attention, and forms a feature vector that comprehensively considers multiple perspectives; during the update process of the bidirectional GRU, x t and h t represent the input and output of the network respectively, W z 、W r 、W h are trainable parameters, σ is the sigmoid activation function, z t and r t are the update gate and the reset gate respectively, ⊙ is the Hadamard product, t is the time step, and the specific calculation process is as follows: z t = σ(W z [h t-1 , x t ) r t = σ(W r [h t-1 , x t ) In the process of multi-channel semantic feature dynamic fusion, is the representation vector after splicing three attention features. Input it into a bidirectional GRU to capture the differential information between multi-channel features separately from the forward and backward directions. and Splice them to obtain the comprehensive attention information at the current time step. The specific calculation process is as follows: Integrate the comprehensive attention information Through a multi-layer perceptron, obtain the intermediate storage matrix Then, through the gating mechanism of RMC, obtain the storage matrix M at the current time step t and the output vector o t , where ψ1 and ψ2 represent the gating operations of RMC, and the calculation process is as follows: Use the output vector o t Sample and output the generated text; Using Gumbel-Softmax to generate discrete generated text y t and a differentiable approximation based on the argmax function where g t represents the Gumbel distribution, d represents the d-th dimension, and τ is the temperature control parameter.

2. The category text generation system based on feature fusion according to claim 1, wherein, The category text preprocessing module converts the tokenized text data into a vector form of One-Hot encoding, introduces additional category information, and forms an input vector through an embedding layer.

3. The category text generation system based on feature fusion according to claim 1, characterized in that The local semantic feature extraction module uses convolutional kernels of different sizes to extract features from each generated text and real text, and splices the respective feature maps after max pooling to obtain local context dependency features.

4. A category text generation system based on feature fusion according to claim 1, characterized in that The global semantic feature extraction module uses bidirectional GRU to extract global semantic features of the sentence, thereby obtaining long-distance context dependency features in the sentence and making up for the deficiency that the convolutional neural network cannot fully capture the global semantic information of the text.

5. The category text generation system based on feature fusion according to claim 1, wherein The discriminator fusion output module fuses the local semantic features and global semantic features, enables more gradient information to be backpropagated through a high-speed neural network, and feeds the discrimination result back to the generator as a learning signal, enabling the generator to generate high-quality category text that is more likely to deceive the discriminator.

Images