Comparison text style migration method based on de-entanglement representation

Abstract

The invention relates to a text style migration technology, provides a contrast text style migration method based on deentanglement representation, and aims to solve the problems that existing text style migration cannot accurately represent style features in style migration, and a text with a target style is difficult to generate through learned hidden representation. The method mainly comprises the following steps: pre-training a de-entanglement representation model, and dissociating feature representation into style representation and content representation to obtain a more accurate text representation method; according to the deentanglement representation-based contrast text style migration method, after more accurate representation is obtained based on the modules, a self-generated enhanced data sample is introduced as a new sample to construct a contrast sample; and proper comparison loss and comparison training strategies are selected to guide the learning of the style migration model, so that the final model learns the ability of migrating the existing style of the input text to the target style.

Description

Technical Field

[0001] The present invention relates to natural language processing technology, and in particular to a text style transfer method. Background Art

[0002] Text style transfer aims to convert text of one style into another style while keeping the original meaning of the text content unchanged. With the rapid development of deep learning technology, text style transfer technology has received widespread attention and application. At present, text style transfer technology has made significant progress and has been applied in many fields. For example, in the field of literary creation, text style transfer technology can be used to convert modern novels into classical style, providing new inspiration and expression for literary creation. In the field of social media, text style transfer technology can be used to convert formal text into colloquial style to improve the readability and affinity of the text.

[0003] Traditional text style transfer methods mainly rely on rules or templates, and achieve style transfer by manually defining the rules of style transfer. However, this method is limited by the design of rules and the generalization ability of templates, and it is often difficult to handle complex text styles and diverse conversion requirements. In recent years, the rise of deep learning technology has provided new solutions for text style transfer. Text style transfer methods based on deep learning use technologies such as attention mechanisms to capture key information of text. Most of these methods are based on the encoder-decoder architecture, but there is also another type of method based on generative adversarial networks that is used to solve the style transfer problem. The encoder-decoder architecture for style transfer usually uses an encoder to create a latent representation of the input sentence, uses a decoder to generate an output sentence conditioned on the latent representation, and a classifier determines the style label of the output sentence. In the style transfer model based on generative adversarial networks, the encoder corresponds to the generator and the classifier corresponds to the discriminator.

[0004] However, the above two methods face challenges, namely, it is difficult to completely separate the style information from the semantics of a sentence. In addition, the encoder and decoder based on recurrent neural networks cannot handle the long-term dependency problem well under the intervention of latent representation, resulting in poor preservation of non-style semantic content. In general, the existing methods are not enough for the model to obtain accurate representation. Summary of the invention

[0005] In order to solve the problem in the prior art that style and content information are difficult to separate and non-style semantic content information is poorly retained, resulting in the model being insufficient to obtain accurate representation, the present invention provides a text style transfer method based on disentangled representation and contrastive learning.

[0006] The technical solution adopted by the present invention to solve the above technical problems is a contrast text style transfer method based on disentangled representation, comprising the steps of:

[0007] 1. Pre-training steps of disentanglement representation model:

[0008] 1.1 Obtain training corpus and preprocess the training corpus; for example, select the widely used StylePTB dataset as training corpus;

[0009] 1.2 Input the input samples in the training corpus into the parser in the disentanglement representation model for disentanglement; the parser consists of a style encoder and a content encoder, which is used to decompose the text representation into style representation and content representation;

[0010] 1.3 Construct positive and negative sample pairs based on style representation and content representation; the content needs to be retained, so the content representation of the source sentence and the target sentence is constructed as a positive sample pair; the remaining style representations and content representations are constructed into a negative sample pair;

[0011] 1.4 Using the decoder in the disentangled representation model, the disentangled style representation and content representation are restored to the text representation and output to the comparative text style transfer model;

[0012] 1.5 Use the constructed positive and negative sample pairs to calculate the disentanglement loss, and calculate the inverse disentanglement loss between the positive and negative samples and the text representation restored by the decoder to guide the learning of the disentanglement representation model, so that the disentanglement representation model has the ability to correctly disentangle the style representation and content representation, thereby obtaining a more accurate text representation;

[0013] 2. Training steps of contrastive text style transfer model based on disentangled representation:

[0014] 2.1 Comparison Text Style Transfer Model receives text representation;

[0015] 2.2 The contrastive text style transfer model outputs automatically generated candidate samples through the beam search algorithm, and introduces the candidate samples into the same batch of contrast sample sets, expanding the contrast sample set by enhancement;

[0016] 2.3 Select N-Pairs loss as the contrast loss to guide the contrast text style transfer model to learn to transfer the style of the output candidate samples to the target style; use the decoding loss to guide the contrast text style transfer model to learn to output the content of the candidate samples relative to the input text representation remains unchanged;

[0017] 3. Test steps:

[0018] Determine the target style, input the source text into the disentangled representation model, the disentangled representation model outputs the text representation to the contrast text style transfer model, the output content of the disentangled representation model remains unchanged, and the style is transferred to the text of the target style.

[0019] The beneficial effects of the present invention are:

[0020] By utilizing disentangled representation learning, we can correctly separate style representation and content representation, so that the model can obtain more accurate text representation, solving the problems of existing methods such as difficulty in separating style and content and difficulty in retaining non-style information.

[0021] At the same time, based on the disentangled representation, contrastive learning is introduced to generate text that meets the expected target style, which to a certain extent alleviates the exposure bias problem caused by data inconsistency during training and testing, and achieves the ultimate goal of text style transfer.

[0022] The beneficial effect of the present invention is that text style transfer is more efficient and convenient, and the style transfer effect is also guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is the overall flow chart of the present invention;

[0024] Figure 2 This is a flow chart of the pre-trained disentanglement representation model of the present invention;

[0025] Figure 3 A flow chart of a contrastive text style transfer model based on disentangled representation for training of the present invention; DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with the accompanying drawings:

[0027] Example

[0028] The contrastive text style transfer method based on disentangled representation in this embodiment includes two parts: training a disentangled representation model and training a contrastive text style transfer model based on disentangled representation, which will be specifically introduced below in conjunction with the accompanying drawings:

[0029] like Figure 2 As shown, training the disentangled representation model includes:

[0030] 1.1 Select appropriate pre-training corpus and pre-process it:

[0031] 1.1.1 The StylePTB dataset, a commonly used benchmark dataset in the field of text style transfer, is selected as the pre-training corpus. The dataset includes 21 text style transfer tasks. Each task corresponds to a subset containing sentence pairs (source, target). In these sentence pairs, the style of the source sentence is converted to the style of the target sentence, involving the transfer of vocabulary, semantics, and syntax. Some examples of the dataset can be seen in the following table:

[0032]

[0033]

[0034] 1.1.2 Convert the input sample into an embedding vector: The word vector, text vector, and position vector of the input sample are represented as embedding vectors.

[0035] 1.1.3 Preprocess the input samples: limit the length of the embedding vector representation of the input samples to a maximum of 512 tokens, and truncate the excess.

[0036] 1.2 Detangling the input text in the training corpus through a parser

[0037] 1.2.1 Constructing the parser: The parser consists of a style encoder and a content encoder, which can decompose the text into style representation and content representation. Both encoders are implemented by a model formed by stacking 12 layers of Transformer encoders. The intermediate encoder uses the attention mechanism to enhance the semantic representation ability of the model.

[0038] 1.2.2 Obtaining disentangled results: Inputting the above embedding vector into the parser can obtain the disentangled corresponding style representation and content representation. Given a sample pair {X, Y}, four representations {X S ,X C} and {Y S ,Y C}, representing the style and content representation of the source sentence X and the target sentence Y respectively.

[0039] 1.3 Construct positive and negative sample pairs based on style representation and content representation:

[0040] 1.3.1 Design construction strategy: Content information needs to be preserved, so the content representation of the source sentence and the target sentence is constructed as a positive sample pair. The remaining style representations and content representations are constructed as a negative sample pair.

[0041] 1.3.2 Constructing positive samples: For the given sample pair {X, Y}, after the style encoder and content encoder obtain their style representation and content representation respectively, {X C ,Y C} is constructed as a positive sample pair.

[0042] 1.3.3 Constructing negative samples: For the given sample pair {X, Y} mentioned above, after the style encoder and content encoder obtain their style representation and content representation respectively, {X C ,Y S}, {X C ,X S}, {X S ,YS}, {X S ,Y C}, {Y C ,Y S} is constructed as a negative sample pair.

[0043] 1.4 Use a decoder to restore the disentangled style representation and content representation to the original text representation:

[0044] 1.4.1 Constructing the decoder: Similar to the parser, the decoder consists of the Decoder part of the Transformer, which contains 12 layers of Decoder.

[0045] 1.4.2 Restoring the source representation: Use the obtained style representation and content representation to restore the original text representation. After the parser dissociation, the model can learn a more accurate representation. That is, according to the above dissociation of the style representation and content representation, the decoder converts them into {X * ,Y *} to match the original input.

[0046] 1.5 Use the constructed positive and negative sample pairs to calculate the disentanglement loss:

[0047] 1.5.1 Design of disentanglement loss:

[0048] The formula for calculating the disentanglement loss is as follows:

[0049]

[0050] in, Represents the training corpus, the numerator uses the positive sample pairs as the input of the similarity function, and the denominator uses all negative sample pairs as the input of the similarity function. Function φ(u,v)=exp(sim(u,v) / τ), τ is a hyperparameter, sim(u,v) is the cosine similarity function between sample u and sample v.

[0051] 1.5.2 Design of inverse solution of entanglement loss:

[0052] The formula for calculating the inverse disentanglement loss is as follows:

[0053]

[0054] Among them, B represents all samples in the same batch, and The calculation method is similar.

[0055] 1.5.3 Loss function guides model learning:

[0056] The loss of the entire disentanglement representation model is:

[0057]

[0058] By calculating the loss between positive and negative sample pairs during the disentanglement process and the inverse disentanglement loss between the samples and the representation restored by the decoder, we guide model learning, enabling the model to correctly disentangle style and content, thereby obtaining a more accurate text representation.

[0059] 2. Train a contrastive text style transfer model based on disentangled representation: Figure 3 As shown in Figure 1, after obtaining an accurate representation, it is necessary to transfer the existing style of the text to the target style while keeping the content unchanged. This step is achieved by training a text style transfer model based on disentangled representation. The detailed process is as follows.

[0060] 2.1 Preparation:

[0061] 2.1.1 Determine the training data and input: The dataset used for training the model is still the Yelp dataset. The text representation learned by the above disentangled representation model is used as the input for the comparative text style transfer, and the generated content remains unchanged, but the style is transferred to the target style. * ,Y *} as the input of the contrastive text style transfer model, where X * As source text x, Y * As a true sentence y exists.

[0062] 2.1.2 Constructing a model for style transfer: This model uses an encoder-decoder architecture. The encoder part consists of an input layer, an embedding layer, an encoding layer, and an output layer. The decoder takes the encoder output as input, and passes through the encoder-decoder attention layer and the final output of the decoding layer. The model is then trained based on the contrastive learning method.

[0063] 2.2 Enhanced contrast samples:

[0064] 2.2.1 Generate candidate samples: Given a real sentence y, a beam width b, and a pre-trained decoder g, generate b candidate samples, their likelihood probabilities and representations through the decoder g. The selected b samples are used as candidate samples.

[0065] 2.2.2 Constructing a sample set: First, the above candidate samples are introduced into the sample set B of the same batch, and the sample set is expanded by enhancement. At this time, the sample set in the same batch becomes K+b, where K is the preset number of expanded samples.

[0066] 2.2.3 Design a ranking function: Based on the given true value, calculate the similarity between the comparison samples in the same batch and the true value, and then sort the comparison samples according to the similarity. This function is represented by rank(·).

[0067] 2.2.4 Constructing comparison sample pairs: First, specify a threshold, and use the above sorting results to determine whether the sample is a positive sample or a negative sample. That is, if the similarity is greater than the threshold, the sample can be constructed as a positive sample, and if the similarity is less than the threshold, the sample can be constructed as a negative sample.

[0068] 2.3 Designing a contrastive loss function: In order to guide the model to learn the ability to correctly generate text that meets the target style, it is necessary to design a suitable contrastive loss function. Most existing contrastive loss functions do not consider the differences between negative samples, so N-Pairs loss is selected as the contrastive loss, which is calculated as follows:

[0069]

[0070] where ξ=r*(rank(y - )-rank(y + )) is used to reflect the quality difference between the contrasting sample pairs. Among them, r is a hyperparameter.

[0071] 2.4 Design decoding strategy: Introducing the similarity function learned when calculating the contrast loss into the decoding loss can further enhance the model's ability to retain content information. The calculation formula for the decoding target is:

[0072]

[0073] in represents all comparison samples except y, and α is used to balance the strength of the similarity function and the model likelihood probability, usually α = 0.5.

[0074] At this point, the entire model training is completed. During inference, the text with the original style first passes through the disentanglement representation model to obtain a more accurate representation, and then the representation is input as the original style representation to the contrastive style transfer model. The model uses the target style as a control condition and conditionally generates text with the target style for the original style representation, that is, realizing the style transfer of the text.

[0075] The present invention proposes a comparative text style transfer method based on disentangled representation. The method dissociates text features into style representation and content representation by pre-training a disentangled representation model, thereby achieving more accurate text representation. Furthermore, a comparative sample is constructed using self-generated enhanced data samples, and the model is guided to learn style transfer through comparative learning. Ultimately, the invention successfully achieves the ability to transfer the existing style of the input text to the target style, solving the problems that the existing methods cannot accurately represent style features and that it is difficult to retain the original content features. Experiments have shown that the method has good performance and performance in text style transfer, and can obtain more accurate and effective results.

[0076] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by technicians in this technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A contrastive text style transfer method based on disentangled representation, It is characterized in that Includes steps: 1) Pre-training steps of disentanglement representation model: Obtain pre-training corpus and pre-process the pre-training corpus; The preprocessed input samples in the pre-training corpus are input into the parser in the disentanglement representation model for disentanglement; the parser consists of a style encoder and a content encoder, which is used to decompose the text representation into style representation and content representation; construct positive and negative sample pairs according to the style representation and content representation; among them, the content representation of the source sentence and the target sentence is constructed as a positive sample pair; the remaining style representations and content representations are constructed as a negative sample pair; The decoder in the disentangled representation model restores the disentangled style representation and content representation into a text representation, and outputs the text representation to the contrastive text style transfer model; In the pre-training step, the constructed positive and negative sample pairs are used to calculate the disentanglement loss, and the inverse disentanglement loss between the positive and negative samples and the text representation restored by the decoder is calculated to guide the learning of the disentanglement representation model, so that the disentanglement representation model has the ability to correctly disentangle the style representation and content representation and output the correct text representation; 2) Training steps of contrastive text style transfer model based on disentangled representation: The contrastive text style transfer model receives text representation, outputs automatically generated candidate samples through the beam search algorithm, and introduces the candidate samples into the same batch of contrastive sample sets, thereby expanding the contrastive sample set through enhancement. In the training step, N-Pairs loss is selected as the contrast loss to guide the contrastive text style transfer model to learn to transfer the style of the output candidate samples to the target style; decoding loss is used to guide the contrastive text style transfer model to learn to keep the content of the output candidate samples relative to the input text representation unchanged; 3) Test steps: Determine the target style, input the source text into the disentangled representation model, the disentangled representation model outputs the text representation to the contrast text style transfer model, the output content of the disentangled representation model remains unchanged, and the style is transferred to the text of the target style.

2. The method according to claim 1, It is characterized in that The pre-training corpus is the benchmark dataset StylePTB.

3. The method according to claim 1, It is characterized in that The specific method for preprocessing the pre-training corpus is: The input samples in the pre-training corpus are converted into embedded vector representations, the maximum length of the embedded vector representation is limited, and the excess part is truncated; the embedded vector representation of the input sample includes the word vector, text vector and position vector of the input sample.

4. The method according to claim 3, It is characterized in that Both the style encoder and the content encoder are formed by stacking 12 layers of Transformer encoders. The encoders in the middle layer use the attention mechanism to enhance the semantic representation ability of the model. The decoder is formed by stacking 12 layers of Transformer decoders.

5. The method according to claim 1, It is characterized in that The content representation of the source sentence and the target sentence is constructed as a positive sample pair represented as {X C ,Y C }; The remaining style representations and content representations are constructed into a negative sample pair, including: {X C ,Y S }, {X C ,X S }, {X S ,Y S }, {X S ,Y C }, {Y C ,Y S }; Among them, X S represents the style representation of the source sentence X, X C is the content representation of the source sentence X, Y S is the style representation of the target sentence Y, Y C is the content representation of the target sentence Y.

6. A computer program product comprising a computer program / instructions, It is characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

Images