A contrastive text style transfer method based on disentangled representation

Abstract

The present application relates to text style transfer technology, and provides a contrast text style transfer method based on disentangled representation, aiming to solve the problems that the existing text style transfer is not accurate enough to represent the style features in style transfer and it is difficult to generate text with target style from learned hidden representation, etc. The present application mainly comprises: a pre-trained disentangled representation model, which obtains a more accurate text representation method by separating the feature representation into style representation and content representation; and a contrast text style transfer method based on disentangled representation, which, after obtaining a more accurate representation based on the above module, constructs contrast samples by introducing self-generated enhanced data samples as new samples, and selects appropriate contrast loss and contrast training strategy to guide the learning of the style transfer model, so that the final model learns the ability to transfer the existing style of the input text to the target style.

Description

TECHNICAL FIELD

[0001] The present application relates to natural language processing technology, in particular to a text style transfer method. BACKGROUND

[0002] Text style transfer aims to convert a text of one style into a text of another style while keeping the original meaning of the text content unchanged. With the rapid development of deep learning technology, text style transfer technology has attracted extensive attention and application. Currently, text style transfer technology has made significant progress and has been applied in many fields. For example, in the field of literary creation, text style transfer technology can be used to convert modern novels into classical style, providing new inspiration and expression for literary creation. In the field of social media, text style transfer technology can be used to convert formal text into colloquial style, improving the readability and affinity of the text.

[0003] Traditional text style transfer methods mainly rely on rules or templates, and style transfer is achieved by manually defining rules for style conversion. However, this method is limited by the design of rules and the generalization ability of templates, and it is often difficult to handle complex text styles and diverse conversion requirements. In recent years, the rise of deep learning technology has provided a new solution to text style transfer. Text style transfer methods based on deep learning use attention mechanisms and other technologies to capture key information in text. Most of these methods are based on encoder-decoder architecture, but another type of method based on generative adversarial networks is also used to solve the style transfer problem. The encoder-decoder architecture for style conversion usually uses an encoder to create a latent representation of the input sentence, and a decoder to generate an output sentence conditioned on the latent representation, while a classifier determines the style label of the output sentence. The encoder of the style transfer model based on generative adversarial network corresponds to the generator, and the classifier corresponds to the discriminator.

[0004] However, the above two methods face challenges, i.e., it is difficult to completely separate style information from the semantics of a sentence. In addition, the encoder and decoder based on recurrent neural networks cannot well handle long-term dependencies under the intervention of hidden representations, resulting in poor preservation of non-style semantic content. Overall, existing methods are not sufficient to enable the model to obtain accurate representations. SUMMARY

[0005] To solve the problem that style and content information are difficult to separate and non-style semantic content information is poorly preserved in the prior art, resulting in the model being insufficient to obtain accurate representations, the present application provides a text style transfer method based on disentangled representation and contrastive learning.

[0006] The technical solution adopted by the present application to solve the above technical problems is a contrastive text style transfer method based on disentangled representation, comprising the steps of:

[0007] 1. A de-entangled representation model pre-training step:

[0008] 1.1 Obtain a training corpus and preprocess the training corpus; for example, select the widely used StylePTB dataset as the training corpus;

[0009] 1.2 Input the input sample in the training corpus into the parser in the de-entangled representation model for de-entanglement; the parser is composed of a style encoder and a content encoder, which is used to decompose the text representation into style representation and content representation;

[0010] 1.3 Construct positive and negative sample pairs according to the style representation and the content representation; the content needs to be preserved, so the content representation of the source sentence and the target sentence is constructed as a positive sample pair; the remaining style representation and content representation are constructed as a negative sample pair;

[0011] 1.4 Use the decoder in the de-entangled representation model to restore the text representation from the de-entangled style representation and content representation and output to the contrastive text style transfer model;

[0012] 1.5 Use the constructed positive and negative sample pairs to calculate the de-entanglement loss, and calculate the inverse de-entanglement loss between the positive and negative samples and the text representation restored by the decoder to guide the learning of the de-entangled representation model, so that the de-entangled representation model has the ability to correctly separate the style representation and the content representation, thereby obtaining a more accurate text representation;

[0013] 2. A contrastive text style transfer model training step based on de-entangled representation:

[0014] 2.1 The contrastive text style transfer model receives the text representation;

[0015] 2.2 The contrastive text style transfer model outputs automatically generated candidate samples through a beam search algorithm, and introduces the candidate samples into the same batch of contrastive sample set, and expands the contrastive sample set in an enhanced manner;

[0016] 2.3 Select N-Pairs loss as the contrastive loss to guide the contrastive text style transfer model to learn to output the style transfer of the candidate sample to the target style; through the decoding loss to guide the contrastive text style transfer model to learn to output the content of the candidate sample relative to the input text representation unchanged;

[0017] 3. Test step:

[0018] Determine the target style, input the source text into the de-entangled representation model, and output the text representation to the contrastive text style transfer model; the de-entangled representation model outputs the text with unchanged content and the style transferred to the target style.

[0019] The beneficial effects of the present application are:

[0020] By utilizing disentangled representation learning, the correct separation of style representation and content representation is achieved, so that the model can obtain more accurate text representation, and the problems of difficult separation of style and content and difficult retention of non-style information in the prior art are solved.

[0021] At the same time, based on the disentangled representation, contrastive learning is introduced to generate text conforming to the expected target style, which to some extent alleviates the exposure bias problem caused by inconsistent data in the training and testing process, and achieves the ultimate goal of text style transfer.

[0022] The beneficial effects of the present application are that the text style transfer is more efficient and convenient, and the style transfer effect is also guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The overall flowchart of the present application is shown in the figure;

[0024] Figure 2 The pre-training disentangled representation model flowchart of the present application is shown in the figure;

[0025] Figure 3 The training of the contrastive text style transfer model based on the disentangled representation flowchart of the present application is shown in the figure; DETAILED DESCRIPTION

[0026] The present application will be further described below in conjunction with the accompanying drawings:

[0027] Embodiment

[0028] The contrastive text style transfer method based on the disentangled representation in the present embodiment includes two parts of training a disentangled representation model and training a contrastive text style transfer model based on the disentangled representation, which will be described in detail below in conjunction with the accompanying drawings:

[0029] As shown in the figure, training the disentangled representation model includes: Figure 2

[0030] 1.1 Select appropriate pre-training corpus and preprocess it:

[0031] 1.1.1 Select the benchmark dataset StylePTB commonly used in the field of text style transfer as the pre-training corpus, which includes 21 text style conversion tasks. Each task corresponds to a subset containing sentence pairs (source, target). In these sentence pairs, the style of the source sentence is converted to the style of the target sentence, involving the transfer of vocabulary, semantics and syntax. Part of the sample data set can be seen in the following table:

[0032] ​

[0033]

[0034] 1.1.2 Convert input sample to embedding vectors: Convert the word vectors, text vectors and position vectors of the input sample as embedding vectors.

[0035] 1.1.3 Preprocess input sample: Limit the length of the embedding vector representation of the input sample, limit its maximum length to 512 tokens, and the excess part is truncated.

[0036] 1.2 Disentangle the input text in the training corpus through a parser

[0037] 1.2.1 Build a parser: The parser consists of a style encoder and a content encoder, which can decompose the text into style representation and content representation. Both encoders are implemented by a model formed by a stack of 12-layer Transformer encoders, and the intermediate encoder enhances the semantic representation ability of the model through attention mechanism.

[0038] 1.2.2 Obtain the disentangled results: input the above embedding vectors into the parser to obtain the corresponding style representation and content representation. Given a sample pair {X, Y}, after passing through the two encoders, four representations {X S , X C} and {Y S , Y C} can be obtained, which represent the style and content representation of the source sentence X and the target sentence Y respectively.

[0039] 1.3 Construct positive and negative sample pairs according to style representation and content representation:

[0040] 1.3.1 Design construction strategy: content information needs to be preserved, so the content representation of the source sentence and the target sentence is constructed as a positive sample pair. The rest of the style representation and content representation are constructed as a negative sample pair.

[0041] 1.3.2 Construct positive samples: for the given sample pair {X, Y} above, after obtaining its style representation and content representation through the style encoder and content encoder, {X C , Y C} is constructed as a positive sample pair.

[0042] 1.3.3 Construct negative samples: for the given sample pair {X, Y} above, after obtaining its style representation and content representation through the style encoder and content encoder, {X C , Y S}, {X C , X S}, {X S , YS},{X S ,Y C},{Y C ,Y S} are constructed as negative sample pairs.

[0043] 1.4 Recovering the original text representation from the disentangled style and content representations using a decoder:

[0044] 1.4.1 Building the decoder: Similar to the parser, the decoder is composed of the Decoder part of the Transformer, which contains 12 layers of Decoder.

[0045] 1.4.2 Recovering the source representation: Using the obtained style and content representations, the original text representation is recovered. After the parser disentangles the input, the model can learn more accurate representations. That is, according to the style and content representations obtained by the above disentanglement, the decoder converts them into {X * ,Y *} so as to match the original input.

[0046] 1.5 Using the constructed positive and negative sample pairs to calculate the disentanglement loss:

[0047] 1.5.1 Designing the disentanglement loss:

[0048] The disentanglement loss calculation formula is as follows:

[0049]

[0050] where, represents the training corpus, the numerator takes the positive sample pair as the input of the similarity function, and the denominator takes all negative sample pairs as the input of the similarity function. The function φ(u,v) = exp(sim(u,v) / τ), τ is a hyperparameter, and sim(u,v) is the cosine similarity function between sample u and sample v.

[0051] 1.5.2 Designing the inverse disentanglement loss:

[0052] The inverse disentanglement loss calculation formula is as follows:

[0053]

[0054] where B represents all samples in the same batch, and are calculated in a similar manner.

[0055] 1.5.3 The loss function guides the model learning:

[0056] The loss of the entire disentangled representation model is:

[0057]

[0058] The model is guided to learn by calculating the loss between positive and negative sample pairs during disentanglement and the inverse disentanglement loss between the sample and the decoder recovered representation, so that the model has the ability to correctly disentangle style and content, thereby obtaining more accurate text representation.

[0059] 2. Training a contrastive text style transfer model based on disentangled representation: as shown in Figure 3 After obtaining accurate representation, the existing style of the text needs to be transferred to the target style under the premise of keeping the content unchanged. This step is achieved by training a text style transfer model based on disentangled representation, and the detailed process is as follows.

[0060] 2.1 Preparation:

[0061] 2.1.1 Determine training data and input: the data set used to train the model is still the Yelp data set. The text representation learned by the above disentangled representation model is used as the input of the contrastive text style transfer, and further generates the text whose content remains unchanged but the style is transferred to the target style. That is, {X * ,Y *} is used as the input of the contrastive text style transfer model, where X * is the source text x, and Y * is the real sentence y.

[0062] 2.1.2 Build a model for style transfer: the model uses an encoder-decoder architecture, which consists of an input layer, an embedding layer, an encoding layer, and an output layer. The output of the encoder is used as the input of the decoder, which goes through the encoder-decoder attention layer and the decoding layer to finally output. The model is then retrained based on the contrastive learning method.

[0063] 2.2 Enhance contrastive samples:

[0064] 2.2.1 Generate candidate samples: given a real sentence y, a bundle width b, and a pre-trained decoder g, generate b candidate samples and their likelihood probabilities and representations through the decoder g. And select b samples as candidate samples.

[0065] 2.2.2 Build a sample set: first, introduce the above candidate samples into the same batch sample set B, and expand the sample set through enhancement. At this time, the sample set within the same batch becomes K+b, where K is the preset number of expanded samples.

[0066] 2.2.3 Design a ranking function: according to the given true value, calculate the similarity between the contrastive samples in the same batch and the true value, and then sort the contrastive samples according to the similarity. The function is represented as rank(·).

[0067] 2.2.4 Constructing contrastive sample pairs: First, a threshold is specified, and the ranking results described above are used to determine whether the sample is a positive sample or a negative sample. That is, if the similarity of the sample is greater than the threshold, the sample can be constructed as a positive sample, and if the similarity of the sample is less than the threshold, the sample is constructed as a negative sample.

[0068] 2.3 Designing a contrastive loss function: In order to guide the model to learn the ability to generate text that conforms to the target style, a suitable contrastive loss function needs to be designed. Most existing contrastive loss functions do not consider the differences between negative samples, so the N-Pairs loss is selected as the contrastive loss, which is calculated as follows:

[0069]

[0070] wherein ξ = r * (rank(y - )-rank(y + )), which is used to reflect the quality difference between the contrastive sample pairs. Wherein r is a hyperparameter.

[0071] 2.4 Designing a decoding strategy: The similarity function learned when calculating the contrastive loss is introduced into the decoding loss, which can further enhance the ability of the model to retain content information. The calculation formula of the decoding target is:

[0072]

[0073] wherein represents all contrastive samples except y, and a is used to balance the strength of the similarity function and the model likelihood probability, and a is usually 0.5.

[0074] At this point, the entire model training is complete. During inference, the text with the original style is first passed through the disentangled representation model to obtain a more accurate representation, and then the representation is input into the contrastive style transfer model as the original style representation. The model generates text with the target style as the control condition, that is, the style transfer of the text is realized.

[0075] The present application proposes a contrastive text style transfer method based on disentangled representation. The method disentangles the text features into style representation and content representation by pre-training the disentangled representation model, and realizes more accurate text representation. Furthermore, the self-generated enhanced data samples are used to construct contrastive samples, and the model is guided to learn style transfer through contrastive learning. Finally, the present application successfully realizes the ability to transfer the existing style of the input text to the target style, and solves the problems of existing methods, such as the difficulty of accurately representing style features and retaining original content features. Experiments prove that the method has good performance and performance in text style transfer, and can obtain more accurate and effective results.

[0076] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited thereto, any change or replacement within the technical range disclosed by the present application, which can be easily thought by any person skilled in the art, should be covered within the protection scope of the present application.

Claims

1. A contrastive text style transfer method based on deentangled representation, characterized in that, The method comprises the steps of: 1) a disentangled representation model pre-training step: obtain a pre-training corpus, and pre-process the pre-training corpus; input the pre-processed input samples in the pre-training corpus into a parser in the disentangled representation model for disentanglement; the parser is composed of a style encoder and a content encoder, and is used for decomposing a text representation into a style representation and a content representation; a positive-negative sample pair is constructed according to the style representation and the content representation; wherein the content representations of the source sentence and the target sentence are constructed as a positive sample pair; the rest of the style representations and the content representations are constructed into a negative sample pair two by two; a decoder in the disentangled representation model restores the disentangled style representation and the content representation into a text representation, and outputs the text representation to a contrastive text style transfer model; in the pre-training step, the disentangled loss is calculated by using the constructed positive-negative sample pair, and the inverse disentangled loss between the positive-negative sample and the text representation restored by the decoder is calculated to guide the learning of the disentangled representation model, so that the disentangled representation model has the ability to correctly separate the style representation and the content representation and output the correct text representation; 2) a contrastive text style transfer model training step based on the disentangled representation: the contrastive text style transfer model receives a text representation, outputs an automatically generated candidate sample through a beam search algorithm, and introduces the candidate sample into the same batch of contrastive sample set in an enhanced manner to expand the contrastive sample set; in the training step, the N-Pairs loss is selected as the contrast loss to guide the learning of the contrastive text style transfer model to output the style transfer of the candidate sample to the target style; the decoding loss is used to guide the learning of the contrastive text style transfer model to output the content of the candidate sample that remains unchanged relative to the input text representation; 3) a test step: determine a target style, input a source text into a disentangled representation model, output a text representation from the disentangled representation model to a contrastive text style transfer model, and output a text with unchanged content and transferred style to the target style from the disentangled representation model.

2. The method of claim 1, wherein, The pre-training corpus is a benchmark dataset StylePTB.

3. The method of claim 1, wherein, The pre-processing method of the pre-training corpus is as follows: convert the input samples in the pre-training corpus into embedded vector representations, limit the embedded vector representations to a maximum length, and truncate the excess part; the embedded vector representations of the input samples include word vectors, text vectors and position vectors of the input samples.

4. The method of claim 3, wherein, Both the style encoder and the content encoder are formed by stacking 12 layers of the encoder Encoder of the Transformer; the encoders in the middle layer enhance the semantic representation ability of the model through attention mechanism; the decoder is formed by stacking 12 layers of the decoder Decoder of the Transformer.

5. The method of claim 1, wherein, The content representation of the source sentence and the target sentence is constructed as a positive sample pair represented as {X C ,Y C}; The two-by-two between the remaining style representations and content representations are constructed into a negative sample pair, including: {X C ,Y S}, {X C ,X S}, {X S ,Y S}, {X S ,Y C}, {Y C ,Y S} ; wherein X S represents the style representation of source sentence X, X C represents the content representation of source sentence X, Y S represents the style representation of target sentence Y, Y C represents the content representation of target sentence Y.

6. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to realize the steps of the method of claim 1.

Images