A parallel corpus data pair construction method and device and a storage medium

Abstract

Embodiments of the present application disclose a parallel corpus data pair construction method and device and a storage medium. The method comprises: inputting first corpus data with first style characteristics into a first style conversion model to obtain second corpus data with second style characteristics; the first style conversion model is obtained based on back translation training; inputting the first corpus data into a second style conversion model to obtain third corpus data with the second style characteristics; the second style conversion model is obtained based on adversarial training; calculating a first score corresponding to the second style characteristics of the second corpus data; calculating a second score corresponding to the second style characteristics of the third corpus data; if the first score is greater than the second score, constructing a first parallel corpus data pair using the second corpus data and the first corpus data; if the first score is not greater than the second score, constructing the first parallel corpus data pair using the third corpus data and the first corpus data, thereby solving the problem of scarcity of parallel corpus data pairs.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and more specifically, to a method, apparatus and storage medium for constructing parallel corpus data pairs. Background Technology

[0002] In today's era of booming internet development, information exchange between people is becoming increasingly frequent. Whether in oral conversation or written expression, language, as a means of human communication, is context-dependent. Different times, specific places, and scenarios often convey the speaker's personality traits, state, or intentions through each utterance. For example, when people question the correctness of an answer, they often ask, "Is that really the result?" rather than expressing a definitive statement like, "Yes, that's right." In relatively formal settings, attention should be paid to the standardization of expression. Comparing "Please sit down" and "Come, sit down," the former is relatively polite and formal, while the latter is more casual. Therefore, plain and straightforward expression is not a singular form; personalized needs make the ability of machines to understand and implement style transfer behind language particularly important.

[0003] Style transfer, while preserving the main content as much as possible, generates text in a different style by editing style-related words or rewriting the text. Current research approaches involve supervised training of a pre-trained language model using parallel corpora, and then using the trained model to process the text into different styles. Parallel corpora refer to sentences with attribute style 'a' and their paired sentences with another attribute style 'a'. For example, a sentence with a positive style, "This restaurant is really good," would be paired with a negative style sentence, "This restaurant is really bad." However, the scarcity of parallel corpora makes this approach difficult to implement practically. Summary of the Invention

[0004] In view of this, embodiments of this application disclose a method, apparatus and storage medium for constructing parallel corpus data pairs, thereby realizing the construction of parallel corpus data pairs and solving the problem of the scarcity of parallel corpus data pairs.

[0005] The technical solutions provided in this application are as follows:

[0006] In a first aspect, embodiments of this application provide a method for constructing parallel corpus data pairs, the method comprising:

[0007] The first corpus data with the first style features is input into the first style conversion model to obtain the second corpus data with the second style features; the first style conversion model is obtained based on back-translation training;

[0008] The first corpus data is input into the second style transfer model to obtain a third corpus data with the second style features; the second style transfer model is obtained based on adversarial training.

[0009] Calculate the first score corresponding to the second style feature of the second corpus data;

[0010] Calculate the second score corresponding to the second style feature of the third corpus data;

[0011] If the first score is greater than the second score, then a first parallel corpus data pair is constructed using the second corpus data and the first corpus data;

[0012] If the first score is not greater than the second score, then a first parallel corpus data pair is constructed using the third corpus data and the first corpus data.

[0013] In conjunction with the first aspect described above, in one possible implementation, the method further includes:

[0014] The first pre-trained language model is subjected to supervised fine-tuning using the first parallel corpus data to obtain the second pre-trained language model; the first parallel corpus data includes: the first corpus data and the parallel corpus data corresponding to the first corpus data, wherein the parallel corpus data corresponding to the first corpus data is the second corpus data or the third corpus data.

[0015] The first corpus data is input into the second pre-trained language model to obtain a first output result with the second style features;

[0016] Calculate the first variation value between the first output result and the parallel corpus data corresponding to the first corpus data;

[0017] If the first change magnitude value is greater than a preset threshold, then the first output result is used to replace the parallel corpus data corresponding to the first corpus data to obtain the updated first parallel corpus data pair.

[0018] In conjunction with the first aspect above, in one possible implementation, calculating the first change magnitude value between the first output result and the parallel corpus data corresponding to the first corpus data includes:

[0019] Calculate the first bilingual translation quality assessment BLEU value between the first corpus data and the first output result;

[0020] Calculate the second BLEU value between the first corpus data and the parallel corpus data corresponding to the first corpus data;

[0021] Calculate the third score corresponding to the second style feature of the first output result;

[0022] When the parallel corpus data corresponding to the first corpus data is the second corpus data, the first change magnitude value is calculated using the first score, the first BLEU value, the second BLEU value, and the third score; or,

[0023] When the parallel corpus data corresponding to the first corpus data is the third corpus data, the first change magnitude value is calculated using the second score, the first BLEU value, the second BLEU value, and the third score.

[0024] In conjunction with the first aspect described above, in one possible implementation, the method further includes:

[0025] The first parallel corpus data pair is updated by multiple iterative calculations to obtain a successfully constructed first parallel corpus data pair.

[0026] The current iterative calculation process in the multiple iterative calculations includes:

[0027] The first pre-trained language model is fine-tuned in a supervised manner using the updated first parallel corpus data from the previous loop calculation process to obtain the third pre-trained language model.

[0028] The first corpus data is input into the third pre-trained language model to obtain a second output result with second style features;

[0029] Calculate the second change magnitude value between the second output result and the parallel corpus data corresponding to the first corpus data in the current loop calculation process;

[0030] If the second change magnitude value is greater than the preset threshold, then the second output result is used to replace the parallel corpus data corresponding to the first corpus data in the current iterative calculation process to obtain the updated first parallel corpus data pair in the current iterative calculation process, and then the following steps are performed: the first pre-trained language model is supervisedly fine-tuned using the updated first parallel corpus data pair in the previous iterative calculation process to obtain the third pre-trained language model, until the difference between the transfer accuracy of the third pre-trained language model in the current iterative calculation process and the transfer accuracy of the third pre-trained language model in the previous iterative calculation process is not greater than the preset value, and the successfully constructed first parallel corpus data pair is obtained.

[0031] In conjunction with the first aspect above, in one possible implementation, calculating the second change magnitude value between the second output result and the parallel corpus data corresponding to the first corpus data in the current loop calculation process includes:

[0032] Calculate the third BLEU value between the first corpus data and the second output result;

[0033] Calculate the fourth BLEU value between the first corpus data and the parallel corpus data corresponding to the first corpus data in the current loop calculation process;

[0034] Calculate the fourth score corresponding to the second style feature of the second output result;

[0035] Calculate the fifth score corresponding to the second style feature of the parallel corpus data corresponding to the first corpus data in the current loop calculation process;

[0036] The second change magnitude value is calculated using the third BLEU value, the fourth BLEU value, the fourth score, and the fifth score.

[0037] In conjunction with the first aspect described above, in one possible implementation, the method further includes:

[0038] The fourth corpus data with the second style features is input into the first style conversion model to obtain the fifth corpus data with the first style features;

[0039] The fourth corpus data is input into the second style conversion model to obtain the sixth corpus data with the first style features;

[0040] Calculate the sixth score corresponding to the first style feature of the fifth corpus data;

[0041] Calculate the seventh score corresponding to the first style feature of the sixth corpus data;

[0042] If the sixth score is greater than the seventh score, then a second parallel corpus data pair is constructed using the fifth corpus data and the fourth corpus data;

[0043] If the sixth score is not greater than the seventh score, then a second parallel corpus data pair is constructed using the sixth corpus data and the fourth corpus data.

[0044] In conjunction with the first aspect above, in one possible implementation, the first style transfer model includes a first translation model, a second translation model, a first style decoder, a second style decoder, and a first attribute classifier; the method further includes:

[0045] The first style decoder is trained using the following training process:

[0046] The first training text with the first style features is input into the first translation model to obtain the first translated text with the first style features;

[0047] The first translated text is input into the encoder of the second translation model to obtain the first latent vector without style features;

[0048] The first latent vector is input into the first style decoder to obtain the first back-translated text with the first style features;

[0049] The first back-translated text is input into the first attribute classifier to obtain the first classification result;

[0050] The first style decoder is trained using the first classification result;

[0051] The second style decoder is trained using the following process:

[0052] The second training text with the second style features is input into the first translation model to obtain the second translated text with the second style features;

[0053] The second translated text is input into the encoder of the second translation model to obtain the second latent vector without style features;

[0054] The second latent vector is input into the second style decoder to obtain the second back-translated text with the second style features;

[0055] The second translated text is input into the first attribute classifier to obtain the second classification result;

[0056] The second style decoder is trained using the second classification result.

[0057] In conjunction with the first aspect described above, in one possible implementation, the second style transfer model includes a first style generator, a second style generator, and a second attribute classifier, and the method further includes:

[0058] The first style generator and the second style generator are trained using the following training process:

[0059] The first training text with the first style feature is subjected to noise processing to obtain the first noisy text;

[0060] The first noisy text is input into the first style generator for data reconstruction to obtain a first reconstructed text with the first style features;

[0061] Calculate the cross-entropy loss value between the first reconstructed text and the first training text to obtain the first loss value;

[0062] The first reconstructed text is input into the second attribute classifier to obtain the third classification result;

[0063] The first training text is input into the second style generator for data reconstruction to obtain a second reconstructed text with the second style features;

[0064] The second reconstructed text is then subjected to noise processing to obtain the second noisy text;

[0065] The second noisy text is input into the first style generator for data reconstruction to obtain a third reconstructed text with the first style features;

[0066] Calculate the cross-entropy loss value between the third reconstructed text and the first training text to obtain the second loss value;

[0067] The third reconstructed text is input into the second attribute classifier to obtain the fourth classification result;

[0068] The second training text with the second style feature is subjected to noise processing to obtain the third noisy text;

[0069] The third noisy text is input into the second style generator for data reconstruction to obtain a fourth reconstructed text with the second style features;

[0070] Calculate the cross-entropy loss value between the fourth reconstructed text and the second training text to obtain the third loss value;

[0071] The fourth reconstructed text is input into the second attribute classifier to obtain the fifth classification result;

[0072] The second training text is input into the first style generator for data reconstruction to obtain a fifth reconstructed text with the first style features;

[0073] The fifth reconstructed text is then subjected to noise processing to obtain the fourth noisy text;

[0074] The fourth noisy text is input into the second style generator for data reconstruction to obtain a sixth reconstructed text with the second style features;

[0075] Calculate the cross-entropy loss value between the sixth reconstructed text and the second training text to obtain the fourth loss value;

[0076] The sixth reconstructed text is input into the second attribute classifier to obtain the sixth classification result;

[0077] The first style generator and the second style generator are trained using the first loss value, the second loss value, the third loss value, the fourth loss value, the third classification result, the fourth classification result, the fifth classification result, and the sixth classification result.

[0078] Secondly, embodiments of this application provide an apparatus for constructing parallel corpus data pairs, the apparatus comprising:

[0079] A style conversion unit is used to input first corpus data with first style features into a first style conversion model to obtain second corpus data with second style features; the first style conversion model is obtained based on back-translation training;

[0080] The style conversion unit is further configured to input the first corpus data into the second style conversion model to obtain third corpus data with the second style features; the second style conversion model is obtained based on adversarial training.

[0081] A style score calculation unit is used to calculate the first score corresponding to the second style feature of the second corpus data;

[0082] The style score calculation unit is also used to calculate the second score corresponding to the second style feature of the third corpus data;

[0083] A construction unit is configured to construct a first parallel corpus data pair using the second corpus data and the first corpus data if the first score is greater than the second score;

[0084] The construction unit is further configured to construct a first parallel corpus data pair using the third corpus data and the first corpus data if the first score is not greater than the second score.

[0085] Thirdly, embodiments of this application provide an apparatus for constructing parallel corpus data pairs, including:

[0086] Memory, used to store instructions;

[0087] A processor for executing the instructions in the memory to perform the method for constructing parallel corpus data pairs as described in any of the first aspects above.

[0088] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform the method for constructing parallel corpus data pairs as described in any of the first aspects above.

[0089] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the method for constructing parallel corpus data pairs as described in any of the first aspects above.

[0090] Based on the above technical solution, this application has the following beneficial effects:

[0091] This application discloses a method, apparatus, and storage medium for constructing parallel corpus data pairs. The method includes: inputting first corpus data with a first style feature into a first style conversion model to obtain second corpus data with a second style feature; the first style conversion model is trained based on back-translation; inputting the first corpus data into a second style conversion model to obtain third corpus data with the second style feature; the second style conversion model is trained based on adversarial training; calculating a first score corresponding to the second style feature of the second corpus data; calculating a second score corresponding to the second style feature of the third corpus data; if the first score is greater than the second score, then constructing a first parallel corpus data pair using the second corpus data and the first corpus data; if the first score is not greater than the second score, then constructing a first parallel corpus data pair using the third corpus data and the first corpus data. It can be seen that this application embodiment, through models trained using two different training methods—back-translation and adversarial training—obtains corpus data with different style conversions, enriching the methods for constructing parallel corpus data pairs and solving the problem of the scarcity of parallel corpus data pairs. Furthermore, style scores are calculated on the style-transformed corpus data, and selection is performed based on these scores. This process constructs high-quality first parallel corpus data pairs, thereby improving the overall quality of the parallel corpus data pairs. Attached Figure Description

[0092] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0093] Figure 1 This is a flowchart of a method for constructing parallel corpus data pairs disclosed in an embodiment of this application;

[0094] Figure 2 This is a schematic diagram of the structure of an attribute classifier disclosed in an embodiment of this application;

[0095] Figure 3 A flowchart illustrating another method for constructing parallel corpus data pairs disclosed in an embodiment of this application;

[0096] Figure 4This is a schematic diagram of the framework of a first style transfer model disclosed in an embodiment of this application;

[0097] Figure 5 This is a schematic diagram of the framework of a second style transfer model disclosed in an embodiment of this application;

[0098] Figure 6 This is a schematic diagram of the structure of a device for constructing parallel corpus data pairs disclosed in an embodiment of this application. Detailed Implementation

[0099] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0100] The terms “comprising,” “including,” “having,” and variations thereof, used in this specification, all mean “including but not limited to,” unless otherwise specifically emphasized. It should be noted that in the description of embodiments in this application, terms such as “first,” “second,” etc., are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0101] This application discloses a method, apparatus, and storage medium for constructing parallel corpus data pairs. The method includes: inputting first corpus data with a first style feature into a first style conversion model to obtain second corpus data with a second style feature; the first style conversion model is trained based on back-translation; inputting the first corpus data into a second style conversion model to obtain third corpus data with the second style feature; the second style conversion model is trained based on adversarial training; calculating a first score corresponding to the second style feature of the second corpus data; calculating a second score corresponding to the second style feature of the third corpus data; if the first score is greater than the second score, then constructing a first parallel corpus data pair using the second corpus data and the first corpus data; if the first score is not greater than the second score, then constructing a first parallel corpus data pair using the third corpus data and the first corpus data. It can be seen that this application embodiment, through models trained using two different training methods—back-translation and adversarial training—obtains corpus data with different style conversions, enriching the methods for constructing parallel corpus data pairs and solving the problem of the scarcity of parallel corpus data pairs. Furthermore, style scores are calculated on the style-transformed corpus data, and selection is performed based on these scores. This process constructs high-quality first parallel corpus data pairs, thereby improving the overall quality of the parallel corpus data pairs.

[0102] See Figure 1The flowchart of a method for constructing parallel corpus data pairs disclosed in this application includes:

[0103] S101. Input the first corpus data with the first style features into the first style conversion model to obtain the second corpus data with the second style features;

[0104] The first style transfer model is obtained based on back-translation training.

[0105] S102. Input the first corpus data into the second style conversion model to obtain the third corpus data with the second style features;

[0106] The second style transfer model is obtained based on adversarial training. In one possible implementation, the second style transfer model can be trained using adversarial training of a Generative Adversarial Network (GAN).

[0107] It should be noted that the first style feature and the second style feature are opposing style features. For example, if the first style feature is a positive emotion and the second style feature is a negative emotion, the first corpus data with the first style feature is "This restaurant tastes really good," the second corpus data with the second style feature is "This restaurant tastes really bad," and the third corpus data with the second style feature is "This restaurant tastes really terrible." It is understood that the above is merely an illustrative example and should not be construed as a limitation of this application.

[0108] It should be noted that there is no specific order between S101 and S102. S101 can be executed first and then S102, or S102 can be executed first and then S101, or S101 and S102 can be executed simultaneously. This application does not limit this.

[0109] S103. Calculate the first score corresponding to the second style feature of the second corpus data;

[0110] S104. Calculate the second score corresponding to the second style feature of the third corpus data;

[0111] It should be noted that there is no specific order between S103 and S104. S103 can be executed first and then S104, or S104 can be executed first and then S103, or S103 and S104 can be executed simultaneously. This application does not limit this.

[0112] For the same input, the outputs of the first style transfer model and the second style transfer model will differ in quality, so filtering is necessary.

[0113] In one possible implementation, this embodiment of the application can train an attribute classifier based on supervised data, allowing the attribute classifier to determine the style of an input sentence, such as negative or positive. For example, if the input is "The sun is shining brightly, I'm full of energy every day, I'm so happy," the attribute classifier can determine that it is a positive sentence. The attribute classifier can adopt a Text Convolutional Neural Network (TextCNN) network commonly used in text classification; see [link to specific structure] for details. Figure 2 Here, supervised data refers to a given data point x with a corresponding label y, where the content of label y clearly defines the relationship to be established between x and y, such as y being a style feature corresponding to x. Subsequently, in this embodiment, a trained attribute classifier can be used to calculate the probability of each style category on the second and third corpora, serving as the scoring basis.

[0114] Given an input sentence x′, the attribute classifier classifies it as s. j The formula for calculating a fraction is as follows:

[0115] p(s j |x′)=softmax j (TextCNN(x′,θ)), j=1,2,

[0116] Where x′ represents the input sentence, which can be the second or third corpus data, θ represents the parameters of the attribute classifier, softmax represents the normalization exponential function, TextCNN represents the attribute classifier, s1 represents the first style feature, and s2 represents the second style feature.

[0117] It is understood that other methods can also be used to calculate the scores corresponding to style features, and this application does not limit this.

[0118] S105. If the first score is greater than the second score, then construct a first parallel corpus data pair using the second corpus data and the first corpus data.

[0119] S106. If the first score is not greater than the second score, then construct a first parallel corpus data pair using the third corpus data and the first corpus data.

[0120] As can be seen, the models trained through back-translation and adversarial training methods in this embodiment yield different style-transformed corpus data, enriching the construction methods of parallel corpus data pairs and solving the problem of scarce parallel corpus data pairs. Furthermore, style scores are calculated on the style-transformed corpus data, and selection is performed based on these scores, thus constructing high-quality first parallel corpus data pairs and improving the overall quality of parallel corpus data pairs.

[0121] In one possible implementation, the method for constructing parallel corpus data pairs provided in this application embodiment further includes:

[0122] S201. Supervised fine-tuning of the first pre-trained language model using the first parallel corpus data pair yields the second pre-trained language model; the first parallel corpus data pair includes: the first corpus data, the parallel corpus data corresponding to the first corpus data, and the parallel corpus data corresponding to the first corpus data is the second corpus data or the third corpus data.

[0123] It should be noted that the first pre-trained language model can be a BART-large model, consisting of an encoder and a decoder, which has good text generation capabilities. Fine-tuning refers to retraining the pre-trained language model based on the requirements of the task. Supervised fine-tuning involves using the correspondence between the two types of corpus data in the given first parallel corpus data to perform corresponding training.

[0124] S202. Input the first corpus data into the second pre-trained language model to obtain the first output result with the second style features;

[0125] S203. Calculate the first change magnitude value between the first output result and the parallel corpus data corresponding to the first corpus data;

[0126] While supervised training can mitigate content loss while maintaining high style classification accuracy, its effectiveness is limited. Therefore, this application introduces a style intensity reward mechanism and a content retention reward mechanism based on bilingual evaluation understudy (BLEU) scores. The BLEU algorithm essentially determines the similarity between two sentences. The model's output needs a metric to judge its quality, typically by comparing the reference text with the model's output. A higher BLEU score between the model's output and the reference text indicates better performance; BLEU scores generally range from 0 to 1.

[0127] The style intensity reward mechanism can evaluate the degree of matching between the transferred sentence and the target style, providing a clear signal to guide the model to change the style of the sentence. If s2 is the target style, the target sentence y is constructed by sampling from the distribution generated by the encoder at each time step. s The reward mechanism for its style intensity is represented as: R cls =λ cls [p(s2|y s )-p(s1|y sAt the content level, consider the target transfer text y′ obtained by the model through a greedy algorithm, and the sentence y generated by the decoder at each step. s The reward mechanism based on the referenced target text y is: R bleu =λ bleu [bleu(y′,y)-bleu(y s ,y)];λ cls , λ bleu The parameter value can be set to 1, and there are no specific restrictions. λ can be set according to actual needs. cls , λ bleu The value of .

[0128] It should be noted that model predictions have errors. The model is optimized by calculating the loss function between the model output and the label. The reward mechanism here is R. cls With R bleu Multiplying the corresponding loss functions can have a certain regularization effect. Because y s It is obtained by random sampling in each step of the generation process, therefore the corresponding loss function is: in This represents the sampling result of step i. This represents the sampling result up to step i. There are two types of loss functions, one of which is the sampling loss mentioned above. sample Another is the generation loss. Where y′ i Generate the result y′ for the model in step i. ＜i This represents the model generation result before step i. R cls With R bleu Its role is to construct the final optimization objective: loss. total =R cls loss sample +R bleu loss sample +loss gen .

[0129] The method for calculating the first variation magnitude value in this embodiment of the application may specifically be as follows: calculating the first bilingual translation quality assessment BLEU value between the first corpus data and the first output result; calculating the second BLEU value between the first corpus data and the parallel corpus data corresponding to the first corpus data; calculating the third score corresponding to the second style feature of the first output result; when the parallel corpus data corresponding to the first corpus data is the second corpus data, calculating the first variation magnitude value using the first score, the first BLEU value, the second BLEU value, and the third score; or, when the parallel corpus data corresponding to the first corpus data is the third corpus data, calculating the first variation magnitude value using the second score, the first BLEU value, the second BLEU value, and the third score.

[0130] In one possible implementation, the specific formula for calculating the first change magnitude value can be as follows:

[0131]

[0132] Where H1 represents the first change magnitude value, This represents the first BLEU value. This indicates the second BLEU value. This indicates the third score, when the parallel corpus data corresponding to the first corpus data is the third corpus data. This represents the first score, and when the parallel data corresponding to the first corpus is the third corpus, it represents... Second place score.

[0133] S204. If the first change magnitude value is greater than the preset threshold, the first output result is used to replace the parallel corpus data corresponding to the first corpus data to obtain the updated first parallel corpus data pair.

[0134] It is understandable that if the first change amplitude value is not greater than the preset threshold τ th Then the first parallel corpus data pair remains unchanged, τ th It can be set according to the actual situation; this application does not impose any restrictions on it.

[0135] S205. The first parallel corpus data pair is updated through multiple iterative calculation processes to obtain the successfully constructed first parallel corpus data pair.

[0136] The current loop calculation process in the multiple loop calculations includes:

[0137] a1. Using the updated first parallel corpus data from the previous loop calculation process, the first pre-trained language model is subjected to supervised fine-tuning to obtain the third pre-trained language model.

[0138] It should be noted that in the first loop calculation process, the first pre-trained language model is supervisedly fine-tuned using the updated first parallel corpus data obtained from S204 to obtain the third pre-trained language model, and then the subsequent steps are executed.

[0139] a2. Input the first corpus data into the third pre-trained language model to obtain the second output result with the second style features;

[0140] a3. Calculate the second change magnitude value between the second output result and the parallel corpus data corresponding to the first corpus data in the current loop calculation process;

[0141] The method for calculating the second variation amplitude value in this embodiment of the application can be as follows: calculating the third BLEU value between the first corpus data and the second output result; calculating the fourth BLEU value between the first corpus data and the parallel corpus data corresponding to the first corpus data in the current loop calculation process; calculating the fourth score corresponding to the second style feature of the second output result; calculating the fifth score corresponding to the second style feature of the parallel corpus data corresponding to the first corpus data in the current loop calculation process; and calculating the second variation amplitude value using the third BLEU value, the fourth BLEU value, the fourth score, and the fifth score. It is understood that the calculation formula for the second variation amplitude value is similar to the calculation formula for the first variation amplitude value described above, and therefore will not be repeated here.

[0142] a4. If the second change magnitude value is greater than the preset threshold, then the second output result is used to replace the parallel corpus data corresponding to the first corpus data in the current loop calculation process to obtain the updated first parallel corpus data pair in the current loop calculation process, and return to execute a1 until the difference between the transfer accuracy of the third pre-trained language model in the current loop calculation process and the transfer accuracy of the third pre-trained language model in the previous loop calculation process is not greater than the preset value, and the successfully constructed first parallel corpus data pair is obtained.

[0143] It's important to note that the model's data is divided into training, validation, and test sets. The training set is used for model training, while the validation set data is input into the model to verify its performance. Transfer accuracy is calculated by inputting the validation set data into the BART model, then feeding the BART model's output into the attribute classifier to determine its category, and finally calculating the accuracy rate at which the attribute classifier's classification matches the true category.

[0144] For example: the first parallel corpus data pair is (x, y) 0 ), using (x, y 0 Supervised fine-tuning of BART yields BART. 1 Enter x into BART 1In the middle, the first output result y is obtained. 1 ; Calculate y 1 and y 0 The first change range value between; if the first change range value is greater than the preset threshold, then use y 1 Replace y 0 The updated first parallel corpus data pair (x, y) is obtained. 1 Using (x, y) 1 Supervised fine-tuning of BART yields BART. 2 Input the first corpus data into BART 2 In the middle, the second output result y is obtained. 2 ; Calculate y 2 and y 1 The second change amplitude value between; if the second change amplitude value is greater than the preset threshold, then use y 2 Replace y 1 This yields the updated first parallel corpus data pair (x, y) in the current iterative computation process. 2 ); using (x, y 2 Supervised fine-tuning of BART yields BART. 3 Input the first corpus data into BART 3 In the middle, the second output result y is obtained. 3 ; Calculate y 3 and y 2 The second change amplitude value between; if the second change amplitude value is greater than the preset threshold, then use y 3 Replace y 2 This yields the updated first parallel corpus data pair (x, y) in the current iterative computation process. 3 ...Understandably, the subsequent calculation process is similar until the transfer accuracy improvement of the model in each training iteration is less than 0.05%, at which point data updates cease. It is understood that the above is merely illustrative and should not be construed as a limitation of this application.

[0145] As can be seen, this embodiment effectively aligns corpus data with different style features to construct a first parallel corpus, enabling the pre-trained language model to learn good feature representations through supervised training and generate higher-quality transfer text. Simultaneously, since the pre-trained language model does not require retraining from scratch and exhibits excellent performance after fine-tuning in downstream tasks, the use of a generative pre-trained language model enhances the model's style transfer capability and generalization. Furthermore, the introduction of both content-level and style-level reward mechanisms, through direct alignment, establishes a directional connection between the two corpora, avoiding the measurement of the separation between content and style, and helping to further alleviate the imbalance between content retention and style in the generated corpus. In addition, the original input data is fed into the fine-tuned model to obtain new outputs. Based on the established criteria of ensemble BLEU scores and style scores, parallel corpus data pairs are iteratively filtered and updated, and the generative language model is readjusted to continuously improve the quality of parallel corpus data generation, resulting in more fluent and compliant style transfer corpus data.

[0146] In one possible implementation, the method for constructing parallel corpus data pairs provided in this application embodiment further includes:

[0147] S301. Input the fourth corpus data with the second style features into the first style conversion model to obtain the fifth corpus data with the first style features;

[0148] S302. Input the fourth corpus data into the second style conversion model to obtain the sixth corpus data with the first style features;

[0149] It should be noted that there is no specific order between S301 and S302. S301 can be executed first and then S302, or S302 can be executed first and then S301, or S301 and S302 can be executed simultaneously. This application does not limit this.

[0150] S303. Calculate the sixth score corresponding to the first style feature of the fifth corpus data;

[0151] S304. Calculate the seventh score corresponding to the first style feature of the sixth corpus data;

[0152] It should be noted that there is no specific order for S303 and S304. S303 can be executed first and then S304, or S304 can be executed first and then S303, or S303 and S304 can be executed simultaneously. This application does not limit this.

[0153] S305. If the sixth score is greater than the seventh score, then construct a second parallel corpus data pair using the fifth and fourth corpus data.

[0154] S306. If the sixth score is not greater than the seventh score, then construct a second parallel corpus data pair using the sixth corpus data and the fourth corpus data.

[0155] S307. Supervised fine-tuning of the fourth pre-trained language model using the second parallel corpus data to obtain the fifth pre-trained language model; the fourth parallel corpus data includes: the fourth corpus data, the parallel corpus data corresponding to the fourth corpus data, and the parallel corpus data corresponding to the fourth corpus data is the fifth corpus data or the sixth corpus data.

[0156] It should be noted that the fourth pre-trained language model can be a BART-large model, consisting of an encoder and a decoder, which has good text generation capabilities.

[0157] S308. Input the fourth corpus data into the fifth pre-trained language model to obtain the third output result with the first style features;

[0158] S309. Calculate the third variation value between the third output result and the parallel corpus data corresponding to the fourth corpus data;

[0159] The method for calculating the third variation amplitude value in this embodiment can be as follows: calculating the fifth BLEU value between the fourth corpus data and the third output result; calculating the sixth BLEU value between the fourth corpus data and the parallel corpus data corresponding to the fourth corpus data; calculating the eighth score corresponding to the first style feature of the third output result; when the parallel corpus data corresponding to the fourth corpus data is the fifth corpus data, the third variation amplitude value is calculated using the sixth score, the fifth BLEU value, the sixth BLEU value, and the eighth score; or, when the parallel corpus data corresponding to the fourth corpus data is the sixth corpus data, the third variation amplitude value is calculated using the seventh score, the fifth BLEU value, the sixth BLEU value, and the eighth score. It is understood that the calculation formula for the third variation amplitude value is similar to the calculation formulas for the first and second variation amplitude values ​​described above, and therefore will not be repeated here.

[0160] S310. If the third change value is greater than the preset threshold, the third output result is used to replace the parallel corpus data corresponding to the fourth corpus data to obtain the updated second parallel corpus data pair.

[0161] S311. The updated fourth parallel corpus data pair is updated through multiple iterative calculation processes to obtain the successfully constructed second parallel corpus data pair.

[0162] The current loop calculation process in the multiple loop calculations includes:

[0163] b1. Using the updated fourth parallel corpus data from the previous loop calculation process, the fourth pre-trained language model is fine-tuned in a supervised manner to obtain the sixth pre-trained language model.

[0164] b2. Input the fourth corpus data into the sixth pre-trained language model to obtain the fourth output result with the first style features;

[0165] b3. Calculate the fourth variation value between the fourth output result and the parallel corpus data corresponding to the fourth corpus data in the current loop calculation process;

[0166] The specific method for calculating the fourth variation amplitude value in this embodiment of the application can be as follows: calculating the seventh BLEU value between the fourth corpus data and the fourth output result; calculating the eighth BLEU value between the fourth corpus data and the parallel corpus data corresponding to the fourth corpus data in the current loop calculation process; calculating the ninth score corresponding to the first style feature of the fourth output result; calculating the tenth score corresponding to the first style feature of the parallel corpus data corresponding to the fourth corpus data in the current loop calculation process; and using the seventh BLEU value, the eighth BLEU value, the ninth score, and the tenth score, the fourth variation amplitude value is calculated. It is understood that the calculation formula for the fourth variation amplitude value is similar to the calculation formulas for the first, second, and third variation amplitude values ​​described above, and therefore will not be repeated here.

[0167] b4. If the fourth change magnitude value is greater than the preset threshold, then use the fourth output result to replace the parallel corpus data corresponding to the fourth corpus data in the current loop calculation process to obtain the updated second parallel corpus data pair in the current loop calculation process, and return to execute b1 until the difference between the transfer accuracy of the sixth pre-trained language model in the current loop calculation process and the transfer accuracy of the sixth pre-trained language model in the previous loop calculation process is not greater than the preset value, and the successfully constructed second parallel corpus data pair is obtained.

[0168] As can be seen, in this embodiment of the application, the second parallel corpus data pair is constructed using the fourth corpus data with the second style features, which further enriches the number of parallel corpus data pairs and solves the problem of the scarcity of parallel corpus data pairs.

[0169] See Figure 3 This application discloses a flowchart of another method for constructing parallel corpus data pairs. Among them, Figure 3 In this context, model M1 represents the first style transfer model, model M2 represents the second style transfer model, and the BART model represents the pre-trained language model; text x represents the first corpus data with the first style features, and y represents the second style transfer model. M1 This represents second corpus data with second style characteristics, y M2The third corpus data with second style features, the new parallel corpus (x, y*) represents the first parallel corpus data pair or the updated first parallel corpus data pair, y bart Indicates the first or second output result, (x, y) final () represents the first parallel corpus data pair after construction; text y represents the fourth corpus data with second style features, x M1 This represents the fifth corpus data exhibiting the first style characteristics, x M2 The sixth corpus data represents the data with the first style characteristics. The new parallel corpus (x*, y) represents the second parallel corpus data pair or the updated second parallel corpus data pair, x bart This indicates the third or fourth output result, (y, x) final ) indicates the completed second parallel corpus data pair.

[0170] The method for constructing parallel corpus data pairs in this application embodiment can be divided into two parts: the first part is to construct the initial parallel corpus data pairs, and the second part is to update and optimize the parallel corpus data pairs.

[0171] (a) Constructing initial parallel corpus data pairs.

[0172] The first corpus dataset with first style features is X = {x1, x2, ..., x...} n} and the fourth corpus dataset Y = {y1, y2, ..., y} with second style features n The corresponding styles are s1 and s2, respectively. X and Y are input into the corresponding parts of model M1 to generate a second parallel corpus dataset with the second style features. Compared with the fifth corpus dataset with first style features The corresponding styles are s2 and s1, respectively. Similarly, X and Y are input into the corresponding parts of model M2 to generate a third corpus dataset with the second style features. Compared with the sixth parallel corpus dataset with first style features That is, the corresponding styles are s2 and s1, respectively. At this point, the initial set of parallel corpus data pairs has been generated: X and... or Y and or

[0173] Given input X, by... and For each piece of corpus data, a style score is calculated to filter the data; that is, for any i = 1, 2, ..., n, if... Then choose Otherwise choose The selection result is denoted as At this point, the parallel corpus dataset corresponding to X is constructed as follows: The first parallel corpus data pair set is {X,Y} 0 Similarly, construct the parallel corpus dataset corresponding to Y. The second parallel corpus data pair set is {Y,X} 0}

[0174] (ii) Update and optimize the parallel corpus data pairs.

[0175] Using the first parallel corpus data to analyze the set Data set with the second parallel corpus Supervised fine-tuning was performed on two models, BART1 and BART2, respectively. Then, inputting X and Y into each model, the output Y of the model was obtained. BART1 With X BART2 .

[0176] For input X of style s1, compare Y 0 With Y BART1 For each corresponding result, the weighted sum of the changes in content and style is multiplied by a set positive threshold τ. th1 Comparison, that is, for any i = 1, 2, ..., n, if

[0177]

[0178] Then replace Otherwise, it remains unchanged as new corresponding corpus data.

[0179] For input Y with style s2, compare X 0 With X BART2 For each corresponding result, the weighted sum of the changes in content and style is multiplied by a set positive threshold τ. th2 Comparison, that is, for any i = 1, 2, ..., n, if

[0180]

[0181] Then replace Otherwise, it remains unchanged as new corresponding corpus data.

[0182] In this way, a new first parallel corpus data pair set {X,Y} can be obtained through updating. 1} and the new second parallel corpus data pair {Y,X 1 By fine-tuning the original two models, BART1 and BART2, a new corresponding output Y can be obtained. BART1 With X BART2The model is then compared and updated again with the most recently updated parallel corpus data. This process is repeated multiple times until the transfer accuracy improvement of each training iteration falls short of a preset value, at which point data updates cease. This preset value can be 0.05%, 0.06%, etc., and this application does not impose any restrictions on it.

[0183] The parallel corpus data {X,Y} obtained after the final training is completed will be updated. final} and {Y,X final The reward mechanism is then reused for fine-tuning the training of the two initial BART models to obtain the final output.

[0184] As can be seen, this embodiment of the application obtains pre-prepared parallel corpus data by leveraging existing implicit style transfer methods. Although unsupervised training is not as effective as supervised learning, these model methods continuously improve the output quality of the model by employing special techniques such as back-translation and adversarial training. Moreover, two data filtering strategies are proposed: first, in the process of constructing the initial parallel corpus data, a style scoring strategy is used to compare the scores of target style sentences output by different models; second, in the process of training and updating the parallel corpus data using the pre-trained model, this embodiment of the application introduces a discrimination criterion that integrates both BLEU value and style score. This further filters the generated data, continuously updates and improves the quality of the parallel corpus, and generates more fluent and compliant parallel corpus data pairs.

[0185] See Figure 4 This is a schematic diagram of the framework of a first style transfer model disclosed in an embodiment of this application. The first style transfer model includes a first translation model, a second translation model, a first style decoder, a second style decoder, and a first attribute classifier. Both the encoder and decoder of the two translation models can adopt bidirectional long short-term memory (LSTM) networks. Figure 4 In this diagram, x represents the first training text, y represents the second training text, x' represents the first translated text, y' represents the second translated text, z1 represents the first latent vector, z2 represents the second latent vector, and x... pos Indicates the first translation of the text, y neg This indicates the second translation. It should be noted that... Figure 4 The first translation model, the second translation model, and the attribute classifier are pre-trained, while the first style decoder and the second style decoder need to be trained.

[0186] In one possible implementation, the method for constructing parallel corpus data pairs provided in this application embodiment further includes:

[0187] The first-style decoder is trained using the following process:

[0188] S401. Input the first training text with the first style features into the first translation model to obtain the first translated text with the first style features;

[0189] S402. Input the first translated text into the encoder of the second translation model to obtain the first latent vector without style features;

[0190] The first translation model can be an English-to-Chinese model, and the second translation model can be a Chinese-to-English model. This application does not impose any restrictions on this and the model can be selected and set according to the actual situation.

[0191] S403. Input the first latent vector into the first style decoder to obtain the first back-translated text with the first style features;

[0192] S404. Input the first back-translated text into the first attribute classifier to obtain the first classification result;

[0193] The attribute classifier in step S104 above can be the first attribute classifier here, so the structure and acquisition method of the first attribute classifier will not be described again.

[0194] S405. Train the first-style decoder using the first classification result;

[0195] The second-style decoder is trained using the following process:

[0196] S406. Input the second training text with the second style features into the first translation model to obtain the second translated text with the second style features;

[0197] S407. Input the second translated text into the encoder of the second translation model to obtain the second latent vector without style features;

[0198] S408. Input the second latent vector into the second style decoder to obtain the second back-translated text with the second style features;

[0199] S409. Input the second translated text into the first attribute classifier to obtain the second classification result;

[0200] S410. Train the second style decoder using the second classification result.

[0201] It should be noted that the training process of the first style decoder and the training process of the second style decoder are not sequential. That is, S401 and S406 are not sequential. S401 can be executed first and then S406, or S406 can be executed first and then S401, or S401 and S406 can be executed simultaneously. This application does not limit this.

[0202] For example: The two styles to be transferred are positive and negative, and the training corpus contains both positive and negative training texts. The two translation models are English-to-Chinese and Chinese-to-English, respectively. The training steps are as follows: For positive training texts: First, the English positive training text is input into the English-to-Chinese model to obtain positive Chinese training text. Then, the positive Chinese training text is input into the Chinese-to-English encoder to obtain a style-neutral first latent vector. The first latent vector is then input into the positive style decoder to be trained, and its output is input into the attribute classifier for judgment, guiding the positive decoder training. For negative training texts: First, the English negative training text is input into the English-to-Chinese model to obtain negative Chinese training text. Then, the negative Chinese training text is input into the Chinese-to-English encoder to obtain a style-neutral second latent vector. The second latent vector is then input into the negative style decoder to be trained, and its output is input into the attribute classifier for judgment, guiding the negative decoder training. It is understood that the above is merely an illustrative example and should not be construed as a limitation of this application.

[0203] Understandably, through the above training process, a trained first-style decoder and a trained second-style decoder will eventually be obtained. Next, the first corpus data with the first-style features is input into the first translation model, then the output of the first translation model is input into the encoder of the second translation model, and then the output of the encoder of the second translation model is input into the trained second-style decoder, resulting in second corpus data with the second-style features. This second corpus data is then input into the first attribute classifier to obtain the corresponding classification result. Finally, the score corresponding to the second-style features of the second corpus data is calculated based on this classification result. Similarly, the fourth corpus data with the second-style features is input into the first translation model, then the output of the first translation model is input into the encoder of the second translation model, and then the output of the encoder of the second translation model is input into the trained first-style decoder, resulting in fifth corpus data with the first-style features. This fifth corpus data is then input into the first attribute classifier to obtain the corresponding classification result. Finally, the score corresponding to the first-style features of the fifth corpus data is calculated based on this classification result.

[0204] As can be seen, the back-translation training method is adopted in the embodiments of this application, and the attribute classifier guides the training of the decoder. This enables the trained first decoder to decode the latent vectors without style features into corpora with first style features, and the trained second decoder to decode the latent vectors without style features into corpora with second style features. This allows the first style conversion model to achieve style conversion of the corpus data, improves the output quality of the first style conversion model, and makes the style-converted corpus data obtained by using the first style conversion model of the subsequent higher quality, thereby improving the construction quality of parallel corpus data pairs.

[0205] See Figure 5 This is a schematic diagram of the framework of a second style transfer model disclosed in an embodiment of this application; the second style transfer model includes a first style generator, a second style generator, and a second attribute classifier. Both generators are composed of autoencoders, and both the encoder and decoder parts can adopt a bidirectional LSTM structure. Figure 5 In the text, text x represents the first training text, text y represents the second training text, and style S x Indicates the primary stylistic characteristic, style S y Let g represent the second style feature, f represent the first style generator, and L represent the second style generator. g (c) Indicates the first loss value, L g (t) Indicates the second loss value, L f (t) Indicates the third loss value, L f (c) This represents the fourth loss value.

[0206] In one possible implementation, the method for constructing parallel corpus data pairs provided in this application embodiment further includes:

[0207] The first style generator and the second style generator are trained using the following training process:

[0208] S501. Noise is added to the first training text with the first style feature to obtain the first noisy text.

[0209] S502. Input the first noisy text into the first style generator for data reconstruction to obtain the first reconstructed text with the first style features;

[0210] S503. Calculate the cross-entropy loss value between the first reconstructed text and the first training text to obtain the first loss value;

[0211] S504. Input the first reconstructed text into the second attribute classifier to obtain the third classification result;

[0212] It should be noted that since both the first and second style transfer models require additional supervised parallel corpus data to train the attribute classifier, when training two models based on the same corpus, the second model can reuse the attribute classifier already trained by the first model without needing to train it again.

[0213] S505. Input the first training text into the second style generator to reconstruct the data and obtain the second reconstructed text with the second style features.

[0214] S506. Noise is added to the second reconstructed text to obtain the second noisy text;

[0215] S507. Input the second noisy text into the first style generator for data reconstruction to obtain the third reconstructed text with the first style features;

[0216] S508. Calculate the cross-entropy loss value between the third reconstructed text and the first training text to obtain the second loss value;

[0217] S509. Input the third reconstructed text into the second attribute classifier to obtain the fourth classification result;

[0218] S510. Noise is added to the second training text with the second style feature to obtain the third noisy text.

[0219] S511. Input the third noisy text into the second style generator for data reconstruction to obtain the fourth reconstructed text with the second style features;

[0220] S512. Calculate the cross-entropy loss value between the fourth reconstructed text and the second training text to obtain the third loss value;

[0221] S513. Input the fourth reconstructed text into the second attribute classifier to obtain the fifth classification result;

[0222] S514. Input the second training text into the first style generator for data reconstruction to obtain the fifth reconstructed text with the first style features;

[0223] It should be noted that there is no specific order for S501, S505, S510, and S514. S501 can be executed first, followed by S505, S510, and S514, or S505 can be executed first, followed by S501, S510, and S514, and so on. S501, S505, S510, and S514 can also be executed simultaneously. This application does not impose any restrictions on this.

[0224] S515. Noise is added to the fifth reconstructed text to obtain the fourth noisy text;

[0225] S516. Input the fourth noisy text into the second style generator for data reconstruction to obtain the sixth reconstructed text with the second style features.

[0226] S517. Calculate the cross-entropy loss value between the sixth reconstructed text and the second training text to obtain the fourth loss value;

[0227] S518. Input the sixth reconstructed text into the second attribute classifier to obtain the sixth classification result;

[0228] S519. Train the first style generator and the second style generator using the first loss value, the second loss value, the third loss value, the fourth loss value, the third classification result, the fourth classification result, the fifth classification result, and the sixth classification result.

[0229] Understandably, through the above training process, a first style generator and a second style generator will eventually be obtained. Then, the first corpus data with the first style features is input into the trained second style generator to obtain the third corpus data with the second style features. Finally, the fourth corpus data with the second style features is input into the trained first style generator to obtain the sixth corpus data with the first style features.

[0230] As can be seen, in this embodiment, the generalization ability of the model is improved by adding noise to the original training data, training the generator, and reconstructing the original training data; the transferability and robustness of the model are improved by sequentially inputting the original data into two style generators and introducing noise in the intermediate input process, thereby improving the transferability and robustness of the model; thus, the quality of the style-converted corpus data obtained by using the second style conversion model is higher, thereby improving the construction quality of parallel corpus data pairs.

[0231] See Figure 6 The present application discloses a schematic diagram of a device for constructing parallel corpus data pairs, the device comprising:

[0232] Style conversion unit 601 is used to input first corpus data with first style features into first style conversion model to obtain second corpus data with second style features; the first style conversion model is obtained based on back-translation training;

[0233] The style transfer unit 601 is also used to input the first corpus data into the second style transfer model to obtain the third corpus data with the second style features; the second style transfer model is obtained based on adversarial training.

[0234] Style score calculation unit 602 is used to calculate the first score corresponding to the second style feature of the second corpus data;

[0235] The style score calculation unit 602 is also used to calculate the second score corresponding to the second style feature of the third corpus data;

[0236] Construction unit 603 is used to construct a first parallel corpus data pair using the second corpus data and the first corpus data if the first score is greater than the second score;

[0237] The construction unit 603 is also used to construct a first parallel corpus data pair using the third corpus data and the first corpus data if the first score is not greater than the second score.

[0238] As can be seen, the models trained through back-translation and adversarial training methods in this embodiment yield different style-transformed corpus data, enriching the construction methods of parallel corpus data pairs and solving the problem of scarce parallel corpus data pairs. Furthermore, style scores are calculated on the style-transformed corpus data, and selection is performed based on these scores, thus constructing high-quality first parallel corpus data pairs and improving the overall quality of parallel corpus data pairs.

[0239] In one possible implementation, the apparatus for constructing parallel corpus data pairs provided in this application embodiment further includes:

[0240] The fine-tuning unit is used to perform supervised fine-tuning of the first pre-trained language model using the first parallel corpus data to obtain the second pre-trained language model; the first parallel corpus data includes: first corpus data, parallel corpus data corresponding to the first corpus data, and the parallel corpus data corresponding to the first corpus data is the second corpus data or the third corpus data.

[0241] The result output unit is used to input the first corpus data into the second pre-trained language model to obtain a first output result with second style features;

[0242] The variation amplitude calculation unit is used to calculate the first variation amplitude value between the first output result and the parallel corpus data corresponding to the first corpus data;

[0243] The replacement unit is used to replace the parallel corpus data corresponding to the first corpus data with the first output result if the first change amplitude value is greater than a preset threshold, so as to obtain the updated first parallel corpus data pair.

[0244] In one possible implementation, the variation magnitude calculation unit in the parallel corpus data pair construction apparatus provided in this application embodiment includes:

[0245] The BLEU value calculation unit is used to calculate the first bilingual translation quality assessment BLEU value between the first corpus data and the first output result.

[0246] The BLEU value calculation unit is also used to calculate the second BLEU value between the first corpus data and the parallel corpus data corresponding to the first corpus data;

[0247] The style score calculation subunit is used to calculate the third score corresponding to the second style feature of the first output result;

[0248] The variation magnitude calculation subunit is used to calculate the first variation magnitude value using the first score, the first BLEU value, the second BLEU value, and the third score when the parallel corpus data corresponding to the first corpus data is the second corpus data; or, when the parallel corpus data corresponding to the first corpus data is the third corpus data, to calculate the first variation magnitude value using the second score, the first BLEU value, the second BLEU value, and the third score.

[0249] In one possible implementation, the fine-tuning unit in the parallel corpus data pair construction device provided in this application embodiment is further used to perform supervised fine-tuning of the first pre-trained language model using the updated first parallel corpus data pair in the previous loop calculation process to obtain the third pre-trained language model.

[0250] The result output unit is also used to input the first corpus data into the third pre-trained language model to obtain a second output result with second style features;

[0251] The variation amplitude calculation unit is used to calculate the second variation amplitude value between the second output result and the parallel corpus data corresponding to the first corpus data in the current loop calculation process;

[0252] The replacement unit is further configured to, if the second change magnitude value is greater than a preset threshold, replace the parallel corpus data corresponding to the first corpus data in the current iterative calculation process with the second output result, to obtain the updated first parallel corpus data pair in the current iterative calculation process, and return to execute the following steps: use the updated first parallel corpus data pair in the previous iterative calculation process to perform supervised fine-tuning on the first pre-trained language model to obtain the third pre-trained language model, until the difference between the transfer accuracy of the third pre-trained language model in the current iterative calculation process and the transfer accuracy of the third pre-trained language model in the previous iterative calculation process is not greater than a preset value, and obtain the successfully constructed first parallel corpus data pair.

[0253] In one possible implementation, the BLEU value calculation unit in the parallel corpus data pair construction device provided in this application embodiment is further used to calculate a third BLEU value between the first corpus data and the second output result;

[0254] The BLEU value calculation unit is also used to calculate the fourth BLEU value between the first corpus data and the parallel corpus data corresponding to the first corpus data in the current loop calculation process;

[0255] The style score calculation subunit is also used to calculate the fourth score corresponding to the second style feature of the second output result;

[0256] The style score calculation subunit is also used to calculate the fifth score corresponding to the second style feature of the parallel corpus data corresponding to the first corpus data in the current loop calculation process;

[0257] The variation range calculation subunit is also used to calculate the second variation range value using the third BLEU value, the fourth BLEU value, the fourth score, and the fifth score.

[0258] In one possible implementation, the style conversion unit 601 in the parallel corpus data pair construction device provided in this application embodiment is further used to input the fourth corpus data with the second style feature into the first style conversion model to obtain the fifth corpus data with the first style feature.

[0259] The style conversion unit 601 is also used to input the fourth corpus data into the second style conversion model to obtain the sixth corpus data with the first style features;

[0260] The style score calculation unit 602 is also used to calculate the sixth score corresponding to the first style feature of the fifth corpus data;

[0261] The style score calculation unit 602 is also used to calculate the seventh score corresponding to the first style feature of the sixth corpus data;

[0262] The construction unit 603 is also used to construct a second parallel corpus data pair using the fifth corpus data and the fourth corpus data if the sixth score is greater than the seventh score;

[0263] The construction unit 603 is also used to construct a second parallel corpus data pair using the sixth corpus data and the fourth corpus data if the sixth score is not greater than the seventh score.

[0264] In one possible implementation, the parallel corpus data pair construction apparatus provided in this application embodiment includes a first style transfer model comprising a first translation model, a second translation model, a first style decoder, a second style decoder, and a first attribute classifier. The apparatus further includes a first training unit, which comprises:

[0265] A translation model is used to input a first training text with first style features into a first translation model to obtain a first translated text with first style features;

[0266] The encoding unit is used to input the first translated text into the encoder of the second translation model to obtain the first latent vector without style features;

[0267] The decoding unit is used to input the first latent vector into the first style decoder to obtain the first back-translated text with the first style features;

[0268] The first classification unit is used to input the first back-translated text into the first attribute classifier to obtain the first classification result;

[0269] The training subunit is used to train the first-style decoder using the first classification result;

[0270] The translation model is also used to input a second training text with second style features into the first translation model to obtain a second translated text with second style features;

[0271] The encoding unit is also used to input the second translated text into the encoder of the second translation model to obtain the second latent vector without style features;

[0272] The decoding unit is also used to input the second latent vector into the second style decoder to obtain the second back-translated text with the second style features;

[0273] The first classification unit is also used to input the second back-translated text into the first attribute classifier to obtain the second classification result;

[0274] The training subunit is also used to train the second-style decoder using the second classification results.

[0275] In one possible implementation, the parallel corpus data pair construction apparatus provided in this application embodiment includes a second style transfer model comprising a first style generator, a second style generator, and a second attribute classifier. The apparatus further includes a second training unit, which comprises:

[0276] A noise-adding unit is used to add noise to the first training text with the first style features to obtain the first noisy text.

[0277] The reconstruction unit is used to input the first noisy text into the first style generator for data reconstruction to obtain the first reconstructed text with the first style features.

[0278] The loss calculation unit is used to calculate the cross-entropy loss value between the first reconstructed text and the first training text to obtain the first loss value.

[0279] The second classification unit is used to input the first reconstructed text into the second attribute classifier to obtain the third classification result;

[0280] The reconstruction unit is also used to input the first training text into the second style generator for data reconstruction to obtain a second reconstructed text with second style features;

[0281] The noise-adding unit is also used to add noise to the second reconstructed text to obtain the second noisy text;

[0282] The reconstruction unit is also used to input the second noisy text into the first style generator for data reconstruction to obtain a third reconstructed text with the first style features;

[0283] The loss calculation unit is also used to calculate the cross-entropy loss value between the third reconstructed text and the first training text to obtain the second loss value;

[0284] The second classification unit is also used to input the third reconstructed text into the second attribute classifier to obtain the fourth classification result;

[0285] The noise-adding unit is also used to add noise to the second training text with the second style features to obtain the third noisy text;

[0286] The reconstruction unit is also used to input the third noisy text into the second style generator for data reconstruction, so as to obtain a fourth reconstructed text with the second style features;

[0287] The loss calculation unit is also used to calculate the cross-entropy loss value between the fourth reconstructed text and the second training text to obtain the third loss value;

[0288] The second classification unit is also used to input the fourth reconstructed text into the second attribute classifier to obtain the fifth classification result;

[0289] The reconstruction unit is also used to input the second training text into the first style generator for data reconstruction to obtain a fifth reconstructed text with the first style features;

[0290] The noise-adding unit is also used to add noise to the fifth reconstructed text to obtain the fourth noisy text;

[0291] The reconstruction unit is also used to input the fourth noisy text into the second style generator for data reconstruction, so as to obtain the sixth reconstructed text with the second style features;

[0292] The loss value calculation unit is also used to calculate the cross-entropy loss value between the sixth reconstructed text and the second training text to obtain the fourth loss value;

[0293] The second classification unit is also used to input the sixth reconstructed text into the second attribute classifier to obtain the sixth classification result;

[0294] The training subunit is used to train the first style generator and the second style generator using the first loss value, the second loss value, the third loss value, the fourth loss value, the third classification result, the fourth classification result, the fifth classification result, and the sixth classification result.

[0295] Furthermore, embodiments of this application also provide an apparatus for constructing parallel corpus data pairs, including:

[0296] Memory, used to store instructions;

[0297] A processor for executing instructions in memory to perform any of the implementations of the method for constructing the parallel corpus data pairs described above.

[0298] Furthermore, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform any of the above-described implementations of the method for constructing parallel corpus data pairs.

[0299] Furthermore, this application also provides a computer program product that, when run on a terminal device, causes the terminal device to execute any of the above-described methods for constructing parallel corpus data pairs.

[0300] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0301] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0302] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0303] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0304] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing parallel corpus data pairs, characterized in that, The method includes: The first corpus data with the first style features is input into the first style conversion model to obtain the second corpus data with the second style features; the first style conversion model is obtained based on back-translation training; The first corpus data is input into the second style transfer model to obtain a third corpus data with the second style features; the second style transfer model is obtained based on adversarial training, and the first corpus data, the second corpus data, and the third corpus data are all text; Calculate the first score corresponding to the second style feature of the second corpus data; Calculate the second score corresponding to the second style feature of the third corpus data; If the first score is greater than the second score, then a first parallel corpus data pair is constructed using the second corpus data and the first corpus data; If the first score is not greater than the second score, then the third corpus data and the first corpus data are used to construct a first parallel corpus data pair. The first parallel corpus data pair is used to perform supervised training on the pre-trained language model. The pre-trained language model after supervised training is used to perform style transfer on the text.

2. The method according to claim 1, characterized in that, The method further includes: The first pre-trained language model is subjected to supervised fine-tuning using the first parallel corpus data to obtain the second pre-trained language model; the first parallel corpus data includes: the first corpus data and the parallel corpus data corresponding to the first corpus data, wherein the parallel corpus data corresponding to the first corpus data is the second corpus data or the third corpus data. The first corpus data is input into the second pre-trained language model to obtain a first output result with the second style features; Calculate the first variation value between the first output result and the parallel corpus data corresponding to the first corpus data; If the first change magnitude value is greater than a preset threshold, then the first output result is used to replace the parallel corpus data corresponding to the first corpus data to obtain the updated first parallel corpus data pair.

3. The method according to claim 2, characterized in that, The calculation of the first change magnitude value between the first output result and the parallel corpus data corresponding to the first corpus data includes: Calculate the first bilingual translation quality assessment BLEU value between the first corpus data and the first output result; Calculate the second BLEU value between the first corpus data and the parallel corpus data corresponding to the first corpus data; Calculate the third score corresponding to the second style feature of the first output result; When the parallel corpus data corresponding to the first corpus data is the second corpus data, the first change magnitude value is calculated using the first score, the first BLEU value, the second BLEU value, and the third score; or, When the parallel corpus data corresponding to the first corpus data is the third corpus data, the first change magnitude value is calculated using the second score, the first BLEU value, the second BLEU value, and the third score.

4. The method according to claim 2, characterized in that, The method further includes: The first parallel corpus data pair is updated by multiple iterative calculations to obtain a successfully constructed first parallel corpus data pair. The current iterative calculation process in the multiple iterative calculations includes: The first pre-trained language model is fine-tuned in a supervised manner using the updated first parallel corpus data from the previous loop calculation process to obtain the third pre-trained language model. The first corpus data is input into the third pre-trained language model to obtain a second output result with second style features; Calculate the second change magnitude value between the second output result and the parallel corpus data corresponding to the first corpus data in the current loop calculation process; If the second change magnitude value is greater than the preset threshold, then the second output result is used to replace the parallel corpus data corresponding to the first corpus data in the current iterative calculation process to obtain the updated first parallel corpus data pair in the current iterative calculation process, and then the following steps are performed: the first pre-trained language model is supervisedly fine-tuned using the updated first parallel corpus data pair in the previous iterative calculation process to obtain the third pre-trained language model, until the difference between the transfer accuracy of the third pre-trained language model in the current iterative calculation process and the transfer accuracy of the third pre-trained language model in the previous iterative calculation process is not greater than the preset value, and the successfully constructed first parallel corpus data pair is obtained.

5. The method according to claim 4, characterized in that, The calculation of the second change magnitude value between the second output result and the parallel corpus data corresponding to the first corpus data in the current loop calculation process includes: Calculate the third BLEU value between the first corpus data and the second output result; Calculate the fourth BLEU value between the first corpus data and the parallel corpus data corresponding to the first corpus data in the current loop calculation process; Calculate the fourth score corresponding to the second style feature of the second output result; Calculate the fifth score corresponding to the second style feature of the parallel corpus data corresponding to the first corpus data in the current loop calculation process; The second change magnitude value is calculated using the third BLEU value, the fourth BLEU value, the fourth score, and the fifth score.

6. The method according to claim 1, characterized in that, The method further includes: The fourth corpus data with the second style features is input into the first style conversion model to obtain the fifth corpus data with the first style features; The fourth corpus data is input into the second style conversion model to obtain the sixth corpus data with the first style features; Calculate the sixth score corresponding to the first style feature of the fifth corpus data; Calculate the seventh score corresponding to the first style feature of the sixth corpus data; If the sixth score is greater than the seventh score, then a second parallel corpus data pair is constructed using the fifth corpus data and the fourth corpus data; If the sixth score is not greater than the seventh score, then a second parallel corpus data pair is constructed using the sixth corpus data and the fourth corpus data.

7. The method according to claim 1, characterized in that, The first style transfer model includes a first translation model, a second translation model, a first style decoder, a second style decoder, and a first attribute classifier; the method further includes: The first style decoder is trained using the following training process: The first training text with the first style features is input into the first translation model to obtain the first translated text with the first style features; The first translated text is input into the encoder of the second translation model to obtain the first latent vector without style features; The first latent vector is input into the first style decoder to obtain the first back-translated text with the first style features; The first back-translated text is input into the first attribute classifier to obtain the first classification result; The first style decoder is trained using the first classification result; The second style decoder is trained using the following process: The second training text with the second style features is input into the first translation model to obtain the second translated text with the second style features; The second translated text is input into the encoder of the second translation model to obtain the second latent vector without style features; The second latent vector is input into the second style decoder to obtain the second back-translated text with the second style features; The second translated text is input into the first attribute classifier to obtain the second classification result; The second style decoder is trained using the second classification result.

8. The method according to claim 1, characterized in that, The second style transfer model includes a first style generator, a second style generator, and a second attribute classifier. The method further includes: The first style generator and the second style generator are trained using the following training process: The first training text with the first style feature is subjected to noise processing to obtain the first noisy text; The first noisy text is input into the first style generator for data reconstruction to obtain a first reconstructed text with the first style features; Calculate the cross-entropy loss value between the first reconstructed text and the first training text to obtain the first loss value; The first reconstructed text is input into the second attribute classifier to obtain the third classification result; The first training text is input into the second style generator for data reconstruction to obtain a second reconstructed text with the second style features; The second reconstructed text is then subjected to noise processing to obtain the second noisy text; The second noisy text is input into the first style generator for data reconstruction to obtain a third reconstructed text with the first style features; Calculate the cross-entropy loss value between the third reconstructed text and the first training text to obtain the second loss value; The third reconstructed text is input into the second attribute classifier to obtain the fourth classification result; The second training text with the second style feature is subjected to noise processing to obtain the third noisy text; The third noisy text is input into the second style generator for data reconstruction to obtain a fourth reconstructed text with the second style features; Calculate the cross-entropy loss value between the fourth reconstructed text and the second training text to obtain the third loss value; The fourth reconstructed text is input into the second attribute classifier to obtain the fifth classification result; The second training text is input into the first style generator for data reconstruction to obtain a fifth reconstructed text with the first style features; The fifth reconstructed text is then subjected to noise processing to obtain the fourth noisy text; The fourth noisy text is input into the second style generator for data reconstruction to obtain a sixth reconstructed text with the second style features; Calculate the cross-entropy loss value between the sixth reconstructed text and the second training text to obtain the fourth loss value; The sixth reconstructed text is input into the second attribute classifier to obtain the sixth classification result; The first style generator and the second style generator are trained using the first loss value, the second loss value, the third loss value, the fourth loss value, the third classification result, the fourth classification result, the fifth classification result, and the sixth classification result.

9. A device for constructing parallel corpus data pairs, characterized in that, The device includes: A style conversion unit is used to input first corpus data with first style features into a first style conversion model to obtain second corpus data with second style features; the first style conversion model is obtained based on back-translation training; The style conversion unit is further configured to input the first corpus data into the second style conversion model to obtain third corpus data with the second style features; the second style conversion model is obtained based on adversarial training, and the first corpus data, the second corpus data, and the third corpus data are all text; A style score calculation unit is used to calculate the first score corresponding to the second style feature of the second corpus data; The style score calculation unit is also used to calculate the second score corresponding to the second style feature of the third corpus data; A construction unit is configured to construct a first parallel corpus data pair using the second corpus data and the first corpus data if the first score is greater than the second score; The construction unit is further configured to construct a first parallel corpus data pair using the third corpus data and the first corpus data if the first score is not greater than the second score. The first parallel corpus data pair is used to perform supervised training on the pre-trained language model, and the pre-trained language model after supervised training is used to perform style transfer on the text.

10. An apparatus for constructing parallel corpus data pairs, characterized in that, include: Memory, used to store instructions; A processor for executing the instructions in the memory to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method described in any one of claims 1 to 8.

Images