Language Model Training Method, Copywriting Generation Method, and Related Devices

Abstract

Embodiments of the present disclosure relate to the technical field of text processing. More specifically, embodiments of the present disclosure relate to a language model training method, a copywriting generation method, and related devices. The language model training method includes: obtaining a sequence of text elements that make up a training text, where the sequence of text elements is divided into a source segment sequence and a target segment sequence spliced together, the sequence of text elements starts with a sequence start marker, and both the source segment sequence and the target segment sequence end with a sequence end marker; adding a mask to the sequence of text elements; inputting the masked sequence of text elements into a transformer network of the language model, and encoding each text element in the sequence of text elements into a vector with context information through the transformer network; obtaining a predicted value of the mask using the representation vector, and training the transformer network according to the cross-loss function between the predicted value and the true value until it converges to obtain a language model. The technical solution of the present disclosure can enhance the scalability of the language model.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of text processing. More specifically, embodiments of the present disclosure relate to a language model training method, a copywriting generation method, and related devices. Background Art

[0002] This section aims to provide background or context for the embodiments of the present disclosure recited in the claims. The description herein is not admitted to be prior art merely because it is included in this section.

[0003] Currently, language models based on machine learning technology have deep applications in the technical field of text processing, and the accuracy of text processing can be improved by designing language models with complex network structures. However, this reduces the scalability of language models in text processing scenarios. Summary of the Invention

[0004] Currently, the following general solutions are generally adopted to implement text processing tasks:

[0005] Solution 1: Use a copywriting generation model based on Long Short-Term Memory (LSTM), such as the Embedding from Language Models (ELMo). As a typical autoregressive pre-training model, ELMo includes two independent unidirectional LSTMs to implement autoregressive pre-training of a unidirectional language model.

[0006] Solution 2: Use the Generative Pre-Training (GPT) model based on OpenAI. The GPT model is a classic model in the field of text generation, and uses a Transformer decoder to predict text sequences word by word from left to right. However, in Solutions 1 and 2, the language model network structures used for text processing tasks are complex, but both are unidirectional language models, and the scalability is insufficient.

[0007] Therefore, there is a great need for an improved language model training method, a copywriting generation method, and related devices to improve the scalability of language models.

[0008] In this context, embodiments of the present invention are expected to provide a language model training method, a copywriting generation method, and related devices.

[0009] According to one aspect of the present disclosure, there is provided a language model training method, the method comprising:

[0010] Obtain a sequence of text elements that make up the training text. The sequence of text elements is divided into a source segment sequence and a target segment sequence that are concatenated together. The sequence of text elements starts with a sequence start marker, and both the source segment sequence and the target segment sequence end with a sequence end marker;

[0011] Add masks to the sequence of text elements according to set rules;

[0012] Input the sequence of text elements with masks into the transformer network of the language model. Encode each text element in the sequence of text elements into a vector with context information through the transformer network, and output a representation vector;

[0013] Obtain the predicted values of the masks using the representation vectors, and train the transformer network according to the cross-loss function between the predicted values and the true values to make it converge, thereby obtaining the language model.

[0014] In an exemplary embodiment of the present disclosure,

[0015] In an exemplary embodiment of the present disclosure, the language model training method further includes:

[0016] Generate a mask matrix applicable to the task according to the sequence start marker and the sequence end marker, and add the mask matrix to the transformer network;

[0017] Encoding each text element in the sequence of text elements into a vector with context information through the transformer network includes:

[0018] Using the transformer network with the mask matrix to encode the output vector of the adjacent upper layer at each layer to generate the output vector of each layer.

[0019] In an exemplary embodiment of the present disclosure, using the transformer network with the mask matrix to encode the output vector of the adjacent upper layer at each layer to generate the output vector of each layer includes:

[0020] In the transformer network, use the mask matrix to determine the visible range of the text elements described by the output vector of the adjacent upper layer;

[0021] Use multiple self-attention heads at each layer to aggregate the output vectors of the adjacent upper layer within the visible range, and output the output vector with context information of each layer.

[0022] In an exemplary embodiment of the present disclosure, generating a mask matrix applicable to the task according to the sequence start marker and the sequence end marker includes:

[0023] Generate a mask matrix applicable to at least one task among the seq2seq model, the unidirectional language model, or the bidirectional language model according to the sequence start marker and the sequence end marker.

[0024] In an exemplary embodiment of the present disclosure, in the case where the task is a seq2seq model, generating a mask matrix applicable to the task according to a sequence start marker and a sequence end marker includes:

[0025] According to the sequence start marker and the sequence end marker, setting the source segment sequence to be bidirectionally visible and the target segment sequence to be forward visible according to a set rule, and generating a mask matrix applicable to the seq2seq model task.

[0026] In an exemplary embodiment of the present disclosure, adding a mask to a text element sequence according to a set rule includes:

[0027] Adding masks to both the source segment sequence and the target segment sequence, and using the resulting language model as a pre-trained language model.

[0028] In an exemplary embodiment of the present disclosure, obtaining a text element sequence constituting a training text includes:

[0029] Obtaining each text element constituting the text element sequence;

[0030] Performing text element encoding, position encoding, and segment encoding on each text element;

[0031] Constituting the encoded text elements into a text element sequence.

[0032] In an exemplary embodiment of the present disclosure, adding a mask to a text element sequence according to a set rule includes:

[0033] Setting the text elements in the text element sequence to be invisible or performing text element replacement.

[0034] According to one aspect of the present disclosure, there is provided a copywriting generation method, the method including:

[0035] Obtaining an input text element sequence, the input text element sequence including a source segment sequence starting with a sequence start marker and ending with a first sequence end marker, and a second sequence end marker for predicting the termination of the target segment sequence;

[0036] Inserting a mask for predicting the target segment text element between the first sequence end marker and the second sequence end marker according to a set rule;

[0037] Inputting the input text element sequence with the mask into a copywriting generation model trained by the language model training method according to any one of claims 1-8, encoding each text element in the input text element sequence into a vector with context information through a transformer network in the copywriting generation model, and outputting a representation vector;

[0038] Predict target segment text elements for the mask using the representation vector until reaching the end-of-second-sequence marker, and generate a target segment sequence using the target segment text elements, and output a text consisting of the source segment sequence and the target segment sequence.

[0039] In an exemplary embodiment of the present disclosure, the text generation method further includes:

[0040] Generate a mask matrix according to the start-of-sequence marker, the end-of-first-sequence marker, and the end-of-second-sequence marker, and add the mask matrix to the transformer network;

[0041] Encode each text element in the input text element sequence into a vector with context information through the transformer network in the text generation model, including:

[0042] Use the transformer network with the mask matrix to encode the output vector of the adjacent upper layer at each layer to generate the output vector of each layer.

[0043] In an exemplary embodiment of the present disclosure, use the transformer network with the mask matrix to encode the output vector of the adjacent upper layer at the target layer to generate the output vector of the target layer, including:

[0044] In the transformer network, use the mask matrix to determine the visible range of the text elements described by the output vector of the adjacent upper layer;

[0045] Use multiple self-attention heads at each layer to aggregate the output vectors of the adjacent upper layer within the visible range, and output the output vector with context information at each layer.

[0046] In an exemplary embodiment of the present disclosure, before generating the mask matrix according to the start-of-sequence marker, the end-of-first-sequence marker, and the end-of-second-sequence marker, the method further includes:

[0047] Receive user input;

[0048] In response to the user input, obtain the length of the target segment sequence, and determine the end-of-second-sequence marker representing the end of the target segment sequence according to the length of the target segment sequence.

[0049] In an exemplary embodiment of the present disclosure, predicting the target segment text elements for the mask using the representation vector includes:

[0050] Predict target segment text elements according to the representation vector to obtain the probabilities of multiple candidate text elements;

[0051] Use beam search to select the target number of target text elements with the top probabilities among the multiple candidate text elements.

[0052] In an exemplary embodiment of the present disclosure, obtaining an input text element sequence includes:

[0053] Obtaining each text element that constitutes the input text element sequence;

[0054] Performing text element encoding, position encoding, and segment encoding on each text element;

[0055] Forming the encoded text elements into an input text element sequence.

[0056] According to one aspect of the present disclosure, there is provided a language model training device, which includes:

[0057] A first acquisition module that acquires a text element sequence constituting the training text. The text element sequence is divided into a source segment sequence and a target segment sequence spliced together. The text element sequence starts with a sequence start marker, and both the source segment sequence and the target segment sequence end with a sequence end marker;

[0058] A first mask addition module that adds a mask to the text element sequence according to a set rule;

[0059] A first model input module that inputs the masked text element sequence into the transformer network of the language model, encodes each text element in the text element sequence into a vector with context information through the transformer network, and outputs a representation vector;

[0060] A training module that obtains a predicted value of the mask using the representation vector, and trains the transformer network according to the cross-loss function between the predicted value and the true value to make it converge, thereby obtaining a language model.

[0061] In an exemplary embodiment of the present disclosure, the language model training device further includes:

[0062] A first mask matrix generation module that generates a mask matrix applicable to the task according to the sequence start marker and the sequence end marker, and adds the mask matrix to the transformer network;

[0063] The first model input module is specifically configured to:

[0064] Use the transformer network with the mask matrix to encode the output vector of the adjacent upper layer at each layer to generate the output vector of each layer.

[0065] In an exemplary embodiment of the present disclosure, the first model input module is further specifically configured to:

[0066] In the transformer network, use the mask matrix to determine the visible range of the text element described by the output vector of the adjacent upper layer;

[0067] Aggregate the output vectors of the adjacent upper layer within the visible range using multiple self-attention heads in each layer, and output the output vectors with context information for each layer.

[0068] In an exemplary embodiment of the present disclosure, the first mask matrix generation module is specifically configured to:

[0069] Generate a mask matrix applicable to at least one task among the seq2seq model, the unidirectional language model, or the bidirectional language model according to the sequence start marker and the sequence end marker.

[0070] In an exemplary embodiment of the present disclosure, when the task is the seq2seq model, the first mask matrix generation module is specifically further configured to:

[0071] According to the sequence start marker and the sequence end marker, set the source segment sequence to be bidirectionally visible and the target segment sequence to be forward visible according to the set rules, and generate a mask matrix applicable to the seq2seq model task.

[0072] In an exemplary embodiment of the present disclosure, the first mask addition module is specifically configured to:

[0073] Add masks to both the source segment sequence and the target segment sequence, and obtain the language model as the pre-trained language model.

[0074] In an exemplary embodiment of the present disclosure, the first acquisition module is specifically configured to:

[0075] Obtain each text element constituting the text element sequence;

[0076] Perform text element encoding, position encoding, and segment encoding on each text element;

[0077] Construct the encoded text elements into a text element sequence.

[0078] In an exemplary embodiment of the present disclosure, the first mask addition module is specifically configured to:

[0079] Set the text elements in the text element sequence to be invisible or perform text element replacement.

[0080] According to one aspect of the present disclosure, there is provided a copywriting generation device, the device includes:

[0081] A second acquisition module, which acquires an input text element sequence, the input text element sequence includes a source segment sequence starting with a sequence start marker and ending with a first sequence end marker, and a second sequence end marker for terminating the prediction of the target segment sequence;

[0082] The second mask addition module inserts a mask for predicting the target segment text elements between the first sequence end marker and the second sequence end marker according to a set rule;

[0083] The second model input module inputs the sequence of input text elements with the mask into the copywriting generation model trained by the language model training method of any of the above embodiments of the present disclosure, encodes each text element in the sequence of input text elements into a vector with context information through the transformer network in the copywriting generation model, and outputs a representation vector;

[0084] The prediction module uses the representation vector to predict the target segment text elements until it terminates at the second sequence end marker, generates a target segment sequence using the target segment text elements, and outputs a copywriting composed of the source segment sequence and the target segment sequence.

[0085] In an exemplary embodiment of the present disclosure, the copywriting generation device further includes:

[0086] The second mask matrix generation module generates a mask matrix according to the sequence start marker, the first sequence end marker, and the second sequence end marker, and adds the mask matrix to the transformer network;

[0087] The second model input module is specifically configured to:

[0088] Use the transformer network with the mask matrix to encode the output vector of the adjacent upper layer at each layer to generate the output vector of each layer.

[0089] In an exemplary embodiment of the present disclosure, the second model input module is further specifically configured to:

[0090] In the transformer network, use the mask matrix to determine the visible range of the text elements described by the output vector of the adjacent upper layer;

[0091] Use multiple self-attention heads at each layer to aggregate the output vectors of the adjacent upper layer within the visible range, and output the output vector with context information of each layer.

[0092] In an exemplary embodiment of the present disclosure, the copywriting generation device further includes:

[0093] The receiving module receives user input before generating the mask matrix according to the sequence start marker, the first sequence end marker, and the second sequence end marker;

[0094] The response module, in response to the user input, obtains the length of the target segment sequence, and determines the second sequence end marker representing the end of the target segment sequence according to the length of the target segment sequence.

[0095] In an exemplary embodiment of the present disclosure, the prediction module is specifically configured to:

[0096] Predict target segment text elements based on the representation vectors to obtain the probabilities of multiple candidate text elements;

[0097] Use beam search to select the target number of target text elements with the top probabilities among multiple candidate text elements.

[0098] In an exemplary embodiment of the present disclosure, the second acquisition module is specifically configured to:

[0099] Acquire each text element constituting the input text element sequence;

[0100] Perform text element encoding, position encoding, and segment encoding on each text element;

[0101] Constitute the encoded text elements into an input text element sequence.

[0102] According to one aspect of the present disclosure, there is provided a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it implements: a language model training method and / or a copywriting generation method.

[0103] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0104] A processor; and

[0105] A memory for storing executable instructions of the processor;

[0106] Wherein, the processor is configured to execute the language model training generation method and / or the copywriting generation method of any one of the above by executing the executable instructions.

[0107] According to the language model training method and the copywriting generation method of the embodiments of the present disclosure, by adding a sequence start marker and a sequence end marker to the text element sequence, at this time the text element sequence can be regarded as a segmented text in which a pair of source segment sequences and target segment sequences are packed, or it can also be a two-segment segmented text. In this way, the process of predicting the mask is actually a sentence completion task. In this case, using the attention mechanism of a single transformer network, it is possible to encode the text elements by controlling the context information visible to the mask to apply to the training tasks of the above one or two cases, and the feasibility of this solution is high.

[0108] The embodiments of the present disclosure can achieve the training objectives of multiple training tasks without distinguishing between the encoder and decoder of the transformer network, because the encoder and decoder are unified. Not only is the structure of the transformer network simple, but obviously it can also improve the scalability of the language model. The language model training tasks disclosed have strong generality and high training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the drawings, several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0110] Figure 1 One of the flowcharts of the language model training method according to an embodiment of the present disclosure is schematically shown;

[0111] Figure 2 A schematic diagram of a language model according to an embodiment of the present disclosure is schematically shown;

[0112] Figure 3 A schematic diagram of the input text element sequence during training according to an embodiment of the present disclosure is schematically shown;

[0113] Figure 4 Another flowchart of the language model training method according to an embodiment of the present disclosure is schematically shown;

[0114] Figure 5 and Figure 6 is schematically shown Figure 2 a schematic diagram of the internal operation principle of the shown language model during training;

[0115] Figure 7 One of the flowcharts of the copywriting generation method according to an embodiment of the present disclosure is schematically shown;

[0116] Figure 8 Another flowchart of the copywriting generation method according to an embodiment of the present disclosure is schematically shown;

[0117] Figure 9 A schematic diagram of the principle of the copywriting generation method according to an embodiment of the present disclosure is schematically shown;

[0118] Figure 10 One of the structural diagrams of the language model training device according to an embodiment of the present disclosure is schematically shown;

[0119] Figure 11 Another structural diagram of the language model training device according to an embodiment of the present disclosure is schematically shown;

[0120] Figure 12 One of the structural diagrams of the copywriting generation device according to an embodiment of the present disclosure is schematically shown;

[0121] Figure 13 Another structural diagram of the copywriting generation device according to an embodiment of the present disclosure is schematically shown;

[0122] Figure 14Schematically shows the third structural diagram of the copywriting generation device according to an embodiment of the present disclosure;

[0123] Figure 15 Schematically shows a schematic diagram of a storage medium according to an embodiment of the present disclosure; and

[0124] Figure 16 Schematically shows a block diagram of an electronic device according to an embodiment of the disclosure.

[0125] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts. Detailed implementation manners

[0126] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and then implement the present invention, rather than limiting the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to be able to fully convey the scope of the present disclosure to those skilled in the art.

[0127] Those skilled in the art know that the embodiments of the present disclosure can be implemented as a system, a device, an apparatus, a method, or a computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: completely hardware, completely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0128] According to an embodiment of the present disclosure, a language model training method, a copywriting generation method, and related devices are provided.

[0129] In this article, the number of any element in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0130] The principles and spirit of the present disclosure will be elaborated in detail below with reference to several representative embodiments of the present disclosure. Summary of the Invention

[0132] The inventor of the present invention has found that in text processing tasks, the accuracy of text processing can be improved by designing a language model with a complex network structure, but this reduces the scalability of the language model in text processing scenarios.

[0133] Currently, the following general scheme is generally adopted to implement text processing tasks:

[0134] Solution 1: Use ELMo based on LSTM. As a typical autoregressive pre-training model, ELMo performs autoregressive pre-training using two independent unidirectional LSTMs to implement a unidirectional language model. The forward language model reads the text from left to right for encoding, and the backward language model reads the text from right to left for encoding;

[0135] Solution 2: Use the GPT (Generative Pre-Training) model based on OpenAI. The GPT model is also a classic model in the field of text generation. It uses a Transformer encoder-decoder to predict the text sequence word by word from left to right.

[0136] Among them, the reason why Solution 1 does not use bidirectional LSTM for encoding is that in the pre-training task, the bidirectional model will see the context representation in advance and affect the prediction result. Therefore, in order not to affect the prediction result, ELMo essentially still belongs to a unidirectional language model because it only encodes and represents in one direction, only splicing two unidirectional language models together.

[0137] The feature extractor of ELMo is LSTM. Although LSTM is naturally suitable for processing sequence information, it cannot handle long-distance dependencies due to problems such as gradient disappearance. Moreover, since ELMo needs to train 2 independent unidirectional LSTMs, its parallel ability is poor.

[0138] The GPT model in Solution 2 uses the decoder of the Transformer network for feature extraction, which can solve the problem of weak feature extraction ability of ELMo. However, it is a left-to-right restricted Transformer and belongs to a unidirectional language model, that is, an autoregressive language model, which determines that the GPT model cannot perform bidirectional feature representation through context information and its scalability is insufficient.

[0139] Therefore, both Solution 1 and Solution 2 can only achieve the training task of unidirectional language models and have insufficient scalability.

[0140] In view of the above content, the basic idea of the present invention is:

[0141] By adding a sequence start marker and a sequence end marker to the text element sequence, the text element sequence can be regarded as a split text in which a pair of source segment sequences and target segment sequences are packed, or it can also be a two-segment split text. In this way, the process of predicting the mask is actually a sentence completion task. In this case, using the attention mechanism of a single Transformer network, the text elements can be encoded by controlling the context information visible to the mask to apply to the training tasks of the above one or two situations.

[0142] Embodiments of the present disclosure can achieve the training objectives of various training tasks without distinguishing between the encoder and decoder of the transformer network, because the encoder and decoder are unified. Not only is the structure of the transformer network simple, but it is also obvious that the scalability of the language model can be improved. The disclosed language model training tasks are highly versatile and have high training efficiency.

[0143] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be specifically introduced below.

[0144] Exemplary Method

[0145] The following Figure 1 will be used to describe the language model training method according to an exemplary embodiment of the present disclosure. Referring to Figure 1 as shown, the language model training method may include the following steps:

[0146] Step 110: Obtain a sequence of text elements that make up the training text. The sequence of text elements is divided into a source segment sequence and a target segment sequence spliced together. The sequence of text elements starts with a sequence start marker, and both the source segment sequence and the target segment sequence end with a sequence end marker;

[0147] Step 120: Add a mask to the sequence of text elements according to a set rule;

[0148] Step 130: Input the sequence of text elements with the mask into the transformer network of the language model, and encode each text element in the sequence of text elements into a vector with context information through the transformer network, and output a representation vector;

[0149] Step 140: Obtain the predicted value of the mask using the representation vector, and train the transformer network according to the cross-loss function between the predicted value and the true value to make it converge, and obtain the language model.

[0150] In the embodiments of the present disclosure, a mask is added to the sequence of text elements, and the function of the mask is to mask the text elements to be predicted during the training process. During the encoding process, each text element is encoded into a representation vector with context messages, and the context messages indicate whether the text element is affected by other text elements or the degree of influence.

[0151] Among them, the attention mechanism of the transformer network can control the context information of the text elements.

[0152] A language model training method and a copywriting generation method according to an embodiment of the present disclosure. By adding a sequence start marker and a sequence end marker to a text element sequence, the text element sequence can be regarded as a segmented text in which a source segment sequence and a target segment sequence are packed together, or it can also be a two-segment segmented text. In this way, the process of predicting the mask is actually a sentence completion task. In this case, by using the attention mechanism of a single transformer network, the text elements can be encoded by controlling the context information visible to the mask to apply to the training tasks of the above one or two situations.

[0153] The embodiments of the present disclosure can achieve the training objectives of multiple training tasks without distinguishing between the encoder and decoder of the transformer network because the encoder and decoder are unified. Not only is the structure of the transformer network simple, but obviously, the scalability of the language model can also be improved. The disclosed language model training tasks have strong versatility and high training efficiency.

[0154] According to an exemplary embodiment of the present disclosure, referring to Figure 2 , the language model includes a single multi-layer transformer network, and the multi-layer transformer network is sequentially connected by multiple transformer blocks. The output of the previous transformer block in two adjacent layers of transformer blocks is used as the input of the current layer of transformer blocks.

[0155] By using a multi-layer transformer network, more abstract relationships between text element sequences can be mined, enabling the representation vector to express the input text element sequence more abstractly, making the language model have better generalization ability and striving to improve the prediction results of the language model.

[0156] As Figure 2 shown, given a text element sequence x = [x1,..., x n of the input training text, Figure 2 taking n = 5 as an example for illustration, only for example, each x i represents a text element of the input sequence, which can be a word or a character;

[0157] The transformer network obtains a context vector representation for each text element and outputs a representation vector h = [h1,..., h n . Corresponding to n = 5, 5 groups of representation vectors are output. Among them, to control the access to the context of the text element marked at the position of the mask to be predicted, the transformer network can be implemented using different self-attention mechanisms. The following specifically elaborates each step of the language model training method.

[0158] According to an exemplary embodiment of the present disclosure, obtaining the text element sequence constituting the training text specifically includes the following steps:

[0159] Obtaining each text element constituting the text element sequence;

[0160] Perform text element encoding, position encoding, and segment encoding on each text element;

[0161] Construct a text element sequence from the encoded text elements.

[0162] The original training text is composed of a series of ordered text elements, where a text element is the basic building block of the text, which can be a word or a character, and is not limited here.

[0163] According to an exemplary embodiment of the present disclosure, based on the powerful feature extraction ability of the transformer, word segmentation is no longer necessary, and word-level feature learning can be incorporated as internal features for representation learning. Therefore, each text element in the input sequence x can be a single character, which can solve the problem that word input exacerbates out-of-vocabulary (OOV, full English name: out of vocabulary), thereby reducing the input space and avoiding using a much larger corpus.

[0164] Combine Figure 2 As shown, the original training text element sequence passes through the embedding layer of the language model, and through text element embedding (word embedding), position embedding, and segment embedding, each text element obtains text element encoding, position encoding, and segment encoding. The text features of the text element can be characterized by a vector representation obtained by summing a group of text element encoding, position encoding, and segment encoding.

[0165] Position encoding is introduced because order is very important information. The order of text elements represents the local or global structure of the text. If the order information of the sequence cannot be learned, the performance of the language model will be greatly reduced. Therefore, this embodiment introduces position encoding to encode the absolute position of the text element in the text sequence. And segment encoding is used to distinguish different paragraphs in the training text.

[0166] Refer to Figure 3 As shown, the input training text includes 5 text elements, namely x1, x2, x3, x4, and x5. The training text can be divided into the first paragraph S1 and the second paragraph S2 through the sequence end marker [EOS]. S1 and S2 have different segment encodings to distinguish whether the segment belongs to the source segment or the target segment in the text element sequence.

[0167] Meanwhile, the text element sequence starts with the sequence start marker [SOS], and both the source segment sequence and the target segment sequence end with the sequence end marker [EOS]. Among them, the sequence start marker [SOS] is used to identify the start of the input text element sequence, and each [EOS] represents the decoding process in the termination training task of the language model for a specific text segment.

[0168] According to an exemplary embodiment of the present disclosure, after obtaining the vector encoding of each text element, these vectors are used as input vectors and input into a multi-layer transformer network.

[0169] According to an exemplary embodiment of the present disclosure, the transformer network uses a multi-head self-attention mechanism to perform context vector representation on each text element in the input text element sequence.

[0170] Specifically, according to an exemplary embodiment of the present disclosure, as shown in Figure 4 The language model training method may include the following steps:

[0171] Step 410: Obtain a text element sequence constituting the training text. The text element sequence is divided into a source segment sequence and a target segment sequence spliced together. The text element sequence starts with a sequence start marker, and both the source segment sequence and the target segment sequence end with a sequence end marker;

[0172] Step 420: Add a mask to the text element sequence according to a set rule;

[0173] Step 430: Generate a mask matrix applicable to the task according to the sequence start marker and the sequence end marker, and add the mask matrix to the transformer network;

[0174] Step 440: Use the transformer network with the mask matrix to encode the output vector of the adjacent upper layer at each layer to generate the output vector of each layer until the representation vector of the text element sequence is output;

[0175] Step 450: Use the representation vector to obtain the predicted value of the mask, and train the transformer network according to the cross-loss function between the predicted value and the true value to make it converge and obtain the language model.

[0176] In an exemplary embodiment of the present disclosure, the adjacent upper layer refers to the upper layer transformer block adjacent to each layer in the transformer network. Among them, each layer of transformer block aggregates and encodes the output vector of the adjacent upper layer transformer block, so that each layer of transformer block captures more abstract text information than the previous layer.

[0177] Among them, the output vector of the last layer is used as the representation vector of the text element sequence.

[0178] In each transformer block, multiple self-attention heads are used to aggregate the output vectors of the previous layer, that is, to encode the output vectors of the previous layer. The encoded output of each layer of the transformer block is as follows:

[0179] H l = Transformer l (H l-1 ), l ∈ [1, L]

[0180] The output H l-1 of the previous layer is linearly projected into a triple Q, K, V of sequences (queries), keys, and values using parameter matrices W l Q , W l K , W l V respectively. That is, Q, K, and V are all sequences of text elements of the same input obtained through linear transformation. Attention is Attention(x, x, x), where x is the input sequence. That is, within the sequence of text elements, Attention is performed to find the internal connections within the sequence.

[0181] The input sequence of text elements is a set of vector matrices, and in the transformer block, a mask matrix configured by multiple self-attention heads is used to control the attention range of each text element described by the output vector of the current transformer block to the output vector of the previous layer of the transformer block. That is, through different mask matrices, the number of visible context text elements for predicting text elements can be controlled to achieve different model representations.

[0182] The self-attention calculation of each layer is implemented using a mask matrix, which can accelerate the calculation and obtain the output vectors at all positions at once.

[0183] In an exemplary embodiment of the present disclosure, a transformer network with a mask matrix is used to encode the output vector of the adjacent previous layer at each layer to generate the output vector of each layer. Specifically, the following steps may be included:

[0184] In the transformer network, a mask matrix is used to determine the visible range of the text elements described by the output vector of the adjacent previous layer;

[0185] Multiple self-attention heads of each layer are used to aggregate the output vectors of the adjacent previous layer within the visible range, and an output vector with context information of each layer is output.

[0186] In an exemplary embodiment of the present disclosure, reference is made to Figure 2As shown, the topmost mask matrix A is the mask matrix applicable to the seq2seq model. In this case, the mask matrix applicable to the seq2seq task can be generated based on the sequence start token and the sequence end token.

[0187] Among them, in the case where the task is a seq2seq model, generating a mask matrix applicable to the task based on the sequence start token and the sequence end token includes:

[0188] Based on the sequence start token and the sequence end token, setting the source segment sequence to be bidirectionally visible and the target segment sequence to be forward visible according to the set rules, and generating a mask matrix applicable to the seq2seq model task.

[0189] Specifically, the shaded blocks indicate being invisible and will be masked, while the unfilled blocks represent being visible. Referring to Figure 5 , the input text element sequence includes a first part composed of the source segment sequence S1 and a second part composed of the target segment sequence S2. Among them, the solid line between two adjacent transformer blocks represents the connection between the source segment sequence S1 and other text elements output by the upper-layer transformer block in the lower-layer transformer block. The dashed line represents the connection between the target segment sequence S2 and other text elements output by the upper layer in the lower-layer transformer block. Among them, the connection represents being visible, and the non-connection represents being invisible.

[0190] Specifically, in the lower-layer transformer block, each text element in the source segment sequence S1 is connected to each text element in the source segment sequence S1 output by the upper-layer transformer block. That is, the text elements in the first part can be mutually visible in two directions within the source segment sequence S1, that is, each text element in the source segment sequence S1 is affected by the text elements on both the left and right sides within the source segment sequence S1. However, each text element in the source segment sequence S1 is not connected to the target segment sequence S2 output by the upper-layer transformer block, that is, the target segment sequence S2 output by the upper-layer transformer block is invisible to each text element in the source segment sequence S1.

[0191] Specifically, in the lower-layer transformer block, the text elements in the target segment sequence S2 are connected to the text elements in the text sequence output by the upper-layer transformer block that are before the position of this text element, but not connected to the text elements after the position and itself. That is, the text elements in the target segment sequence S2 in the second part are visible to all the text elements on the left side of itself in the target segment sequence S2 and all the text elements in the source segment sequence S1 in the first part.

[0192] For example, referring to Figure 6, given x1 and x2 in the source segment sequence S1 and x3, x4, and x5 in its target segment sequence, input the input text element sequence "[SOS]x1x2[EOS]x3x4x5 [EOS]" into the transformer network, where x1 and x2 in the source segment sequence S1 are visible to [SOS], x1, x2, [EOS], but x4 in the target segment sequence S2 is visible to [SOS], x1, x2, [EOS], x3, and not visible to x4, x5, [EOS].

[0193] Thus, for the l-th layer transformer block, the self-attention head A l output is calculated as follows:

[0194] Q = H l-1 W l Q , K = H l-1 W l K , V = H l-1 W l V

[0195]

[0196]

[0197] where 0 indicates visible, and negative infinity indicates not visible and will be masked out.

[0198] Using the exemplary embodiments of the present disclosure, since during language model training, the source segment sequence and the target segment sequence are combined into the language model, enabling the model to implicitly learn the relationship between the two texts, achieving the seq-to-seq effect. The core of the language model is to endow the model with Seq2Seq capabilities through special attention masks. Among them, the source segment sequence is contextually visible and can be bidirectional attention Attention, while the text elements in the target segment sequence are outputs, and the output sequence is unidirectionally visible because it is a word-by-word prediction, and the previous output text elements cannot see the subsequent output text elements. Therefore, the connected is unidirectional attention, thus allowing recursive prediction of text elements.

[0199] In this way, the Attention in the input part is bidirectional, and the Attention in the output part is unidirectional, meeting the requirements of Seq2Seq, and there are no additional constraints. Therefore, through this ingenious transformation, only one transformer network needs to be trained to have text generation capabilities. The exemplary embodiments of the present disclosure only need to design to add the mask Mask of the above shape without modifying the model architecture, and the network structure of the language model is simple, and the implementation method of the solution has high feasibility and reliability.

[0200] In an exemplary embodiment of the present disclosure, a mask matrix applicable to at least one task among a seq2seq model, a unidirectional language model, or a bidirectional language model is generated according to a sequence start marker and a sequence end marker.

[0201] Referring again to Figure 2 Mask matrix B applicable to the unidirectional language model shown. The upper triangular matrix is used as the mask matrix, with the shaded part being -∞ and the blank part being 0. From front to back, that is, the masked text element is predicted by all text elements before the text element to which the mask is added. For example, to predict [Mask] in the text sequence "x1x2[MASK]x4", x1, x2, and itself can be used for encoding.

[0202] Referring again to Figure 2 Mask matrix C applicable to the bidirectional language model shown. When predicting the masked text element, all text elements can be observed. For example, to predict [Mask] in the text sequence "x1x2[MASK]x4", x1, x2, x4, and itself can be used for encoding. Specifically, a matrix of all 0s is used as the mask matrix.

[0203] Therefore, for the unidirectional language model, the text element sequence input to the language model is a single text, although the text is split into two segments. For the bidirectional language model and the sequence-to-sequence language model, the text element sequence input to the language model is a text pair, that is, equivalent to two sentences.

[0204] Therefore, for different language models, we can complete joint training by only changing the form of the mask matrix, enabling the same program to perform different tasks. It can also be understood that by different training tasks, the same sub-model parameters are optimized. In the training tasks of different language models, the model parameters are shared and there is no need to train and configure separately on multiple language models. Moreover, parameter sharing enables the learned text vector representations to have stronger generalization ability, and can be jointly optimized for different language model targets, reducing overfitting to any single language model task.

[0205] Therefore, using the exemplary embodiment of the present disclosure, a mask matrix applicable to at least one task among a seq2seq model, a unidirectional language model, or a bidirectional language model can be generated according to a sequence start marker and a sequence end marker.

[0206] Using the exemplary embodiments of the present disclosure, the language model can aggregate context for different types of language models by designing a flexible self-masking matrix, which can be used not only for natural language understanding (NLU) tasks but also for natural language generation (NLG) tasks. The language model training task is general and flexible.

[0207] In the exemplary embodiments of the present disclosure, a mask is added to the text element sequence according to a set rule. Specifically, a mask is added to the target segment sequence according to a set rule by using a transformer network, and the resulting language model is used as a copywriting generation model. Specifically, a certain proportion of text elements in the target segment sequence can be randomly masked, while the source segment sequence is not masked. During the training phase, the language model is allowed to learn to recover the masked text elements.

[0208] In the exemplary embodiments of the present disclosure, the language model can be used as a pre-trained language model. In this way, masks are added to both the source segment sequence and the target segment sequence according to a set rule, and the resulting language model is used as a pre-trained language model. Specifically, when making masks during the training phase, the text elements of both the source segment sequence and the target segment sequence are randomly masked, that is, both segments participate in the training.

[0209] In the exemplary embodiments of the present disclosure, adding a mask to the text element sequence according to a set rule may specifically include:

[0210] Setting the text elements in the text element sequence to be invisible or performing text element replacement.

[0211] In the exemplary embodiments of the present disclosure, referring to Figure 2 As shown, the input text element sequence x1, x2, x3, x4, x5 outputs representation vectors h1, h2, h3, h4, h5 after passing through a multi-layer transformer network. In the exemplary embodiments of the present disclosure, the representation vectors are input into a softmax classifier to predict the predicted values of the masks.

[0212] The training objective of the language model is the cross-entropy loss function between the predicted values and the true values of the masked text elements, which can be represented by the following formula:

[0213]

[0214] where M represents the set of masked words, and |V| is the size of the vocabulary used.

[0215] Among them, cross entropy is an important concept introduced from information theory, which is used to measure the average number of bits required to represent the occurrence of an event using the strategy specified by the non-real distribution q under the given real distribution p. In the machine learning scenario, the cross-entropy loss function, as a loss function, measures the similarity between the real data distribution (true value) and the model prediction result (predicted value).

[0216] In an exemplary embodiment of the present disclosure, the softmax classifier is a normalized exponential function, which belongs to the gradient logarithm normalization of a finite-term discrete probability distribution and can be used in combination with the cross-entropy loss function.

[0217] The following combines Figure 7 to describe a copywriting generation method according to an exemplary embodiment of the present disclosure.

[0218] Refer to Figure 7 , the copywriting generation method includes the following steps:

[0219] Step 710: Obtain an input text element sequence, where the input text element sequence includes a source segment sequence starting with a sequence start marker and ending with a first sequence end marker, and a second sequence end marker for terminating the prediction of the target segment sequence;

[0220] Step 720: Insert a mask for predicting the target segment text element between the first sequence end marker and the second sequence end marker according to a set rule;

[0221] Step 730: Input the input text element sequence with the mask into the copywriting generation model trained by the language model training method of any of the above embodiments, and encode each text element in the input text element sequence into a vector with context information through the transformer network in the copywriting generation model, and output a representation vector;

[0222] Step 740: Use the representation vector to predict the target segment text element for the mask until it terminates at the second sequence end marker, and use the target segment text element to generate a target segment sequence, and output a copywriting composed of the source segment sequence and the target segment sequence.

[0223] In an exemplary embodiment of the present disclosure, the input text element sequence is used to predict the target segment text element until the second sequence end marker, and finally a target segment sequence composed of the target segment text elements in sequence is obtained, and a complete copywriting composed of the source segment sequence and the target segment sequence output by the copywriting generation model is obtained.

[0224] Determine the position of the second sequence end marker based on the length of the pre-obtained target segment sequence, which represents the end of the target segment sequence and is also the marker for terminating the prediction of the target segment sequence. Therefore, from the start to the end of the prediction, multiple rounds of target segment text element predictions may be experienced. The target segment text elements obtained in each round of prediction will be re-inserted into the input text element sequence, and the above steps 710 and 720 will be repeated until the second sequence end marker is decoded.

[0225] According to an exemplary embodiment of the present disclosure, by adopting a copywriting generation model based on a single transformer network, the copywriting generation task is converted into a sentence completion task to achieve the effect of seq2seq. In the case where the network structure of the copywriting generation model is simple, reliable results are obtained in the application of the copywriting generation task.

[0226] According to an exemplary embodiment of the present disclosure, obtaining an input text element sequence may specifically include the following steps:

[0227] Obtain each text element constituting the input text element sequence;

[0228] Perform text element encoding, position encoding, and segment encoding on each text element;

[0229] Construct the encoded text elements into an input text element sequence.

[0230] In an exemplary embodiment of the present disclosure, text features are extracted for each text element in the input text, and each text element can be vector-represented using text element encoding, position encoding, and segment encoding. In this way, a set of vector representations can be obtained for the corresponding input text, and this set of vector representations serves as the input text element sequence.

[0231] In an exemplary embodiment of the present disclosure, the mask added between the first sequence end marker and the second sequence end marker defines the position of the target segment text element to be predicted.

[0232] When predicting the first target segment text element of the target segment sequence, masks are added to the positions of all target segment text elements to be predicted according to the length of the target segment sequence to be predicted, and the masked input is fed into the copywriting generation model to output the target segment text element corresponding to the first mask.

[0233] In the case where the previous target segment text elements have been predicted, masks are added to all positions after the obtained target segment text elements, and the masked input text element sequence is again fed into the copywriting generation model for a new round of target segment text element prediction, so as to obtain new target segment text elements for the masks adjacent to the obtained target segment text elements.

[0234] Therefore, through iteration, all target segment text elements that make up the target segment sequence can be obtained. These target segment text elements are used to generate the target segment sequence and finally output the copywriting.

[0235] In an exemplary embodiment of the present disclosure, the mask added to the original input text element sequence is determined according to the length of the target segment sequence to be predicted, which allows users to customize and brings a good user experience.

[0236] According to an exemplary embodiment of the present disclosure, the transformer network is a multi-layer transformer network. Using a multi-layer transformer network can mine more abstract relationships between text element sequences, enabling the representation vector to express the input text element sequence more abstractly, making the copywriting generation model have better generalization ability, and the results of the copywriting generation task are more reliable and accurate.

[0237] According to an exemplary embodiment of the present disclosure, refer to Figure 8 , the copywriting generation method includes the following steps:

[0238] Step 810: Obtain an input text element sequence, where the input text element sequence includes a source segment sequence starting with a sequence start marker and ending with a first sequence end marker, and a second sequence end marker for terminating the prediction of the target segment sequence;

[0239] Step 820: Insert a mask for predicting target segment text elements between the first sequence end marker and the second sequence end marker according to a set rule;

[0240] Step 830: Generate a mask matrix according to the sequence start marker, the first sequence end marker, and the second sequence end marker, and add the mask matrix to the transformer network;

[0241] Step 840: Input the masked input text element sequence into the copywriting generation model trained by the language model training method of any of the above-mentioned exemplary embodiments of the present disclosure. Use the transformer network with the mask matrix to encode the output vector of the adjacent upper layer at each layer to generate the output vector of each layer until the representation vector of the output text element sequence is obtained;

[0242] Step 850: Use the representation vector to predict the target segment text elements for the mask until it terminates at the second sequence end marker, and use the target segment text elements to generate the target segment sequence, and output the copywriting composed of the source segment sequence and the target segment sequence.

[0243] In each layer of the transformer block, multiple self-attention heads are used to aggregate the output vectors of the previous layer. The encoding output of each layer of the transformer block is as follows:

[0244] H l= Transformer l (H l-1 ),l ∈ [1, L]

[0245] The output H of the previous layer l-1 Respectively use the parameter matrices W l Q ,W l K ,W l V Project linearly to the triple Q, K, V of sequence (query), key and value to find the connections between text elements within the text element sequence, so as to output a representation vector with context information.

[0246] In an exemplary embodiment of the present disclosure, by pre-setting the second sequence end marker, the length of the target segment sequence can be customized, and finally a customized copy length can be obtained, improving the user experience of the copy generation task.

[0247] The input text element sequence uses a mask matrix configured with multiple self-attention heads in the transformer block to control the visible range of each text element, to control the number of visible context text elements of the target segment text elements to be predicted, and to realize the vector representation of the masked context information.

[0248] In an exemplary embodiment of the present disclosure, using a transformer network with a mask matrix to encode the output vector of the adjacent previous layer at the target layer to generate the output vector of the target layer may include the following steps:

[0249] In the transformer network, use the mask matrix to determine the visible range of the text elements described by the output vector of the adjacent previous layer;

[0250] Use multiple self-attention heads of each layer to aggregate the output vectors of the adjacent previous layer within the visible range, and output the output vector with context information of each layer.

[0251] In this case, since the target segment sequence is predicted using the input text element sequence of the previous side, the mask matrix is designed to be visible to the context on the left side of the mask.

[0252] According to an exemplary embodiment of the present disclosure, the length of the target segment sequence can be default set, and the position of the second sequence end marker is default set.

[0253] According to an exemplary embodiment of the present disclosure, the copy generation method further includes:

[0254] Before generating the mask matrix according to the sequence start marker, the first sequence end marker and the second sequence end marker, receive user input;

[0255] In response to user input, obtain the length of the target segment sequence, and determine the second sequence end marker at the end of the target segment sequence to be predicted according to the length of the target segment sequence to be predicted.

[0256] In an exemplary embodiment of the present disclosure, a user customization solution for the copywriting length is provided, and the position of the second sequence end marker can be determined according to the response to user input.

[0257] In addition, according to the user-defined target segment sequence, a corresponding target segment mask sequence can be generated and inserted between the first sequence end marker and the second sequence end marker to mark the position of the target segment sequence.

[0258] The exemplary embodiment of the present disclosure enables the user to customize the length of the target segment sequence to be predicted, enhancing the user experience.

[0259] As described above, corresponding to the length of the target segment sequence to be predicted, multiple rounds of prediction of target segment text elements are performed. Among them, for each round of prediction of target segment text elements, the target segment text elements are predicted using the representation vector, which specifically may include the following steps:

[0260] Predict target segment text elements based on the representation vector to obtain the probabilities of multiple candidate text elements;

[0261] Use beam search to select the target number of target text elements with the top probabilities among multiple candidate text elements.

[0262] In this case, when the target number is greater than 1, multiple target segment text elements can be obtained for each round of prediction of target segment text elements. In this way, each target segment text element can be put into the input text element sequence for the next round of prediction of target segment text elements.

[0263] Therefore, using the exemplary embodiment of the present disclosure, corresponding to multiple possible target segment text element sequences obtained in each round and after multiple rounds of prediction superposition, multiple groups of copywriting can be obtained.

[0264] In an exemplary embodiment of the present disclosure, for an input text element sequence, the above copywriting generation model can predict the probability of each word of this input sequence on the vocabulary. And the Seq2Seq task is to continuously generate a new output sequence based on the known input sequence. That is, under the condition of the known input sequence X, find the output sequence Y with the maximum conditional probability P(Y|X):

[0265] P(Y|X) = P(Y1|X)P(Y2|X,Y1)P(Y3|X,Y1,Y2)……P(Yn|X,Y1,Y2..Yn-1)

[0266] With Figure 9For example, the input text element sequence is X = "I want to eat". The probability value for each round in the vocabulary on the left is the probability value predicted by the copywriting generation model for the input text element sequence in each round on the vocabulary. There are five characters, namely "apple, fruit, kiwi, monkey, peach", in the vocabulary, and the target output is "I want to eat kiwi fruit".

[0267] In the first round of the copywriting generation model, the probability value of outputting "apple" is the highest, which is 0.4. If it is greedy search, the copywriting generation model will confirm the output of the first round as "apple". The search decoder beam search will select the top(k) results for output. If k is set to 2 (the parameter k can be adjusted here), then the model will output two results at the same time, namely "I want to eat apple" and "I want to eat kiwi", that is, the top(2) of P(Y1|X).

[0268] In the second round, greedy search will continue to output the character with the highest probability value based on "I like to eat apple", while beam search will get a total of 2X2 = 4 combinations based on "I want to eat kiwi" and "I want to eat apple" respectively. At this time, the total probability of each combination will be calculated and the top two will be taken again, that is, the top(2) of P(Y2|X,Y1). For example, "I want to eat kiwi fruit" and "I want to eat apple K1" are obtained. Next, continue with the third round. In this way, the output result of each subsequent round can only take the top(2). Although it is essentially a local search, it combines more possibilities. The final result is to multiply the probabilities of each round, and the combination with the largest multiplied probability is used as the final output, making the final generation result of the model better.

[0269] In the exemplary embodiment of the present disclosure, a random sampling generation strategy can also be adopted. When the copywriting generation model decodes each round, it will randomly sample a corresponding target segment text element from the top(k) probability values for generation. This method is beneficial to the diversity of the generation results.

[0270] According to the exemplary embodiment of the present disclosure, the language model training method and the corresponding copywriting generation method can be applied to copywriting generation tasks such as live broadcast copywriting, playlist copywriting, comments, and advertising slogans.

[0271] For example, in the scenario of generating personalized copywriting for the host in the live broadcast recommendation system, first use a large amount of historical corpus data and open-source corpus in the corresponding live broadcast scenario, and train the copywriting generation model using the exemplary language model training method disclosed above.

[0272] In the case of obtaining a trained copywriting generation model, user tags of a known anchor or a self-set title are used as an input text element sequence. The copywriting generation model inputs this input text element sequence and outputs a target segment sequence predicted based on this input text element sequence. The input text element sequence and the target segment sequence together constitute a complete copywriting. This copywriting is automatically generated based on user tags or a self-set title and can better reflect the individuality of the anchor.

[0273] In a specific application scenario, the input from the anchor user can be received through a user interface, and corresponding copywriting can be automatically generated according to the text input by the anchor user and displayed on the anchor interface.

[0274] Exemplary Apparatus

[0275] After introducing the language model training method and the copywriting generation method of the exemplary embodiments of the present disclosure, next, reference is made to Figure 10 to describe the language model training device of the exemplary embodiments of the present disclosure.

[0276] Reference is made to Figure 10 As shown, the language model training device 1000 of the exemplary embodiments of the present disclosure may include:

[0277] A first acquisition module 1010 that acquires a text element sequence constituting a training text. The text element sequence is divided into a source segment sequence and a target segment sequence spliced together. The text element sequence starts with a sequence start marker, and both the source segment sequence and the target segment sequence end with a sequence end marker;

[0278] A first mask addition module 1020 that adds masks to the text element sequence according to a set rule;

[0279] A first model input module 1030 that inputs the text element sequence with masks into the transformer network of the language model, encodes each text element in the text element sequence into a vector with context information through the transformer network, and outputs a representation vector;

[0280] A training module 1040 that obtains the predicted value of the mask using the representation vector and trains the transformer network according to the cross-loss function between the predicted value and the true value to make it converge and obtain a language model.

[0281] According to the exemplary embodiment of the present disclosure, the first mask addition module 1020 is specifically configured to:

[0282] Add masks to both the source segment sequence and the target segment sequence, and use the obtained language model as a pre-trained language model.

[0283] According to the exemplary embodiment of the present disclosure, the first acquisition module 1010 is specifically configured to:

[0284] Obtain each text element that constitutes the text element sequence;

[0285] Perform text element encoding, position encoding, and segment encoding on each text element;

[0286] Construct a text element sequence from the encoded text elements.

[0287] According to an exemplary embodiment of the present disclosure, the first mask addition module 1020 is specifically configured to:

[0288] Set the text elements in the text element sequence to be invisible or perform text element replacement.

[0289] According to an exemplary embodiment of the present disclosure, compared with Figure 10 compared with Figure 11 the language model training device 1100 shown further includes:

[0290] A first mask matrix generation module 1110, which generates a mask matrix applicable to the task according to the sequence start marker and the sequence end marker, and adds the mask matrix to the transformer network;

[0291] The first model input module 1120 is specifically configured to:

[0292] Use the transformer network with the mask matrix to encode the output vector of the adjacent upper layer at each layer, and generate the output vector of each layer.

[0293] According to an exemplary embodiment of the present disclosure, the first model input module 1120 is specifically further configured to:

[0294] In the transformer network, use the mask matrix to determine the visible range of the text elements described by the output vector of the adjacent upper layer;

[0295] Use multiple self-attention heads at each layer to aggregate the output vectors of the adjacent upper layer within the visible range, and output the output vector with context information at each layer.

[0296] According to an exemplary embodiment of the present disclosure, the first mask matrix generation module 1110 is specifically configured to:

[0297] Generate a mask matrix applicable to at least one of the seq2seq model, unidirectional language model, or bidirectional language model according to the sequence start marker and the sequence end marker.

[0298] According to an exemplary embodiment of the present disclosure, in the case where the task is a seq2seq model, the first mask matrix generation module 1110 is specifically further configured to:

[0299] According to the sequence start marker and the sequence end marker, set the source segment sequence to be bidirectionally visible and the target segment sequence to be forward visible according to the set rules, and generate a mask matrix applicable to the seq2seq model task.

[0300] Next, refer to Figure 12 to describe the copywriting generation device 1200 of the exemplary embodiment of the present disclosure.

[0301] Refer to Figure 12 As shown, the copywriting generation device 1200 of the exemplary embodiment of the present disclosure may include:

[0302] A second acquisition module 1210, which acquires an input text element sequence, where the input text element sequence includes a source segment sequence starting with a sequence start marker and ending with a first sequence end marker, and a second sequence end marker for predicting the termination of the target segment sequence;

[0303] A second mask addition module 1220, which inserts a mask for predicting the target segment text element between the first sequence end marker and the second sequence end marker according to the set rules;

[0304] A second model input module 1230, which inputs the masked input text element sequence into a copywriting generation model trained by the language model training method according to any exemplary embodiment of the present disclosure, encodes each text element in the input text element sequence into a vector with context information through a transformer network in the copywriting generation model, and outputs a representation vector;

[0305] A prediction module 1240, which uses the representation vector to predict the target segment text element for the mask until it terminates at the second sequence end marker, and uses the target segment text element to generate a target segment sequence, and outputs a copywriting composed of the source segment sequence and the target segment sequence.

[0306] According to the exemplary embodiment of the present disclosure, the second acquisition module 1210 is specifically configured to:

[0307] Acquire each text element that makes up the input text element sequence;

[0308] Perform text element encoding, position encoding, and segment encoding on each text element;

[0309] Construct the encoded text elements into an input text element sequence.

[0310] According to the exemplary embodiment of the present disclosure, the prediction module 1240 is specifically configured to:

[0311] Predict the target segment text element according to the representation vector to obtain the probabilities of multiple candidate text elements;

[0312] Select the target text elements with the top target number of probabilities among multiple candidate text elements using beam search.

[0313] According to an exemplary embodiment of the present disclosure, compared with Figure 12 compared with Figure 13 the text generation device 1300 shown further includes:

[0314] A second mask matrix generation module 1310, which generates a mask matrix according to the sequence start marker, the first sequence end marker, and the second sequence end marker, and adds the mask matrix to the transformer network;

[0315] The second model input module 1320 is specifically configured to:

[0316] Use the transformer network with the mask matrix to encode the output vectors of the adjacent upper layer at each layer, and generate the output vectors of each layer.

[0317] According to an exemplary embodiment of the present disclosure, the second model input module 1320 is specifically further configured to:

[0318] In the transformer network, use the mask matrix to determine the visible range of the text elements described by the output vectors of the adjacent upper layer;

[0319] Use multiple self-attention heads at each layer to aggregate the output vectors of the adjacent upper layer within the visible range, and output the output vectors with context information at each layer.

[0320] According to an exemplary embodiment of the present disclosure, compared with Figure 13 compared with Figure 14 the text generation device 1400 shown further includes:

[0321] A receiving module 1410, which receives user input before generating a mask matrix according to the sequence start marker, the first sequence end marker, and the second sequence end marker;

[0322] A response module 1420, in response to user input, obtains the length of the target segment sequence, and determines the second sequence end marker representing the end of the target segment sequence according to the length of the target segment sequence.

[0323] Since each functional module of the language model training device and the text generation device of the embodiments of the present disclosure is the same as the publicly disclosed embodiments of the above language model training method and text generation method, they will not be elaborated here.

[0324] Exemplary Storage Medium

[0325] After introducing the language model training method and text generation method of the exemplary embodiments of the present disclosure, as well as the language model training device and text generation device, next, refer to Figure 15A storage medium according to an exemplary embodiment of the present disclosure will be described.

[0326] Referring Figure 15 As shown, a program product 1500 for implementing the above method according to an embodiment of the present disclosure is described. It may be a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a device such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0327] The program product may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but not be limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0328] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which the readable program code is carried. Such a propagated data signal may take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable signal medium may also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0329] The program code contained on the readable medium may be transmitted by any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0330] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., by connecting through the Internet using an Internet service provider).

[0331] Exemplary Electronic Device

[0332] After introducing the storage medium of the exemplary embodiments of the present disclosure, next, the electronic device of the exemplary embodiments of the present disclosure will be described with reference to FIG. 16.

[0333] Figure 16 The displayed electronic device 1600 is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0334] As Figure 16 shown, the electronic device 1600 is presented in the form of a general-purpose computing device. The components of the electronic device 1600 may include, but are not limited to: at least one of the above-mentioned processing units 1610, at least one of the above-mentioned storage units 1620, a bus 1630 connecting different system components (including the storage unit 1620 and the processing unit 1610), and a display unit 1640.

[0335] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 1610, so that the processing unit 1610 executes the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification. For example, the processing unit 1610 may execute the steps as Figure 1 or Figure 7 shown in.

[0336] The storage unit 1620 may include a volatile storage unit, such as a random access storage unit (RAM) 1621 and / or a cache storage unit 1622, and may further include a read-only storage unit (ROM) 1623.

[0337] The storage unit 1620 may also include a program / utilities 1624 having a set (at least one) of program modules 1625. Such program modules 1625 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0338] The bus 1630 may include a data bus, an address bus, and a control bus.

[0339] The electronic device 1600 may also communicate with one or more external devices 1700 (such as a keyboard, a pointing device, a Bluetooth device, etc.). Such communication may be performed through an input / output (I / O) interface 1650. The electronic device 1600 further includes a display unit 1640, which is connected to the input / output (I / O) interface 1650 for display. Additionally, the electronic device 1600 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1660. As shown in the figure, the network adapter 1660 communicates with other modules of the electronic device 1600 through the bus 1630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0340] It should be noted that although several modules or sub-modules of the language model training device and the copywriting generation device are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of the two or more units / modules described above may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided and embodied by multiple units / modules.

[0341] In addition, although the operations of the method of the present disclosure are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0342] Although the spirit and principles of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the specific embodiments disclosed, and the division of each aspect does not mean that the features in these aspects cannot be combined for benefits. This division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for training a language model, characterized in that, Including: Obtain a sequence of text elements that make up a training text, where the sequence of text elements is divided into a source segment sequence and a target segment sequence that are concatenated together. The sequence of text elements starts with a sequence start marker, and both the source segment sequence and the target segment sequence end with a sequence end marker; Generate a mask matrix applicable to at least one task among a seq2seq model, a unidirectional language model, or a bidirectional language model based on the sequence start marker and the sequence end marker, and add the mask matrix to the multi-layer Transformer network of the language model; the multi-layer Transformer network of the language model is composed of multiple Transformer blocks connected in sequence, and the output of the previous Transformer block in two adjacent Transformer blocks is used as the input of the current Transformer block; Add a mask to the sequence of text elements according to a set rule; Input the sequence of text elements with the mask into the multi-layer Transformer network of the language model, and encode each text element in the sequence of text elements into a vector with context information through the multi-layer Transformer network with the mask matrix, and output a representation vector; Obtain a predicted value of the mask using the representation vector, and train the multi-layer Transformer network according to a cross-loss function between the predicted value and the true value until it converges to obtain the language model.

2. The language model training method according to claim 1, wherein The encoding of each text element in the sequence of text elements into a vector with context information through the multi-layer Transformer network with the mask matrix includes: Using the multi-layer Transformer network with the mask matrix to encode the output vector of the adjacent previous layer at each layer to generate the output vector of each layer.

3. The language model training method according to claim 2, wherein The using of the multi-layer Transformer network with the mask matrix to encode the output vector of the adjacent previous layer at each layer to generate the output vector of each layer includes: In the multi-layer Transformer network, use the mask matrix to determine the visible range of the text elements described by the output vector of the adjacent previous layer; Use multiple self-attention heads at each layer to aggregate the output vector of the adjacent previous layer within the visible range and output the output vector with context information of each layer.

4. The language model training method according to claim 1, characterized in that, In the case where the task is a seq2seq model, generating a mask matrix applicable to the task according to the sequence start marker and the sequence end marker includes: According to the sequence start marker and the sequence end marker, set the source segment sequence to be bidirectionally visible and the target segment sequence to be forward visible according to a set rule to generate a mask matrix applicable to the seq2seq model task.

5. The language model training method according to claim 1, wherein Adding a mask to the sequence of text elements according to a set rule includes: Adding masks to both the source segment sequence and the target segment sequence, and using the obtained language model as a pre-trained language model.

6. The language model training method according to claim 1, wherein Obtaining a sequence of text elements that make up a training text includes: Obtain each text element that makes up the sequence of text elements; Perform text element encoding, position encoding, and segment encoding on each text element; Form the sequence of text elements with the encoded text elements.

7. The language model training method according to claim 1, wherein Adding a mask to the sequence of text elements according to a set rule includes: Set the text elements in the text element sequence to invisible or perform text element replacement.

8. A copywriting generation method, characterized in that, Including: Obtain an input text element sequence, where the input text element sequence includes a source segment sequence starting with a sequence start marker and ending with a first sequence end marker, and a second sequence end marker for terminating the prediction of the target segment sequence; Generate a mask matrix based on the sequence start marker, the first sequence end marker, and the second sequence end marker, and add the mask matrix to the multi-layer Transformer network in the text generation model; the text generation model includes: a language model trained according to the language model training method described in any one of claims 1-7; the multi-layer Transformer network is formed by sequentially connecting multiple Transformer blocks, and the output of the previous Transformer block in two adjacent Transformer blocks is used as the input of the current Transformer block; Insert a mask for predicting the target segment text elements between the first sequence end marker and the second sequence end marker according to a set rule; Input the input text element sequence with the mask into the text generation model, and encode each text element in the input text element sequence into a vector with context information through the multi-layer Transformer network in the text generation model with the mask matrix, and output a representation vector; Use the representation vector to predict the target segment text elements for the mask until termination at the second sequence end marker, and use the target segment text elements to generate a target segment sequence, and output a text composed of the source segment sequence and the target segment sequence.

9. The copywriting generation method according to claim 8, wherein, The encoding of each text element in the input text element sequence into a vector with context information through the multi-layer Transformer network in the text generation model with the mask matrix includes: Use the multi-layer Transformer network with the mask matrix to encode the output vector of the adjacent upper layer at each layer to generate the output vector of each layer.

10. The copywriting generation method according to claim 9, characterized in that, The encoding of the output vector of the adjacent upper layer at the target layer by using the multi-layer Transformer network with the mask matrix to generate the output vector of the target layer includes: In the multi-layer Transformer network, use the mask matrix to determine the visible range of the text elements described by the output vector of the adjacent upper layer; Use multiple self-attention heads at each layer to aggregate the output vector of the adjacent upper layer within the visible range, and output the output vector with context information at each layer.

11. The copywriting generation method according to claim 9, wherein Before generating the mask matrix based on the sequence start marker, the first sequence end marker, and the second sequence end marker, the method further includes: Receive user input; In response to the user input, obtain the length of the target segment sequence, and determine the second sequence end marker representing the end of the target segment sequence according to the length of the target segment sequence.

12. The copywriting generation method according to claim 8, characterized in that Using the representation vector to predict the target segment text elements for the mask includes: Predict the target segment text elements according to the representation vector to obtain the probabilities of multiple candidate text elements; Use beam search to select the target number of target text elements with the top probabilities among the multiple candidate text elements.

13. The copywriting generation method according to claim 8, characterized in that, The obtaining of the input text element sequence includes: Obtain each text element that constitutes the sequence of input text elements; Perform text element encoding, position encoding, and segment encoding on each of the text elements; Form the sequence of input text elements with the encoded text elements.

14. A language model training device, characterized in that, Includes: A first acquisition module, configured to obtain a sequence of text elements that constitutes the training text, where the sequence of text elements is divided into a source segment sequence and a target segment sequence spliced together, the sequence of text elements starts with a sequence start marker, and both the source segment sequence and the target segment sequence end with a sequence end marker; A first mask matrix generation module, configured to generate a mask matrix applicable to at least one task among a seq2seq model, a unidirectional language model, or a bidirectional language model according to the sequence start marker and the sequence end marker, and add the mask matrix to the multi-layer transformer network of the language model; the multi-layer transformer network of the language model is formed by sequentially connecting multiple transformer blocks, and the output of the previous transformer block in two adjacent layers of transformer blocks is used as the input of the current layer of transformer block; A first mask addition module, configured to add a mask to the sequence of text elements according to a set rule; A first model input module, configured to input the sequence of text elements with the mask into the multi-layer transformer network of the language model, encode each text element in the sequence of text elements into a vector with context information through the multi-layer transformer network with the mask matrix, and output a representation vector; A training module, configured to obtain a predicted value of the mask by using the representation vector, and train the multi-layer transformer network according to the cross-loss function between the predicted value and the true value to make it converge, so as to obtain the language model.

15. The language model training device according to claim 14, characterized in that, The first model input module is specifically configured to: Use the multi-layer transformer network with the mask matrix to encode the output vector of the adjacent upper layer at each layer to generate the output vector of each layer.

16. The language model training device according to claim 15, wherein The first model input module is specifically further configured to: In the multi-layer transformer network, use the mask matrix to determine the visible range of the text element described by the output vector of the adjacent upper layer; Aggregate the output vectors of the adjacent upper layer within the visible range by using multiple self-attention heads of each layer, and output the output vector with context information of each layer.

17. The language model training device according to claim 14, wherein In the case where the task is a seq2seq model, the first mask matrix generation module is specifically further configured to: According to the sequence start marker and the sequence end marker, set the source segment sequence to be bidirectionally visible and the target segment sequence to be forward visible according to a set rule, and generate a mask matrix applicable to the seq2seq model task.

18. The language model training device according to claim 14, characterized in that, The first mask addition module is specifically configured to: Add masks to both the source segment sequence and the target segment sequence, and use the obtained language model as a pre-trained language model.

19. The language model training device according to claim 14, wherein The first acquisition module is specifically configured to: Obtain each text element that constitutes the sequence of text elements; Perform text element encoding, position encoding, and segment encoding on each of the text elements; Form the sequence of text elements with the encoded text elements.

20. The language model training device according to claim 14, wherein The first mask addition module is specifically configured to: Set the text elements in the text element sequence to be invisible or perform text element replacement.

21. A copywriting generation device, characterized in that, Including: A second acquisition module, configured to acquire an input text element sequence, where the input text element sequence includes A source segment sequence starting with a sequence start marker and ending with a first sequence end marker, and a second sequence end marker for terminating the prediction of the target segment sequence; A second mask matrix generation module, configured to generate a mask matrix according to the sequence start marker, the first sequence end marker, and the second sequence end marker, and add the mask matrix to the multi-layer transformer network in the text generation model; The text generation model includes: a language model trained by the language model training method according to any one of claims 1-7; the multi-layer transformer network is formed by sequentially connecting multiple transformer blocks, and the output of the previous transformer block in two adjacent layers of transformer blocks is used as the input of the current layer of transformer blocks; A second mask addition module, configured to insert a mask for predicting target segment text elements between the first sequence end marker and the second sequence end marker according to a set rule; A second model input module, configured to input the input text element sequence with the mask into the text generation model, encode each text element in the input text element sequence into a vector with context information through the multi-layer transformer network in the text generation model with the mask matrix, and output a representation vector; A prediction module, configured to use the representation vector to predict the target segment text elements for the mask until it terminates at the second sequence end marker, and use the target segment text elements to generate a target segment sequence, and output a text composed of the source segment sequence and the target segment sequence.

22. The copywriting generation device according to claim 21, characterized in that The second model input module is specifically configured to: Use the multi-layer transformer network with the mask matrix to encode the output vector of the adjacent upper layer at each layer to generate the output vector of each layer.

23. The copywriting generation device according to claim 22, wherein The second model input module is specifically further configured to: In the multi-layer transformer network, use the mask matrix to determine the visible range of the text elements described by the output vector of the adjacent upper layer; Use multiple self-attention heads at each layer to aggregate the output vectors of the adjacent upper layer within the visible range, and output the output vector with context information of each layer.

24. The copywriting generation device according to claim 22, wherein The device further includes: A receiving module, configured to receive user input before generating a mask matrix according to the sequence start marker, the first sequence end marker, and the second sequence end marker; A response module, configured to, in response to the user input, obtain the length of the target segment sequence, and determine a second sequence end marker representing the end of the target segment sequence according to the length of the target segment sequence.

25. The copywriting generation device according to claim 21, wherein The prediction module is specifically configured to: Predict target segment text elements according to the representation vector to obtain the probabilities of multiple candidate text elements; Use beam search to select the target number of target text elements with the top probabilities among the multiple candidate text elements.

26. The copywriting generation device according to claim 21, wherein The second acquisition module is specifically configured to: Acquire each text element constituting the input text element sequence; Perform text element encoding, position encoding, and segment encoding on the each text element; The encoded text elements form the input text element sequence.

27. A storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by a processor, it implements: The language model training method according to any one of claims 1 to 7; and / or The copywriting generation method according to any one of claims 8 to 13.

28. An electronic device, characterized in that, Comprising: A processor; And A memory for storing executable instructions of the processor; Wherein, the processor is configured to execute by executing the executable instructions: The language model training method according to any one of claims 1 to 7; and / or The copywriting generation method according to any one of claims 8 to 13.

Images