Method, apparatus and computer program for training machine learning model to generate text and for generating text using trained machine learning model

Abstract

Methods, apparatuses and computer programs for training a machine learning model to generate text and for generating text using the trained machine learning model are disclosed. The present disclosure generally relates to a computer-implemented method for training a machine learning model for text generation, the method comprising: inputting text into the machine learning model; preprocessing the input text to obtain a plurality of character vector representations; encoding each of the plurality of character vector representations using an encoder to obtain a plurality of word vector representations; generating a plurality of predicted word vector representations based on the plurality of word vector representations using a trunk model; decoding the plurality of predicted word vector representations using a decoder to obtain a plurality of character probabilities; and updating the machine learning model based on the plurality of character probabilities. The disclosure also relates to a computer-implemented method for generating text, a corresponding device, a system and a computer program.

Description

Technical Field

[0001] This disclosure relates to a computer-implemented method for training a machine learning model to generate text, and to a computer-implemented method for generating text using a trained machine learning model, as well as corresponding computer programs, devices, and systems. Background Technology

[0002] In the context of natural language processing, tokenization describes the process of breaking down text into smaller segments (i.e., tokens). These tokens serve as the basis for further processing steps, such as embeddings. Therefore, tokenization divides text into meaningful components that are then processed further. The resulting tokens may differ in granularity. Two basic tokenization methods are character-level tokenization and word-level tokenization.

[0003] During character-level tokenization, text is broken down into individual characters, rather than words or sentences. Each individual character is then treated as a separate token. Therefore, the input text "Hello" will produce a total of six tokens: "h", "e", "l", "l", and "o". One advantage of character-level tokenization is the size of the vocabulary. Since each character is a token, the total size of the vocabulary is limited by the number of characters. This type of tokenization is also able to handle previously unseen words because it focuses on the character level, not words. However, character-level tokenization results in very long sequences, which increases the computational complexity of both training and inference processes. For example, the word "nature" will result in a sequence of six different tokens, instead of a single token as in word-character tokenization. Since the computational complexity of machine learning models processing natural language is based on the length of the sequence, character-level tokenization increases the computational cost. Finally, character-level tokenization also makes it difficult to capture long-distance dependencies, such as a word being related to a word in another sentence.

[0004] In contrast, during word-level tokenization, text is broken down into words, rather than individual characters or entire sentences. Each word is then treated as a single token. For example, the sentence "Hello world" will produce the tokens "Hello" and "world". Word-level tokenization is better at preserving semantic meaning compared to character-level tokenization. Word-level tokenization also produces smaller sequences and is generally more computationally efficient than character-level tokenization. However, word-level tokenization suffers in performance when words are misspelled. Furthermore, words that were not present during training are treated as unknown when using word-level tokenization. Therefore, the trained model is extremely sensitive to the corpus used for training. Word-level tokenization also struggles to accurately handle word morphologies such as "run", "running", and "ran". Since these morphologies are different words, they receive different tokens. However, they are likely to convey similar meanings.

[0005] Subtotaling aims to combine character-level and word-level tokenization by breaking text down into smaller units that are larger than characters but smaller than words. Each subtotal is then treated as a token. For example, the sentence "Hello world" could produce the tokens "Hel", "lo", and "world". Subtotaling methods aim to balance the efficiency of word-level methods with the flexibility of character-level methods. However, subtotaling still suffers from several drawbacks, including a lack of adaptability to new domains or languages, sensitivity to spelling errors and variations, and a large vocabulary compared to character-level methods.

[0006] Given these drawbacks, currently known tokenization methods may not always lead to the desired results. In this context, the purpose of this disclosure is to address one or more of the drawbacks mentioned above. Summary of the Invention

[0007] The above-mentioned and other objectives, which will become apparent from the following description, are addressed by the subject matter of the independent claims. Preferred embodiments are the subject matter of the dependent claims.

[0008] The first embodiment of this disclosure relates to a computer-implemented method for training a machine learning model for text generation, the method comprising: inputting text into the machine learning model; preprocessing the input text to obtain a plurality of character vector representations; encoding each of the plurality of character vector representations using an encoder to obtain a plurality of word vector representations; generating a plurality of predicted word vector representations based on the plurality of word vector representations using a backbone model; decoding the plurality of predicted word vector representations using a decoder to obtain a plurality of character probabilities; and updating the machine learning model based on the plurality of character probabilities.

[0009] Preprocessing the input text to obtain multiple character vector representations has the advantage of preparing the input text for further processing during subsequent steps. Because the input text is preprocessed to obtain character-level vector representations, the size of the resulting vocabulary may be limited to the number of possible characters. A smaller vocabulary size may require less memory and thus save computational resources. Character-level initial processing also provides a greater degree of flexibility.

[0010] Encoding each of multiple character vector representations using an encoder to obtain multiple word vector representations has the advantage of converting character-level input into word-level output. In other words, the encoder's input is based on character-level vector representations (i.e., one vector per character), and the decoder's output is based on word-level vector representations (i.e., one vector per word). This character-to-word conversion can save computational resources during further processing of the input text.

[0011] Using a backbone model to generate multiple predicted word vector representations based on multiple word vector representations has the advantage of performing the most computationally expensive part of the model on the word-level representation of the input text. More specifically, in word-level representation, a single word vector represents the entire word. In contrast, in character-level representation, a single word requires x character vectors, where x is the number of characters that make up the word. For example, using word-level representation, the word "nature" requires one word vector but six character vectors, one vector for each character in the word. Therefore, performing backbone computation at the word level can have the advantage of being less computationally complex and thus saving computational resources.

[0012] Decoding multiple predicted word vector representations using a decoder to obtain multiple character probabilities allows for character-level final processing. This can have the advantage of providing more flexible outputs and thus improving the model's performance with respect to output quality. Updating machine learning models based on multiple character probabilities can have the advantages of text-based improvement models.

[0013] Finally, performing input (i.e., initial preprocessing of the input text) and output (i.e., final processing of the output text) at the character level, while performing backbone computation (i.e., the computationally expensive part of the model) at the word level, combines the advantages of character-level and word-level methods. More specifically, the combined approach can benefit from the flexibility of character-level methods and the reduced computational resource requirements of word-level methods.

[0014] According to the second embodiment, the steps of input, preprocessing, encoding, generation, decoding and updating are repeated iteratively.

[0015] Iteratively repeating the steps of input, preprocessing, encoding, generation, decoding, and updating can have the advantage of gradually improving the performance of a machine learning model. This iterative approach also allows machine learning models to be trained on large training datasets. More specifically, large datasets may be so large that the entire training dataset cannot fit into memory. Iterative methods can be used to iteratively load portions of the training dataset into memory and train the model on those loaded portions. Another advantage of iterative methods is that data that becomes available after the initial training of the machine learning model can still be incorporated by performing additional training iterations. Iterative methods also enable performance monitoring. For example, the model's performance can be tested after a predefined number of iterations. If the model does not perform as expected, adjustments can be made to the model before further training.

[0016] According to the third embodiment, the preprocessing includes splitting the input text into multiple character sequences, where each character sequence represents a word; and embedding each character in the multiple character sequences to obtain multiple character vector representations.

[0017] Splitting the input text into multiple character sequences, where each sequence represents a word, offers the advantage of preprocessing the input text at the character level. Embedding each character from multiple character sequences to obtain multiple character vector representations offers the advantage of obtaining a vector representation for each character of the input text.

[0018] According to the fourth embodiment, the preprocessing includes prepending special characters to each character sequence before embedding.

[0019] Prepending special characters to each character sequence before embedding has the advantage of introducing special characters that can later be used to represent the character sequence to which they were prepended. Prepending special characters before embedding ensures that they are accurately incorporated into the training process.

[0020] According to the fifth embodiment, the encoder is a natural language processing model, wherein preferably, the architecture of the first natural language processing model is based on a converter model with a decoder-only variant, and most preferably, the attention mechanism of the converter model is bidirectional.

[0021] An encoder is a first-order natural language processing machine learning model that provides the ability to process a sequence as input and return a sequence as output. The architecture of this first-order natural language processing model is preferably based on a converter model with only decoder variants, which can leverage the performance advantages of the corresponding architecture. The attention mechanism of the converter model is preferably bidirectional, which improves the model's ability to understand the context of the input text. More specifically, the bidirectional attention mechanism considers both preceding and following words simultaneously, enhancing contextual awareness.

[0022] According to the sixth embodiment, the backbone model is a second natural language processing model, wherein preferably, the architecture of the second natural language processing model is based on a converter model with only decoder variants, and most preferably, the attention mechanism of the converter model is causal.

[0023] The backbone model is a second natural language processing machine learning model, wherein preferably, the architecture of the natural language processing model is based on a converter model with only a decoder variant, which can have the same advantages as mentioned with respect to Example 5. The attention mechanism of the converter model is causal, enabling the model to generate text autoregressively. In other words, the causal attention mechanism enables the backbone model to generate outputs, where each output is based on a previously generated output.

[0024] According to the seventh embodiment, prior to the decoding step, each of the plurality of predicted word vector representations is concatenated with its corresponding character vector representation.

[0025] Concatenating each of the multiple predicted word vector representations with its corresponding character vector representation before the decoding step can have the advantage of improving the information upon which the decoding step is based. This is particularly likely because the predicted word vector representation represents the word predicted as the next word by the backbone model, and the corresponding character vector representation represents the actual next word. The combination of word-level representations (i.e., the word vectors of the predicted next word) and character-level representations (i.e., the character vectors of the actual next word) can further improve the information upon which the decoding step is based.

[0026] According to the eighth embodiment, the decoder is a third natural language processing model, wherein preferably, the architecture of the third natural language processing model is based on a converter model with only decoder variants, and most preferably, the attention mechanism of the converter model is causal.

[0027] The backbone model is a third natural language processing model, preferably an architecture based on a converter model with a decoder variant, which can have the same advantages as mentioned with respect to Example 5. The attention mechanism of the converter model is preferably causal and can have the same advantages as discussed in Example 6.

[0028] According to embodiment 9, updating the machine learning model includes updating one or more of the following adjustable parameters: the embedding matrix, encoder, backbone model, and / or decoder used during the preprocessing step.

[0029] Updating one or more adjustable parameters of one or more of the embedding matrix, encoder, backbone model, and / or decoder used during the preprocessing step enables training of the machine learning model. This can have the advantage of improving the performance of the machine learning model. It also provides the flexibility to train some components of the machine learning model while keeping other components of the machine learning model fixed.

[0030] The tenth embodiment of this disclosure relates to a computer-implemented method for generating text using a machine learning model trained according to any of the foregoing embodiments, the method comprising: inputting text into a trained machine learning model; and generating text based on the input text using the trained machine learning model.

[0031] A computer-implemented method for generating text using a machine learning model trained according to any of the foregoing embodiments can have the advantage of reducing computational complexity while maintaining the performance of the machine learning model. As mentioned with respect to the previous embodiments, the architecture of the trained machine learning model can provide computational efficiency at the word level while maintaining the flexibility of character-level processing.

[0032] According to the 11th embodiment, generating text includes generating characters based on multiple character probabilities; updating the input of the decoder based on the generated characters, or updating the input of the backbone model based on one or more generated characters; and iteratively repeating the generation and updating.

[0033] Character-level text generation is achieved by generating characters based on probabilistic multiple characters. In other words, instead of predicting the entire next word, a machine learning model can predict each character of the next word individually. This can improve prediction results. Character-level prediction also has the advantage of making predictions more flexible. Updating the decoder input based on the generated characters, or updating the backbone model input based on one or more generated characters, allows for the consideration of generated characters during the generation of subsequent characters. Iteratively repeating generation and updating can further improve the generated text.

[0034] According to the 12th embodiment, updating the decoder input includes determining that the generated character is not a special character; updating the decoder input based on the character vector representation of the generated character; and decoding the updated input to obtain multiple character probabilities.

[0035] Updating the decoder's input includes determining that the generated character is not a special character, enabling switching between character-level and word-level prediction. Determining that the generated character is not a special character signals the prediction of the next character. Updating the decoder's input based on the character vector representation of the generated character provides the advantage of considering the generated character when generating the next character. This can improve prediction results. Decoding the updated input to obtain multiple character probabilities provides flexibility during text generation by predicting the next word on a single character basis rather than at the word level.

[0036] According to the 13th embodiment, updating the input of the backbone model includes: determining that the generated character is a special character; prepending the special character to one or more generated characters to obtain a predicted character sequence; embedding each character of the predicted character sequence to obtain multiple predicted character vector representations; encoding the predicted character vector representations using an encoder to obtain predicted word vector representations; updating the input of the backbone model based on the predicted word vector representations; generating predicted word vector representations based on the updated input; and decoding the predicted word vector representations to obtain multiple character probabilities.

[0037] Updating the input to the backbone model involves determining that the generated characters are special characters, which offers the advantage of combining character-level prediction with word-level prediction. In other words, determining that a generated character is a special character can trigger word-level prediction. The word vectors of the generated words (i.e., one or more generated characters) and the character vectors of the generated words can then be used to update the input to the backbone model. Thus, the flexibility of character-level prediction is combined with the efficiency of word-level prediction performed by the backbone model. Pre-processing special characters onto one or more generated characters to obtain a predicted character sequence, embedding each character of the predicted character sequence to obtain multiple predicted character vector representations, and encoding the predicted character vector representations using an encoder to obtain the predicted word vector representations can have the advantages of preprocessing and encoding one or more generated characters. This can improve subsequent processing, especially the predictions performed by the backbone model. Updating the input to the backbone model with the predicted word vector representations can have the same advantages mentioned above regarding the efficiency of leveraging word-level prediction. In other words, performing backbone computation on word-level representations may be less computationally complex and therefore saves computational resources.

[0038] Generating predicted word vector representations based on updated input can have the advantage of predicting the next word with the efficiency of using a backbone model. As discussed earlier, word-level processing may be more efficient, among other things, due to the shorter sequence length representing the input. Decoding the predicted word vector representations to obtain multiple character probabilities can provide flexibility during text generation by predicting the next word on a character-by-character basis rather than at the word level.

[0039] The 14th embodiment of this disclosure relates to an apparatus or system including means for performing the method according to any one of embodiments 1 to 13.

[0040] Devices or systems that include means for performing the method according to any one of embodiments 1 to 13 may have all the advantages mentioned with respect to the corresponding embodiments.

[0041] The 15th embodiment of this disclosure relates to a computer program, including instructions that, when executed by a computer, cause the computer to perform the method described in any one of embodiments 1 to 13.

[0042] A computer program, including instructions, which, when executed by a computer, cause the computer to perform the method described in any one of embodiments 1 to 13, may have all the advantages mentioned with respect to the corresponding embodiments. Attached Figure Description

[0043] The relevant aspects of the invention are described in more detail below with reference to the accompanying drawings, but the invention is not limited to the embodiments shown in these drawings.

[0044] Figure 1 An exemplary training process according to an embodiment of this disclosure is illustrated;

[0045] Figure 2 An exemplary reasoning process according to embodiments of this disclosure is illustrated;

[0046] Figure 3 An exemplary reasoning process according to embodiments of this disclosure is illustrated;

[0047] Figure 4 A flowchart depicting a computer-implemented method for training a machine learning model for text generation according to embodiments of the present disclosure is further shown; and

[0048] Figure 5 An example computing device is shown that can be used to implement the features described herein in some embodiments. Detailed Implementation

[0049] The invention is described in more detail below with reference to the accompanying drawings. However, the invention can also be used in other embodiments not explicitly disclosed below. As detailed below, the embodiments are compatible with each other, and a corresponding feature of one embodiment can be applied to another. The drawings do not limit the scope of the claims but only support an understanding of the invention.

[0050] Figure 1 An exemplary training process 100 for a machine learning model according to an embodiment of the present disclosure is shown.

[0051] The training process can generally be divided into three stages: encoding, backbone, and decoding. For example... Figure 1 As shown, the encoding and decoding stages process the input text at the byte level (i.e., character level). In contrast, the backbone stage processes the input text at the word level. The core of each stage can include a natural language processing machine learning model that processes the input text.

[0052] The training process 100 can be based on one or more text corpora. For illustrative purposes, Figure 1 This study focuses on processing a text segment (i.e., "Hello World, my Name") that may have already appeared in a text corpus. "HelloWorld, my Name" is considered input text 101. In the preprocessing step, the input text can be split into words. Word splitting 110 can be performed using fixed splitting rules. More specifically, the text can be split at whitespace. Additionally, whitespace can be added to preceding words. Note that a special marker, here indicated as [W], can be prepended to each word. This special marker can indicate the beginning of each word. For input text 101, this word splitting and prefixing may produce the character sequences 110a: "[W]Hello_", "[W]World,_", "[W]my_", and "[W]Name_". The input text can be referred to as... ,in In other words, input text. It can include one or more characters represented in binary format. Split the input text into a sequence of words. It can also be represented as ,in In this representation, special markers have been prepended to each word in word sequence 110a.

[0053] The resulting character sequence 110a is used as input to embedding step 120, where each character of each sequence is converted into a vector 120a. Embedding step 120 can be implemented using an embedding matrix. The embedding matrix can be a matrix in which each row corresponds to a vector representation (i.e., embedding 120a) of a word's tag (e.g., a character). During embedding step 120, the embedding matrix can be used to find the vector representation 120a for each character and replace each character with its corresponding vector representation 120a.

[0054] The subsequent encoding step 130 uses a natural language processing model to encode each vector representation 120a. The architecture of the natural language processing model can be based on a decoder-only variant of the converter model. The attention mechanism of the decoder-only variant of the converter model can be bidirectional. Figure 1 As shown, encoding step 130 can return an encoded vector representation 130a for each character. However, only one encoded vector representation for each word can be used for further processing, and the other encoded vector representations are discarded. In this way, the machine learning model can learn to represent a word in a single vector (i.e., the vector that was not discarded but is further processed).

[0055] Since discarding information may alter its dimensionality, a linear mapping step 140 may be necessary to merge the remaining encoded word vector representations. The result of the encoding phase can be a dense representation of the input text in the form of an encoded vector representation 140a for each word. Figure 1 As shown, It could be the encoded vector representation of the word "[W]Hello_". It could be the encoded vector representation of the word "[W]World_". It could be the encoded vector representation of the word "[W]my_", and It can be the encoded vector representation of the word "[W]Name_". Therefore, the encoding stage can be used to convert character-level input into word-level output.

[0056] This word-level input text can be used as input text for the subsequent backbone stage. The backbone stage may include a natural language processing model 150, which uses word vector representations (i.e., )140a to predict the corresponding subsequent word vector representation (i.e. (150a) The architecture of a natural language machine learning model can be based on a converter model with only decoder variants. The attention mechanism of a converter model with only decoder variants may be causal. Models that include non-causal attention mechanisms can describe models that generate predictions based on prior information. For example, when given the string "Hello World, my Name" and predicting the word "World", a causal model only considers the word "Hello". If the model is not causal, it may also consider the words "my" and "Name" to predict the word "World".

[0057] The decoding phase can begin with a further linear transformation 160. This linear transformation can again be used to adjust the dimensionality of the input information. More specifically, before entering the decoder 170, the input information is adjusted by concatenating the word vector representation (i.e., word-level representation) 140a of the predicted next word with a sequence of character vector representations (i.e., character-level representations) 160b of the actual next word. For example, character vector representations 160b “W”, “o”, “r”, “l”, “d”, “,”, and “_” are concatenated to the predicted word vector representation 160a. The character vectors representing "m", "y", and "_" are concatenated to the predicted word vectors. And so on. Note that the characters of the actual next 160b word can be embedded using an embedding matrix that may have been used in the initial embedding step 120.

[0058] Character-level input information can be used as input to decoder 170. Decoder 170 can be a natural language processing model. The architecture of the natural language processing machine learning model can be based on a converter model with only decoder variants. The attention mechanism of the converter model with only decoder variants may be causal. Based on the character-level input information, decoder 170 can return multiple character probability vectors 170a, where each position in the character probability vector can describe the probability that a particular character will become the next character in the text. Note that character logit, such as... Figure 1 The processing method mentioned is for character probability 170a, which can be further improved.

[0059] Finally, the cross-entropy loss function can be used to compare the predictions of the machine learning model with the actual values ​​171a. The trainable parameters of the machine learning model can be updated based on the comparison results. This may include updating the parameters of the embedding matrix in the encoding step 120, the embedding matrix in the decoding step, the parameters of the encoder 130, the parameters of the backbone model 150, and / or the parameters of the decoder 170. The steps mentioned above can be repeated iteratively.

[0060] Note that during the inference process, word-level prediction 180, which is related to... Figure 3 Described in more detail, it may include embedding step 120, encoding step 130, linear mapping 140, backbone step 150, and / or further linear mapping 160. Word-level prediction 180 may result in generating predicted word vector representations 160a (i.e., ).

[0061] Figure 2 The character-level reasoning process 200 is shown, which can also be referred to as character completion.

[0062] During inference, a trained machine learning model can be used to generate text based on an input text. The provided input text is processed by the trained machine learning model. The processing of the provided input text can be related to... Figure 1 The descriptions are the same, except that the output may not be used to train the model, but rather to iteratively generate text. Figure 2The iterative generation of characters is illustrated, which can also be referred to as character completion 200. The trained decoder 270 takes the predicted word vector representation 260a as input and generates a character probability vector 270a. As mentioned above, each position in the character probability vector represents the probability that a particular character will be the next character. Such a character probability vector 270a can be shown as ["a" = 0.6, "b" = 0.3, …, "z" = 0.1], where the character "a" has a 60% probability of being the next character, "b" has a 30% probability, and so on.

[0063] about Figure 2 The provided input text is "Hello World,". The predicted word vector representation 260a can therefore represent the words predicted by the backbone model 150 based on the input text 101 "Hello World,". Therefore, the character probability vector 270a can represent the probability that each possible character is the next character in a character sequence that begins with "Hello World,".

[0064] To achieve iterative text generation, the vector representation of the most likely next character, in this case the vector representation 290a of the character "m", can be concatenated to the predicted word vector representation 260a. The concatenation of the predicted word vector representation and the vector representation of the most likely next character can then be used as updated input to the decoder 270. Based on the updated input information, the decoder 270 can predict another character probability vector 271a. In the example, the most likely subsequent character is the letter "y" 290b. The input to the decoder 270 is updated accordingly, and the process continues iteratively until the decoder 270 predicts that the most likely character is a special character (i.e., the character with the highest probability in the final character probability vector 270d). Note that a special character can represent the end of a word. If a special character is predicted, word-level predictions 180 and 380 can be triggered, which involve the backbone model and... Figure 3 It is described in more detail.

[0065] Figure 3 The word-level reasoning process 300 is shown, which can begin as soon as a special character is predicted. For example... Figure 3As shown, given the input text "Hello World," the characters "m", "y", "_" 390 and the special character 390d can be predicted as the most likely next character. The special character 390d can indicate that the previously generated characters 390a-c form a word, and that this word is now finished. The special character 390d can also serve as a trigger for word-level prediction 300. More specifically, when the special character is predicted, the special character 390d can be prepended to the character sequence of the previously generated characters 390a-c. The previously generated character sequence of 390a-c, including the prepended special character 390d, can be used as input to embedding step 120, where each character in the sequence is converted into a character vector representation 120a. The subsequent encoding step 130 can return an encoded vector representation 130a for each character. However, only one encoded vector representation for each word can be used for further processing, and other encoded vector representations are discarded. A linear mapping step 140 may be needed to merge the remaining encoded word vector representations 140a. Regarding... Figure 3 The linear mapping step 140 can merge the encoded word vector representations of the words “[W]Hello_” and “[W]World,_” with the newly generated encoded word vector representation “[W]my_”. This word-level input text can be used as the input text for the subsequent backbone stage, which predicts the word vector representations 150a of subsequent words. This result can be used as the input for decoders 170 and 270, and can initiate functions such as... Figure 2 The character completion value is 200.

[0066] Therefore, the machine learning model can predict the most likely next character and iteratively update the inputs of decoders 170 and 270 to predict subsequent characters 390a-c. Once the model predicts a special character 390d that can indicate the end of a word (i.e., the character with the highest probability in the final character probability vector 270d) as the most likely next character, word-level prediction 300 can be triggered, and the input of the backbone model 150 can be updated to include the previously generated characters 390a-c. Note that updating the input of the backbone model 150 may require the characters 390a-c generated by embedding 120, encoding 130, and linear transformation 140. In this way, the machine learning model can combine processing of input text at both the character and word levels.

[0067] The advantage of the machine learning model disclosed herein can be the reduction in computational cost, which results from lower computational complexity. This reduction in computational complexity can be demonstrated by comparing the complexity of the model disclosed herein (…). ) and the complexity of the baseline model ( This will be used to prove it. The complexity of both models can depend heavily on the length of the sequence 140a through the trunk. In the case of the model disclosed in this paper, the length of the sequence is the length of the word vector representation, and can be represented as... Assuming the base model uses a word segmentation tool, the length of the sequence is the length of the word vector representation multiplied by the number of words present in the sequence, and can be represented as... Both models contain the same number of backbone parameters. The baseline model may need to have parameters. The additional embedding and output matrices. This disclosed model may also include parameters of the encoder and decoder models. The length of the sequence passed through the encoder and decoder can be greater than the length of the sequence passed through the backbone model, and is denoted as... Therefore, the computational complexity of the baseline model can be described as... Furthermore, the computational complexity of the model disclosed herein can be described as... .

[0068] Therefore, the complexity of the model disclosed herein can be lower than that of the baseline model if the following condition is met: (a) and (b) The first condition can relate to the length of the input sequence to the backbone. More specifically, the length of the input sequence to the backbone of the model of this disclosure may have to be shorter than the length of the input sequence to the backbone of the baseline model. Since the backbone of the model of this disclosure processes input at the word level, and the backbone of the baseline model processes input at the sub-word level, it can be assumed that the average length of the sequences in the model of this disclosure is smaller. Therefore, this condition can be achieved. The second condition can relate to the size of the corresponding model (e.g., measured in the number of parameters). More specifically, the encoder model and decoder model may have to be much smaller than the backbone model. Given that the main computation occurs in the backbone model, this condition can also be achieved with the model of this disclosure.

[0069] Figure 4 This is a flowchart illustrating a computer-implemented method 400 for training a machine learning model for text generation according to embodiments of the present disclosure. A first step 410 includes inputting text into the machine learning model. A second step 420 includes preprocessing the input text to obtain multiple character vector representations. A third step 430 includes encoding each of the multiple character vector representations using an encoder to obtain multiple word vector representations. A fourth step 440 includes generating multiple predicted word vector representations based on the multiple word vector representations using a backbone model. A fifth step 450 includes decoding the multiple predicted word vector representations using a decoder to obtain multiple character probabilities; and a sixth step includes updating the machine learning model based on the multiple character probabilities.

[0070] Figure 5 This is a block diagram of an example computing device 500 (which may also be referred to as, for example, a “computing device,” a “computer system,” or a “computing system”) according to some embodiments.

[0071] In some embodiments, computing device 500 includes one or more of the following: one or more processors 502 (which may be referred to as "a plurality of hardware processors" or simply "hard processors"); one or more memory devices 504; one or more network interface devices 506; one or more display interfaces 508; and one or more user input adapters 510. Additionally, in some embodiments, computing device 500 is connected to or includes display devices, input devices, etc. These components (e.g., processors 502, memory devices 504, network interface devices 506, display interfaces 508, user input adapters 510) are hardware devices (e.g., electronic circuits or combinations of circuits) configured to perform various functions of computing device 500. In some embodiments, these components of computing device 500 may be collectively referred to as computing resources (e.g., resources used for executing instructions, and including processors (one or more processors 502), storage devices (one or more memory devices 504), and I / O (network interface devices 506, one or more display interfaces 508, and one or more user input adapters 510).

[0072] In some instances, terminology processing resources can be used interchangeably with terminology computing resources. In some embodiments, multiple instances of computing device 500 can be deployed in a distributed computing system. Computing device 500 can be configured to communicate with one or more external devices 516. External devices 516 can be other instances of computing devices, or they can be different (e.g., just storage devices, sensors, etc.). In some examples, computing device 500 includes multiple computing devices 500. As an example, computing device 500 includes different architectures that can be used in a cloud computing environment.

[0073] In some embodiments, each or any of the processors 502 is or includes, for example, a single-core or multi-core processor, a microprocessor (e.g., which may be referred to as a central processing unit or CPU), a digital signal processor (DSP), a microprocessor associated with a DSP core, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, or a system-on-a-chip (SOC) (e.g., an integrated circuit including a CPU and other hardware components such as memory, network interfaces, and the like). And / or, in some embodiments, each or any of the processors 502 uses an instruction set architecture such as x86 or an advanced RISC machine (ARM).

[0074] In some embodiments, each or any of the memory devices 504 is or includes, or comprises, other types of devices that are random access memory (RAM) (such as dynamic RAM (DRAM) or static RAM (SRAM)), flash memory (based on, for example, NAND or NOR technology), hard disk, magneto-optical media, optical media, cache memory, registers (e.g., those storing instructions), or volatile or non-volatile storage of data and / or instructions (e.g., software executed on or by processor 502). Memory device 504 is an example of a non-transitory computer-readable storage medium.

[0075] In some embodiments, each or any of the network interface devices 506 includes one or more circuits (such as a baseband processor and / or a wired or wireless transceiver) and implements a first layer, second layer and / or higher layer for one or more wired communication technologies (such as Ethernet (IEEE 802.3)) and / or wireless communication technologies (such as Bluetooth, WiFi (IEEE 802.11), GSM, CDMA2000, UMTS, LTE, LTE-Advanced (LTE-A), LTE Pro, Fifth Generation New Radio (5G NR) and / or other short-range, medium-range and / or long-range wireless communication technologies).

[0076] The terms and phrases used in this document, and their variations thereof, should be interpreted as open-ended rather than restrictive, unless otherwise expressly stated. As examples above: “and / or” includes any and all combinations of one or more of the associated listed items (e.g., a and / or b means a, b, or a and b); the singular forms “a,” “an,” and “the” should be interpreted as meaning “at least one” or “one or more,” etc.; the term “example,” which may be used interchangeably with the term “example,” is used to provide an example of the subject matter under discussion, not an exhaustive or restrictive list thereof; the terms “comprise” and “include” (and their other variations and variants) specify the presence of an associated listed element, but do not preclude the presence or addition of one or more other elements; and if an element is described as “optional,” such description should not be construed as indicating the need for other elements not so described.

[0077] As used herein, the term "non-transitory computer-readable storage medium" includes registers, cache memories, ROMs, semiconductor memory devices (such as D-RAM, S-RAM, or other RAMs), magnetic media (such as flash memory), hard disks, magneto-optical media, optical media (such as CD-ROMs, DVDs, or Blu-ray discs), or other types of volatile or non-volatile storage devices used for non-transitory electronic data storage. The term "non-transitory computer-readable storage medium" does not include transient propagation of electromagnetic signals. The computer programs described herein may be stored on non-transitory computer-readable storage media.

[0078] Claims are not intended to be understood / interpreted as means plus function unless they explicitly use the phrases "means for" or "steps for". Claim elements intended to be understood / interpreted as means plus function language, if any, will explicitly indicate this intention by stating the phrases "means for" or "steps for"; the foregoing applies to claim elements in all types of claims (method claims, apparatus claims, or other types of claims), and, for the avoidance of doubt, also to claim elements nested within method claims. Consistent with the foregoing, no claim element (in any claim of any type) should be understood / interpreted using means plus function unless the claim element is explicitly stated using the phrases "means for" or "steps for".

[0079] Although various embodiments have been shown and described in detail, the claims are not limited to any particular embodiment or example. None of the foregoing descriptions should be construed as implying that any particular element, step, scope, or function is essential. All structural and functional equivalents of the elements of the foregoing embodiments known to those skilled in the art are expressly incorporated herein by reference and are intended to be covered. Furthermore, the apparatus or method need not solve every or every problem sought to be solved by this invention, as it is covered by this invention. No embodiment, feature, element, component, or step in this document is intended to be proprietary to the public.

[0080] Embodiments of this disclosure can be implemented in any of a variety of forms, such as software. For example, in some embodiments, the invention can be implemented as a computer-implemented method, a computer-readable storage medium, or a computer system.

[0081] In some embodiments, a non-transitory computer-readable storage medium may be configured to store program instructions and / or data, wherein the program instructions, if executed by a computer system, cause the computer system to perform a method, such as any of the method embodiments described herein, or any combination of the method embodiments described herein, or any subset of any of the method embodiments described herein, or any combination of such subsets.

[0082] In some embodiments, a computing device may be configured to include a processor (or a group of processors) and a memory medium, wherein the memory medium stores program instructions, and wherein the processor is configured to read from the memory medium and execute the program instructions, wherein the program instructions are executable to implement any of the various method embodiments described herein (or any combination of the method embodiments described herein, or any subset of any of the method embodiments described herein, or any combination of such subsets). The device may be implemented in any of a variety of forms.

[0083] Although specific embodiments have been described above, these embodiments are not intended to limit the scope of this disclosure, even where only a single embodiment has been described with respect to a particular feature. Unless otherwise stated, the examples of features provided in this disclosure are intended to be illustrative and not restrictive. The above description is intended to cover alternatives, modifications, and equivalents as will be apparent to those skilled in the art who benefit from this disclosure.

[0084] The scope of this disclosure includes any feature or combination of features disclosed herein (expressly or implicitly), or any generalization thereof, whether or not it alleviates any or all the problems addressed herein. In particular, with reference to the appended claims, features from dependent claims may be combined with features from independent claims, and features from the respective independent claims may be combined in any suitable manner, and not merely in the specific combinations listed in the appended claims.

[0085] Reference Symbol List

[0086] 100 Training Process

[0087] 101 Input text

[0088] 110a, 310a character sequences

[0089] 110 Word Breakdown

[0090] 120 Embedding Steps

[0091] 120a Character Vector Representation

[0092] 130 encoder

[0093] Character vector representation of 130a encoding

[0094] 140, 160 linear mapping

[0095] 140a Word Vector Representation

[0096] 150 backbone model

[0097] 150a Predicted word vector representation

[0098] Word vector representations predicted by 160a and 260a

[0099] 160b is the sequence of character vectors representing the actual next word.

[0100] 170, 270 decoders

[0101] Character probability vectors 170a and 270a-c

[0102] 171a The actual next character

[0103] 180, 300, 380 word-level prediction

[0104] 200-character completion

[0105] Characters generated by 290a-c and 390a-c

[0106] 390d Special Characters

[0107] 400 methods for training

[0108] 410 Input Steps

[0109] 420 Preprocessing Steps

[0110] 430 Encoding Steps

[0111] 440 Main trunk prediction steps

[0112] 450 Decoding Steps

[0113] 460 Update Steps

[0114] 500 computing devices

[0115] 502 One or more processors

[0116] 504 One or more memory devices

[0117] 506 One or more network interface devices

[0118] 508 One or more display interfaces

[0119] 510 One or more user input adapters

[0120] 516 One or more external devices

Claims

1. A computer-implemented method for training a machine learning model for text generation, the method comprising: Input the text into the machine learning model; Preprocess the input text to obtain multiple character vector representations; An encoder is used to encode each character vector in the plurality of character vector representations to obtain a plurality of word vector representations; The backbone model is used to generate multiple predicted word vector representations based on the multiple word vector representations; The multiple predicted word vector representations are decoded using a decoder to obtain multiple character probabilities; and The machine learning model is updated based on the probabilities of the multiple characters.

2. The method according to the preceding claims further includes: Iteratively repeat the steps of input, preprocessing, encoding, generation, decoding, and updating.

3. The method according to any one of the preceding claims, wherein, Preprocessing includes splitting the input text into multiple character sequences, where each character sequence represents a word; and Each character in the plurality of character sequences is embedded to obtain the plurality of character vector representations.

4. The method according to the preceding claim, wherein, Preprocessing includes prepending special characters to each character sequence before embedding.

5. The method according to any one of the preceding claims, wherein, The encoder is a first natural language processing model, wherein preferably, the architecture of the first natural language processing model is based on a converter model with only decoder variants, and most preferably, the attention mechanism of the converter is bidirectional.

6. The method according to any one of the preceding claims, wherein, The backbone model is a second natural language processing model, wherein preferably, the architecture of the second natural language processing model is based on a converter model with only decoder variants, and most preferably, the attention mechanism of the converter is causal.

7. The method according to any one of the preceding claims, the method comprising: Before the decoding step, each of the multiple predicted word vector representations is concatenated with its corresponding character vector representation.

8. The method according to any one of the preceding claims, wherein, The decoder is a third natural language processing model, wherein preferably, the architecture of the third natural language processing model is based on a converter model with only decoder variants, and most preferably, the attention mechanism of the converter is causal.

9. The method according to any one of the preceding claims, wherein updating the machine learning model comprises updating one or more of the following adjustable parameters: the embedding matrix used during the preprocessing step, the encoder, the backbone model, and / or the decoder.

10. A computer-implemented method for generating text using a machine learning model trained according to any one of the preceding claims, the method comprising: Input the text into a trained machine learning model; The trained machine learning model is used to generate text based on the input text.

11. The method according to the preceding claim, wherein, The generated text includes: Characters are generated based on the probability of the multiple characters; and The decoder input is updated based on the generated characters, or the backbone model input is updated based on one or more generated characters; and Iteratively generate and update repeatedly.

12. The method according to the preceding claim, wherein, Updating the input to the decoder includes: It is determined that the generated character is not a special character; and The decoder input is updated based on the character vector representation of the generated characters; and The updated input is decoded to obtain the probabilities of multiple characters.

13. The method according to claim 11, wherein, Updating the inputs to the backbone model includes: It has been determined that the generated characters are special characters; The special character is prepended to one or more generated characters to obtain a predicted character sequence; Each character of the predicted character sequence is embedded to obtain multiple predicted character vector representations; The encoder is used to encode the predicted character vector representation to obtain the predicted word vector representation; The input to the backbone model is updated based on the predicted word vector representations; Generate predicted word vector representations based on the updated input; The predicted word vector representation is decoded to obtain the probabilities of multiple characters.

14. An apparatus or system comprising means for performing the method according to any one of claims 1 to 13.

15. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 13.

Images