Methods, devices, and computer programs for training machine learning models to generate text and for generating text using trained machine learning models.

Abstract

This invention provides a computer implementation method, program, device, and system for training a machine learning model for text generation. [Solution] The method includes inputting text into a machine learning model, preprocessing the input text to obtain multiple character vector representations, encoding each of the multiple character vector representations using an encoder to obtain multiple word vector representations, generating multiple predictive word vector representations based on the multiple word vector representations using a backbone model, decoding the multiple predictive word vector representations using a decoder to obtain multiple character probabilities, and updating the machine learning model based on the multiple character probabilities.

Description

[Technical Field]

[0001] Technical field This disclosure relates to a computer implementation method for training a machine learning model for generating text, a computer implementation method for generating text using a trained machine learning model, and corresponding computer programs, devices, and systems. [Background technology]

[0002] Technical background In the context of natural language processing, tokenization refers to the process of dividing text into smaller parts (i.e., tokens). Tokens serve as the basis for further processing steps, such as embeddings. Thus, tokenization divides text into meaningful constituent elements that will be processed further. The granularity of the resulting tokens can vary. Two basic approaches to tokenization are character-level tokenization and word-level tokenization.

[0003] During character-level tokenization, text is divided into individual characters rather than other words or sentences. Each individual character is then treated as a distinct token. Thus, the input text "Hello" becomes a total of six tokens: "h", "e", "l", "l", and "o". The advantage of character-level tokenization is vocabulary size. Since each character is a token, the total size of the vocabulary is limited to the number of characters. This type of tokenization can handle words that have never been seen before, because it focuses on the character level rather than words. However, character-level tokenization results in very long sequences, which increases the computational complexity of the learning and inference processes. For example, the word "nature" becomes a sequence of six different tokens, rather than just one token as in word-character tokenization. Since the computational complexity of machine learning models that handle natural language is based on the length of the sequence, character-level tokenization increases the computational cost. Finally, character-level tokenization also makes it difficult to capture long-range dependencies, such as words relating to other words in a sentence.

[0004] In contrast, during word-level tokenization, text is divided into words rather than individual characters or whole sentences. Each word is then treated as an individual token. The sentence "Hello world" becomes, for example, the tokens "Hello" and "world". Compared to character-level tokenization, word-level tokenization is better at preserving semantic meaning. Also, word-level tokenization produces smaller sequences and is generally more computationally efficient than character-level tokenization. However, word-level tokenization suffers from performance degradation if words are misspelled. Furthermore, words that did not exist during training are treated as unknowns when using word-level tokenization. Therefore, trained models are highly susceptible to the corpus used for training. Word-level tokenization also struggles to accurately handle word forms such as "run," "running," and "ran." These forms are different words and therefore receive different tokens. However, they are likely to convey similar meanings.

[0005] Subword tokenizers aim to combine both character-level and word-level tokenization by dividing text into smaller units that are larger than characters but smaller than words. Each subword is then treated as a token. The sentence "Hello world" could, for example, be made up of tokens such as "Hel," "lo," and "world." The subword approach aims to balance the efficiency of the word-level approach with the flexibility of the character-level approach. However, subword tokenization still has several drawbacks, including a lack of adaptability to new domains or languages, susceptibility to typographical errors and spelling variations, and a larger vocabulary size compared to the character-level approach.

[0006] Given these shortcomings, currently known tokenization approaches do not necessarily lead to the desired results. Against this backdrop, the purpose of this disclosure is to address one or more of the aforementioned shortcomings, or all of them. [Overview of the project]

[0007] Summary of the present invention The aforementioned objectives and other objectives will become apparent from the following description. These objectives are achieved by the subject matter of the independent claim. Preferred embodiments are the subject matter of the dependent claim.

[0008] A first embodiment of this disclosure is a computer implementation method for training a machine learning model for text generation, comprising: inputting text into the machine learning model; preprocessing the input text to obtain a plurality of character vector representations; encoding each of the plurality of character vector representations using an encoder to obtain a plurality of word vector representations; generating a plurality of predictive word vector representations based on the plurality of word vector representations using a backbone model; decoding the plurality of predictive word vector representations using a decoder to obtain a plurality of character probabilities; and updating the machine learning model based on the plurality of character probabilities.

[0009] Preprocessing input text to obtain multiple character vector representations can have the advantage of preparing the input text for further processing in subsequent steps. Since the input text is preprocessed to obtain character-level vector representations, the size of the resulting vocabulary can be limited to the number of possible characters. A smaller vocabulary size can reduce the memory required and conserve computational resources. Furthermore, character-level initial processing can offer a greater level of flexibility.

[0010] Using an encoder to encode each of multiple character vector representations to obtain multiple word vector representations has the advantage of being able to convert character-level input into word-level output. In other words, the input to the encoder is based on character-level vector representations (i.e., one vector per character), and the output of the decoder is based on word-level vector representations (i.e., one vector per word). Converting from character-level to word-level can save computational resources during further processing of the input text.

[0011] Using a backbone model to generate multiple predictive word vector representations based on multiple word vector representations has the advantage of allowing the most computationally expensive part of the model to be performed on the word-level representation of the input text. More specifically, in a word-level representation, a single word vector represents the entire word. In contrast, in a character-level representation, a single word requires x character vectors, where x is the number of characters that make up the word. For example, using a word-level representation, the word "nature" requires one word vector, but one vector for each character in the word, i.e., six character vectors. Therefore, performing backbone calculations at the word level can have the advantage of being less complex and saving computational resources.

[0012] Decoding multiple predictive word vector representations using a decoder to obtain multiple character probabilities can enable character-level final processing. This may have the advantage of providing more flexible output and improving the model's performance in terms of output quality. Updating a machine learning model based on multiple character probabilities may have the advantage of being able to refine the model based on the processed text.

[0013] Finally, performing the backbone computation (i.e., the computationally most costly part of the model) at the word level, while performing the input (i.e., the initial preprocessing of the input text) and output (i.e., the final processing of the output text) at the character level, can combine the advantages of both character-level and word-level approaches. More specifically, the combined approach benefits from the flexibility of the character-level approach and the reduced computational resource requirements of the word-level approach.

[0014] According to the second embodiment, the steps of input, preprocessing, encoding, generation, decoding, and updating are repeated iteratively.

[0015] The iterative approach of repeatedly performing the steps of input, preprocessing, encoding, generation, decoding, and updating can have the advantage of gradually improving the performance of a machine learning model. This iterative approach also has the advantage of enabling training on large training datasets. More specifically, large datasets can reach a size where the entire training dataset cannot fit into memory. The iterative approach can be used to iteratively load portions of the training dataset into memory and train the model on the loaded portions of the training dataset. A further advantage of the iterative approach is that data that becomes available after the initial training of the machine learning model can still be incorporated through additional training iterations. Furthermore, the iterative approach can enable performance monitoring. For example, the model's performance can be tested after a predetermined number of iterations. If the model is not performing as desired, adjustments can be made to the model before further training.

[0016] According to the third embodiment, the preprocessing includes dividing the input text into a plurality of character sequences, each representing a word, and embedding each character in the plurality of character sequences to obtain the plurality of character vector representations.

[0017] Splitting the input text into a plurality of character sequences each representing a word can have the advantage that the input text can be preprocessed at the character level. Embedding each character in the plurality of character sequences to obtain a plurality of character vector representations can have the advantage that vector representations for each character of the input text can be obtained.

[0018] According to the fourth embodiment, the preprocessing includes adding a special character at the beginning to each character sequence before embedding.

[0019] Adding a special character at the beginning to each character sequence before embedding can have the advantage of introducing a special character that can be used later to represent the character sequence to which it is added at the beginning. Adding a special character at the beginning before embedding can make it possible to accurately incorporate the special character into the learning process.

[0020] According to the fifth embodiment, the encoder is a natural language processing model, preferably, the architecture of the first natural language processing model is based on a variant of the transformer model with only the decoder, and most preferably, the attention mechanism of the transformer is bidirectional.

[0021] The encoder being a first natural language processing model can provide the ability to process a sequence as an input and a return sequence as an output. The architecture of the first natural language processing model being preferably based on a variant of the transformer model with only the decoder can have the advantage of leveraging the performance advantages of each architecture. The attention mechanism of the transformer being preferably bidirectional can improve the ability of the model to understand the context of the input text. More specifically, the bidirectional attention mechanism considers both preceding and succeeding words that improve context awareness simultaneously.

[0022] According to the sixth embodiment, the backbone model is a second natural language processing model, preferably the architecture of the second natural language processing model is based on a transformer model that is a decoder-only variant, and most preferably the attention mechanism of the transformer is causal.

[0023] The fact that the backbone model is a second natural language processing model, and preferably the architecture of the second natural language processing model is based on a transformer model of a decoder-only variant, may have the same advantages as described with respect to Embodiment 5. The causal attention mechanism of the transformer may allow the model to autoregressively generate text. In other words, the causal attention mechanism may allow the backbone model to generate outputs based on outputs that have each been previously generated.

[0024] According to the seventh embodiment, prior to the decoding step, each of the plurality of predictive word vector representations is concatenated to a corresponding character vector representation.

[0025] Concatenating each of multiple predictive word vector representations to their corresponding character vector representations before the decoding step may have the advantage of improving the information on which the decoding step is based. This is particularly true because the predictive word vector representations represent the word predicted by the backbone model as the next word, and the corresponding character vector representations represent the actual next word. The combination of word-level representations (i.e., word vectors of the predicted next word) and character-level representations (i.e., character vectors of the actual next word) may further improve the information on which the decoding step is based.

[0026] According to the eighth embodiment, the decoder is a third natural language processing model, preferably the architecture of the third natural language processing model is based on a transformer model that is a variant of the decoder alone, and most preferably the attention mechanism of the transformer is causal.

[0027] The decoder being a third natural language processing model, and preferably the architecture of the third natural language processing model being based on a transformer model of a decoder-only variant, may have the same advantages as described with respect to Embodiment 5. Preferably the attention mechanism of the transformer being causal, may have the same advantages as discussed in Embodiment 6.

[0028] According to the ninth embodiment, updating the machine learning model includes updating one or more tunable parameters, which consist of one or more of the embedding matrix, the encoder, the backbone model, and / or the decoder used during the preprocessing step.

[0029] Updating one or more tunable parameters, consisting of one or more embedding matrices, encoders, backbone models, and / or decoders, used during the preprocessing step enables training of the machine learning model. This may have the advantage of improving the performance of the machine learning model. This may also provide further flexibility in training some components of the machine learning model while fixing other components of the machine learning model.

[0030] A tenth embodiment of this disclosure is a computer implementation method for generating text using a machine learning model trained according to any of the preceding embodiments, wherein text is input to the trained machine learning model, and the trained machine learning model is used to generate text based on the input text.

[0031] A computer implementation method for generating text using a machine learning model trained according to any of the preceding embodiments described above may have the advantage of reducing computational complexity while maintaining the performance of the machine learning model. As described with respect to the earlier embodiments, the architecture of the trained machine learning model may provide computational efficiency for word-level processing while maintaining flexibility for character-level processing.

[0032] According to the 11th embodiment, generating the text includes generating characters based on the multiple character probabilities, updating the decoder input based on the generated characters, or updating the backbone model input based on one or more generated characters, and repeatedly performing the generation and updating.

[0033] Generating characters based on multiple character probabilities can enable character-level text generation. In other words, a machine learning model can predict each character of the next word individually, rather than predicting the entire word. This can improve prediction results. Character-level prediction can also have the advantage of making predictions more flexible. Updating the decoder input based on generated characters, or updating the backbone model input based on one or more generated characters, can have the advantage of allowing generated characters to be considered during the generation of subsequent characters. Iterative generation and updating can further improve the generated text.

[0034] According to the twelfth embodiment, updating the input to the decoder includes determining that the generated character is not a special character, updating the input to the decoder based on the character vector representation of the generated character, and decoding the updated input to obtain a plurality of character probabilities.

[0035] Updating the decoder input, including determining whether the generated character is a special character, may allow switching between character-level and word-level predictions. Determining whether the generated character is a special character may convey the prediction for the next character. Updating the decoder input based on the character vector representation of the generated character may have the advantage of allowing the generated character to be taken into consideration when generating the next character. This may improve the prediction results. Decoding the updated input to obtain multiple character probabilities may provide flexibility during text generation by predicting the next word on an individual character basis rather than at the word level.

[0036] According to the 13th embodiment, updating the input to the backbone model includes determining that the generated character is a special character, adding the special character to the beginning of one or more generated characters to obtain a predicted character sequence, embedding each character of the predicted character sequence to obtain a plurality of predicted character vector representations, encoding the predicted character vector representations using the encoder to obtain a predicted word vector representation, updating the input to the backbone model based on the predicted word vector representations, generating a predictive word vector representation based on the updated input, and decoding the predictive word vector representations to obtain a plurality of character probabilities.

[0037] Updating the input to the backbone model, including determining whether a generated character is a special character, may have the advantage of combining character-level prediction with word-level prediction. In other words, determining whether a generated character is a special character may trigger word-level prediction. The word vector of the generated word (i.e., one or more generated characters) and the character vector of the generated word can then be used to update the input to the backbone model. Thus, the flexibility of character-level prediction is combined with the efficiency of word-level prediction performed by the backbone model. Obtaining a predicted character sequence by prefixing one or more generated characters with a special character, obtaining multiple predicted character vector representations by embedding each character in the predicted character sequence, and then encoding the predicted character vector representations using an encoder to obtain a predicted word vector representation may have the advantage of preprocessing and encoding one or more generated characters. This may improve subsequent processing, particularly the predictions performed by the backbone model. Updating the input to the backbone model based on the predicted word vector representation may have the same advantages as described above in terms of leveraging the efficiency of word-level prediction. In other words, performing backbone calculations on word-level representations can be less computationally complex and save computational resources.

[0038] Generating predictive word vector representations based on updated input may have the advantage of leveraging the efficiency of the backbone model to predict the next word. As mentioned earlier, processing at the word level may be more efficient, especially if the sequence length representing the input is shorter. Decoding the predictive word vector representations to obtain multiple character probabilities may provide flexibility during text generation by predicting the next word on an individual character basis rather than at the word level.

[0039] A fourteenth embodiment of this disclosure is directed to a device or system having means for performing a method according to any of embodiments 1 to 13.

[0040] A device or system equipped with means for carrying out a method according to any of embodiments 1 to 13 may have all the advantages described with respect to the corresponding embodiment.

[0041] A fifteenth embodiment of this disclosure is directed to a computer program that, when executed on a computer, includes instructions causing the computer to perform a method according to any of embodiments 1 to 13.

[0042] A computer program that, when executed on a computer, includes instructions causing the computer to perform a method according to any of embodiments 1 to 13, may have all the advantages described with respect to the corresponding embodiment. [Brief explanation of the drawing]

[0043] Simple explanation of the diagram Various aspects of the present invention will be described in more detail below with reference to the accompanying drawings, and the present invention is not limited to the embodiments shown in these drawings. [Figure 1] Figure 1 shows an exemplary learning process according to an embodiment of the present disclosure. [Figure 2] Figure 2 shows an exemplary reasoning process according to an embodiment of the present disclosure. [Figure 3] Figure 3 shows an exemplary reasoning process according to an embodiment of the present disclosure. [Figure 4] Figure 4 further illustrates a flowchart showing a computer implementation method for training a machine learning model for text generation according to an embodiment of the present disclosure. [Figure 5] Figure 5 shows an exemplary computing device that may be used in one embodiment to realize the features described herein. [Modes for carrying out the invention]

[0044] Detailed description of preferred embodiments The present invention will be described in more detail below with reference to the accompanying drawings. However, the present invention can also be used in other embodiments not expressly disclosed below. As will be described in detail below, the embodiments are compatible with one another, and individual features of one embodiment may be applied to other embodiments. The drawings are not intended to limit the scope of the claims and are merely to support the understanding of the present invention.

[0045] Figure 1 shows an exemplary training process 100 of a machine learning model according to an embodiment of the present disclosure.

[0046] The learning process 100 can generally be divided into three stages: the encoding stage, the backbone stage, and the decoding stage. As shown in Figure 1, the encoding and decoding stages process the input text 101 at the byte level (i.e., character level). In contrast, the backbone stage processes the input text at the word level. At the core of each stage may be a machine learning model for natural language processing that processes the input text.

[0047] The learning process 100 may be based on one or more corpora of texts. For illustrative purposes, Figure 1 focuses on the processing of a single text (i.e., "Hello World, my Name") that may have occurred in the text corpus. "Hello World, my Name" is considered the input text 101. In the preprocessing step, the input text may be split into words. Word splitting 110 may be performed using fixed splitting rules. More specifically, the text may be split by spaces. In addition, spaces may be added to the preceding word. Note that a special token, shown here as [W], may be added to the beginning of each word. This special token may indicate the start of each word. For the input text 101, such word splitting and prefixing may produce the character sequences 110a "[W]Hello_", "[W]World,_", "[W]my_", and "W]Name_". The input text is Assuming the file is TIFF2026064230000002.tif1055, T=(b1, …, b n ) can be called. In other words, the input text T is one or more characters b represented in binary form. i It may include. Splitting the input text into a sequence of words S is possible. Assuming the filename is TIFF2026064230000003.tif1023, S=(w1, …, w n )=(([W], b1, …, b k(T) ), …, ([W], b l(T) , …, b n It can also be expressed as )). In this expression, the special token has already been added to the beginning of each word in the sequence of word 110a.

[0048] The resulting character sequence 110a serves as input to an embedding step 120 in which each character of each sequence is converted into a vector 120a. The embedding step 120 can be implemented using an embedding matrix, which may be a matrix in which each row corresponds to a vector representation (i.e., embedding 120a) of a vocabulary token (e.g., a character). During the embedding step 120, the embedding matrix may be used to retrieve the vector representation 120a of each character and replace each character with its corresponding vector representation 120a.

[0049] The subsequent encoding step 130 encodes each vector representation 120a using a natural language processing model. The architecture of the natural language processing model may be based on a decoder-only variant of the transformer model. The attention mechanism of the decoder-only variant of the transformer model may be bidirectional. As shown in Figure 1, the encoding step 130 may return an encoded vector representation 130a for each character. However, only one encoded vector representation per word may be used for further processing. Other encoded vector representations are discarded. In this way, the machine learning model may learn to represent one word with one vector (i.e., a vector that is not discarded but processed further).

[0050] Since discarding can change the dimensionality of the information, a linear mapping step 140 may be required to consolidate the remaining encoded word vector representations. The result of the encoding step may be a high-density representation of the input text in the form of one encoded vector representation 140a for each word in Figure 1. As shown in Figure 1, E1 may be the encoded vector representation of the word "[W]Hello_". E2 may be the encoded vector representation of the word "[W]World_". E3 may be the encoded vector representation of the word "[W]my_". E4 may be the encoded vector representation of the word "[W]Name_". Thus, the encoding step can be used to convert character-level input into word-level output.

[0051] This word-level input text can serve as input text for a subsequent backbone stage. The backbone stage may include a natural language processing model 150 that uses word vector representations (i.e., E1, E2, E3, E4) 140a to predict subsequent word vector representations (i.e., P1, P2, P3) 150a, respectively. The architecture of the natural language machine learning model may be based on a decoder-only variant of the transformer model. The attention mechanism of the decoder-only variant of the transformer model may be causal. A model with a non-causal attention mechanism may describe a model that generates predictions based on past information. For example, given the string "Hello World, my Name", if predicting the word "World", a causal model would only consider the word "Hello". If the model is not causal, it may also consider the words "my" and "Name" for the prediction of the word "World".

[0052] The decoding stage may begin with a further linear transformation 160. Again, the linear transformation may serve the purpose of adjusting the dimensionality of the input information. More specifically, the input information is adjusted before entering the decoder 170 by concatenating the word vector representation (i.e., word-level representation) 140a of the predicted next word with a sequence of character vector representations (i.e., character-level representations) 160b of the actual next word. For example, the characters "W", "o", "r", "l", "d", "," and "_" in character vector representation 160b are concatenated to P1 of the predicted word vector representation 160a, the characters "m", "y" and "_" are concatenated to the predicted word vector representation P2, and so on. Note that the characters of the actual next word 160b may be embedded using an embedding matrix different from the embedding matrix that may have been used in the initial embedding step 120.

[0053] Character-level input information can function as input to decoder 170. Decoder 170 may be a natural language processing model. The architecture of the natural language processing machine learning model may be based on a transformer model of a decoder-only variant. The attention mechanism of the transformer model of a decoder-only variant may be causal. Based on character-level input information, decoder 170 may return multiple character probability vectors 170a. Each position in the character probability vector may describe the probability that a particular character is the next character in the text. Note that the character logit mentioned in Figure 1 is a processed form of character probability 170a that may be further processed to improve performance.

[0054] Finally, a cross-entropy loss function may be used to compare the predictions of the machine learning model with the actual values ​​171a. The learnable parameters of the machine learning model may be updated according to the results of the comparison. This may include updating the embedding matrix in the encoding step 120, the parameters of the embedding matrix in the decoding step, the parameters of the encoder 130, the parameters of the backbone model 150 and / or the parameters of the decoder 170. The steps described above may be repeated iteratively.

[0055] It should be noted that during the inference process, the word-level prediction 180, which will be described in more detail with respect to Figure 3, may include an embedding step 120, an encoding step 130, a linear mapping 140, a backbone step 150, and / or further linear mapping 160. The word-level prediction 180 may generate a predictive word vector representation 160a (i.e., P2).

[0056] Figure 2 shows the character-level reasoning process, which can also be called character completion.

[0057] During inference, a trained machine learning model may be used to generate text based on a portion of the input text. The provided input text is processed by the trained machine learning model. The processing of the provided input text may be identical to that described with respect to Figure 1, with the difference that the output may not be used to train the model but rather to iteratively generate text. Figure 2 shows iterative generation of characters, also known as character completion 200. The trained decoder 270 may take a predicted word vector representation 260a as input and generate a character probability vector 270a. As mentioned above, each position in the character probability vector represents the probability that a particular character is the next character. Such a character probability vector 270a may be represented as ["a"=0.6, "b"=0.3, …, "z"=0.1]. The character "a" has a 60% probability of being the next character, "b" has a 30% probability of being the next character, and so on.

[0058] Regarding Figure 2, the provided input text was "Hello World,". Therefore, the predicted word vector representation 260a can represent the words predicted by the backbone model 150 based on the input text 101 "Hello World,". Thus, the character probability vector 270a can represent the probability that each character in the character sequence starting with "Hello World," is the next character.

[0059] To achieve iterative text generation, the vector representation of the most likely next character, in this example the character "m" (vector representation 290a), can be concatenated to the predicted word vector representation 260a. The concatenation of the predicted word vector representation and the vector representation of the most likely next character can then be used as the updated input to the decoder 270. Based on the updated input information, the decoder 270 can predict a further vector of character probabilities 271a. In this example, the most likely subsequent character is the character "y" (vector representation 290b). Accordingly, the input to the decoder 270 is updated, and the process continues iteratively until the decoder 270 predicts that the most likely character is a special character (i.e., the character with the highest probability relative to the final vector of character probabilities 270d). Note that the special character may indicate the end of a word. When a special character is predicted, word-level predictions 180,380, accompanied by a backbone model, can be triggered, as described in more detail in Figure 3.

[0060] Figure 3 shows a word-level inference process 300 that may be initiated when a special character is predicted. As shown in Figure 3, the characters "m", "y", "_" 390 and the special character 390d may be predicted as the most likely next characters for the input text "Hello World". The special character 390d may indicate that previously generated characters 390a-c formed a word and that this word has now ended. The special character 390d may also function as a trigger for word-level prediction 300. More specifically, when a special character is predicted, the special character 390d may be prepended to the sequence of previously generated characters 390a-c. The sequence of previously generated characters 390a-c, prepended with the special character 390d, may be used as input to an embedding step 120 in which each character in the sequence is converted to a character vector representation 120a. A subsequent encoding step 130 may return an encoded vector representation 130a for each character. However, only one encoded vector representation per word may be used for further processing. Other encoded vector representations are discarded. A linear mapping step 140 may be required to combine the remaining encoded word vector representations 140a. With respect to Figure 3, the linear mapping step 140 may combine the encoded word vector representations for the words "[W]Hello_" and "[W]World,_" with the newly generated encoded word vector representation "[W]my_". This word-level input text can serve as input text for a subsequent backbone stage that predicts the word vector representation of the subsequent word 150a. This result can be used as input to decoders 170,270, which may initiate character completion 200 as described in Figure 2.

[0061] Therefore, the machine learning model can predict the most likely next character and can iteratively update the inputs to the decoders 170, 270 to predict subsequent characters 390a-c. A special character 390d (i.e., the character having the highest likelihood with respect to the final vector 270d of character probabilities) may indicate the end of a word, but when the model predicts a special character as the most likely next character, word-level prediction 300 may be triggered and the input to the backbone model 150 may be updated to incorporate the previously generated characters 390a-c. It should be noted that updating the input to the backbone model 150 may require embedding 120, encoding 130, and linear transformation 140 of the generated characters 390a-c. Thus, the machine learning model may combine processing of character-level and word-level input text.

[0062] An advantage of the machine learning model of the present disclosure may be a reduction in computational cost resulting from less computational complexity. The reduction in computational complexity may manifest by comparing the complexity (C disclosure ) of the present disclosure model to the complexity (C baseline ) of a baseline model. The complexity of both models may depend significantly on the length of the sequence 140a passing through the backbone. In the case of the present disclosure model, the length of the sequence is the length of the word vector representation and may be represented as L W . When the base model uses a subword tokenizer, the length of the sequence is the length of the subword vector representation multiplied by the number of subwords present in the sequence and may be represented as L T . Both models contain the same number of backbone parameters P backbone . The baseline model requires additional embedding and may output a matrix having parameters P head . The present disclosure model may further include parameters P char of the encoder and decoder models. The length of the sequence passing through the encoder and decoder may be longer than that passing through the backbone model and may be represented as L+L W . Thus, the computational complexity of the baseline model is Cbaseline =L T (P backbone +P head ) can be described as. The computational complexity of the model in this disclosure is C disclosure =L W P backbone +2(L+L W )P char It can be described as follows.

[0063] Therefore, the complexity of this disclosure is (a)L W <L T and (b)P char < <P backbone The complexity may be lower than that of the baseline model if the following conditions are met. The first condition may relate to the length of the input sequence to the backbone. More specifically, the length of the input sequence to the backbone of the disclosed model may have to be shorter than the length of the input sequence to the backbone of the baseline model. Since the backbone of the disclosed model processes input at the word level and the baseline model processes input at the subword level, the sequence length can be assumed to be shorter on average in the disclosed model. Thus, this condition may be met. The second condition may relate to the size of each model as measured by the number of parameters. More specifically, the encoder model and decoder model may have to be much smaller than the backbone model. This condition may also be met by the models of the disclosed model if the main computation is performed in the backbone model.

[0064] Figure 4 is a flowchart showing a computer implementation method 400 for training a machine learning model for text generation according to an embodiment of the present disclosure. A first step 410 includes inputting text into the machine learning model. A second step 420 includes preprocessing the input text to obtain multiple character vector representations. A third step 430 includes encoding the multiple character vector representations using an encoder to obtain multiple word vector representations. A fourth step 440 includes generating multiple predictive word vector representations based on the multiple word vector representations using a backbone model. A fifth step 450 includes decoding the multiple predictive word vector representations using a decoder to obtain multiple character probabilities. A sixth step includes updating the machine learning model based on the multiple character probabilities.

[0065] Figure 5 is a block diagram of an exemplary computing device 500 (sometimes also referred to as a "computing device," "computer system," or "computing system") according to one embodiment.

[0066] In one embodiment, the computing device 500 includes one or more processors 502 (sometimes referred to as "(multiple) hardware processors" or separately as "(one) hardware processor"), one or more memory devices 504, one or more network interface devices 506, one or more display interfaces 508, and one or more user input adapters 510. Furthermore, in one embodiment, the computing device 700 is connected to or includes a display device, an input device, etc. These elements (e.g., processors 502, memory devices 504, network interface devices 506, display interfaces 508, user input adapters 510) are hardware devices (e.g., electronic circuits or combinations of circuits) configured to perform various different functions for the computing device 500. In one embodiment, these components of the computing device 500 may be collectively referred to as computing resources (for example, resources used to execute instructions, including a processor (one or more processors 502), storage (one or more memory devices 504), and I / O (network interface device 506, one or more display interfaces 508, and one or more user input adapters 510)).

[0067] In some examples, the term "processing resources" may be used interchangeably with the term "computing resources." In some embodiments, multiple computing devices 500 may be deployed in a distributed computing system. A computing device 500 may be configured to communicate with one or more external devices 516. The external devices 516 may be other computing devices or different devices (e.g., mere storage devices, sensors, etc.). In some examples, the computing device 500 comprises multiple computing devices 500. As an example, the computing device 500 includes different architectures that may be used in a cloud computing environment.

[0068] In some embodiments, each or any of the processors 502 is, for example, a single-core or multi-core processor, a microprocessor (sometimes referred to as a central processing unit or CPU), a digital signal processor (DSP), a microprocessor associated with a DSP core, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, or a system-on-a-chip (SOC) (for example, an integrated circuit including a CPU and other hardware components such as memory and network interfaces). Furthermore / or, in some embodiments, each or any of the processors 502 uses an instruction set architecture such as x86 or Advanced RISC Machine (ARM).

[0069] In one embodiment, each or any of the memory devices 504 is or includes random access memory (RAM) (such as dynamic RAM (DRAM) or static RAM (SRAM)), flash memory (e.g., based on NAND or NOR technology), a hard disk, magneto-optical media, optical media, cache memory, registers (e.g., for holding instructions), or other types of devices that perform volatile or non-volatile storage of data and / or instructions (e.g., software executed on or by the processor 502). Memory devices 504 are examples of non-temporary computer-readable media.

[0070] In one embodiment, each or any of the network interface devices 506 includes one or more circuits (such as a baseband processor and / or a wired or wireless transceiver) and implements Layer 1, Layer 2, and / or higher layers for one or more wired communication technologies (such as Ethernet (IEEE 802.3)) and / or one or more wireless communication technologies (such as Bluetooth, WiFi (IEEE 802.11), GSM, CDMA2000, UMTS, LTE, LTE-Advanced (LTE-A), LTE Pro, 5th Generation New Radio (5G NR) and / or other short-range, medium-range, and / or long-range wireless communication technologies).

[0071] The terms and phrases and their variations used in this document should be interpreted as unrestrictive, not restrictive, unless explicitly stated otherwise. For example, “and / or” includes any combination of one or more of the related items listed, and all combinations thereof (e.g., a and / or b means a, b, or a and b). The singular forms “a,” “an,” and “the” should be read as meaning “at least one,” “one or more,” etc. The term “example” may be used interchangeably with the term “embodiment,” but it is used to provide examples of the subject being described, not to provide an exclusive or restrictive list thereof. The terms “equip” and “include” (and other inflections and variations) identify the presence of the related elements listed, not to exclude the presence or addition of one or more other elements. Where an element is described as “optional,” such description should not be understood as indicating that other elements not described in this way are required.

[0072] As used herein, the term “non-temporary computer-readable storage medium” includes registers, cache memory, ROM, semiconductor memory devices (such as D-RAM, S-RAM, or other RAM), magnetic media such as flash memory, hard disks, magneto-optical media, optical media such as CD-ROMs, DVDs, or Blu-ray discs, or other types of volatile or non-volatile storage devices for the non-temporary storage of electronic data. The term “non-temporary computer-readable storage medium” does not include temporary propagating electromagnetic signals. Computer programs described herein may be stored on non-temporary computer-readable storage medium.

[0073] The claims are intended not to give rise to means-plus-function interpretations / understandings unless they explicitly use the phrases “means to” or “steps to.” The elements of the claims are intended to be interpreted / understood as means-plus-function language if their intent is explicitly expressed by including the phrases “means to” or “steps to.” The above applies to the elements of claims in all types of claims (method claims, apparatus claims, or other types of claims), and also applies to nested elements of claims within method claims to avoid doubt. In accordance with the above, the elements of claims (in any type of claim) should not be interpreted / understood using means-plus-function interpretations / understandings unless the elements of the claims are explicitly described using the phrases “means to” or “steps to.”

[0074] Although various embodiments have been described in detail, the claims are not limited to any particular embodiment or example. Nothing in the above description should be understood as suggesting that any particular element, step, scope, or function is essential. All structural and functional equivalents to the elements of the embodiments described above, known to those skilled in the art, are expressly incorporated herein by reference and are intended to be included. Furthermore, it is not necessary for a device or method to solve the respective problem that the invention seeks to solve in order to be included in the invention. The embodiments, features, elements, components, or steps described herein are not intended to be made available to the public.

[0075] Embodiments of this disclosure can be implemented in any of the following forms, for example, by software. For example, in one embodiment, the present invention may be implemented as a computer implementation method, a computer-readable memory medium, or a computer system.

[0076] In one embodiment, a non-temporary computer-readable memory medium may be configured to store program instructions and / or data that, when executed by a computer system, cause the computer system to perform a method, for example, any embodiment of the method described herein, or any combination of embodiments of the method described herein, or any subset of any embodiment of the method described herein, or any combination of such subsets.

[0077] In some embodiments, a computing device may be configured to include a processor (or a set of processors) and a memory medium. The memory medium stores program instructions, and the processor is configured to read and execute program instructions from the memory medium. The program instructions are executable to carry out any of the various embodiments of the methods described herein (or any combination of embodiments of the methods described herein, or any subset of embodiments of the methods described herein, or any combination of such subsets). The device may be implemented in a variety of forms.

[0078] While specific embodiments have been described above, these embodiments are not intended to limit the scope of the disclosure, even if only a single embodiment is described with respect to a particular feature. Examples of features provided in this disclosure are intended to be descriptive, not restrictive, unless otherwise stated. The above description is intended to include alternatives, modifications, and equivalents that would be apparent to those skilled in the art as having the advantages of this disclosure.

[0079] The scope of this disclosure includes any or all of the features disclosed herein (expressly or implicitly), or combinations thereof, or generalizations thereof, whether or not they mitigate any or all of the problems sought to be solved herein. In particular, with reference to the appended claims, the features of the dependent claims may be combined with the features of the independent claims, and each of the features of the independent claims may be combined in an appropriate manner, not only in the specific combinations set forth in the appended claims. [Explanation of symbols]

[0080] List of reference codes 100 Learning Processes 101 Input Text 110a,310a character sequence 110 word segmentation 120 embedding steps 120a Character vector representation 130 encoders 130a Encoded character vector representation 140,160 Linear Mapping 140a Word vector representation 150 Backbone Model 150a Predictive word vector representation 160a,260a Predictive word vector representation 160b Sequence of character vector representations of the actual next word 170,270 decoders 170a, 270a-c Character probability vectors 171a The actual next character Predictions at the 180, 300, and 380 word levels. 200 characters completed 290a-c, 390a-c Generated characters 390d Special Characters 400 methods for training 410 Input Steps 420 Pre-treatment steps 430 encoding steps 440 Backbone prediction steps 450 decode steps 460 update steps 500 computing devices 502 Processors 504 Memory Devices 506 Network Interface Device 508 Display Interface 510 User Input Adapter 516 External devices

Claims

1. A computer implementation method for training a machine learning model for text generation, Input text into the aforementioned machine learning model, The input text is preprocessed to obtain multiple character vector representations. Using an encoder, each of the multiple character vector representations is encoded to obtain multiple word vector representations. Using a backbone model, multiple predictive word vector representations are generated based on the multiple word vector representations. Using a decoder, the multiple predictive word vector representations are decoded to obtain multiple character probabilities. The machine learning model is updated based on the multiple character probabilities mentioned above. Computer implementation method.

2. Furthermore, the method described in the prior claim, further comprising the steps of input, preprocessing, encoding, generation, decoding, and updating, repeated iteratively.

3. The aforementioned preprocessing involves dividing the input text into multiple character sequences, each representing a word. A method according to any one of the preceding claims, comprising embedding each character in the plurality of character sequences to obtain the plurality of character vector representations.

4. The method described in the preceding claim, wherein the preprocessing includes adding a special character to the beginning of each character sequence before embedding.

5. The encoder is a first natural language processing model, preferably the architecture of the first natural language processing model is based on a transformer model of a decoder-only variant, most preferably the attention mechanism of the transformer is bidirectional, as described in any one of the preceding claims.

6. The backbone model is a second natural language processing model, preferably the architecture of the second natural language processing model is based on a transformer model of a decoder-only variant, most preferably the attention mechanism of the transformer is causal, as described in any one of the preceding claims.

7. The method according to any one of the preceding claims, comprising concatenating each of the plurality of predictive word vector representations to a corresponding character vector representation prior to the decoding step.

8. The decoder is a third natural language processing model, preferably the architecture of the third natural language processing model is based on a transformer model of a variant of the decoder alone, most preferably the attention mechanism of the transformer is causal, as described in any one of the preceding claims.

9. The method according to any one of the preceding claims, wherein updating the machine learning model includes updating one or more tunable parameters, which consist of one or more of the embedding matrix, the encoder, the backbone model, and / or the decoder used in the preprocessing step.

10. A computer implementation method for generating text using a machine learning model trained according to any one of the prior claims, The text is input into the aforementioned trained machine learning model, Using the aforementioned trained machine learning model, generate text based on the input text. Computer implementation method.

11. Generating the aforementioned text means Based on the above multiple character probabilities, characters are generated, The decoder input is updated based on the generated characters, or the backbone model input is updated based on the one or more generated characters. The generation and update processes are repeated. A method described in a prior claim, including the following:

12. Updating the input of the decoder means It is determined that the generated character is not a special character, The input to the decoder is updated based on the character vector representation of the generated character. Decode the updated input to obtain multiple character probabilities. A method described in a prior claim, including the following:

13. Updating the input to the aforementioned backbone model means It is determined that the generated character is a special character, The predicted character sequence is obtained by adding the special character to the beginning of one or more generated characters. By embedding each character of the aforementioned predicted character sequence, multiple predicted character vector representations are obtained. Using the encoder, the predicted character vector representation is encoded to obtain the predicted word vector representation. The input to the backbone model is updated based on the predicted word vector representation. Based on the updated input, a predictive word vector representation is generated. Decode the aforementioned predictive word vector representation to obtain multiple character probabilities. The method according to claim 11, including the following:

14. A device or system comprising means for carrying out the method according to any one of claims 1 to 13.

15. A computer program that, when executed by a computer, includes an instruction causing the computer to perform the method according to any one of claims 1 to 13.

Images