Generation of data models for predictive data

Abstract

A system and method for training a data model based on training data is provided. The training includes pre-training and fine-tuning a data model based on a combination of an autoregressive (AR) model and a non-autoregressive (NAR) model. The training data can be and is encoded as a word-meta stream. A pre-trainer during decoding generates a continuum of continuous data structures of a combined AR and NAR model, the combined AR and NAR model comprising a mainstream and a series of prediction streams. The masked word-meta in the prediction stream references or pays attention to one or more previous word-meta in the mainstream or previous prediction stream. A fine-tuner selects a stream to generate a trained model based on a target data model. The target data model is determined based on balancing accuracy constraints and efficiency constraints for predicting word-meta. The decoder acts as a bridge between the AR and NAR models in generating the trained data model.

Description

Background Art

[0001] Traditional systems use methods based on sequence-to-sequence models to predict and generate data, such as keywords from queries. These systems generate or predict each word-element, for example, the encoded form of the words in the query sentence, conditioned on the word-element previously generated in the autoregressive (AR) model. The AR generation model generates output based on the previously generated output, and the process is repeated multiple times depending on the length of the generated sentence. Although AR models can provide accuracy in predicting words, these models are processing intensive and may cause high latency, which may become unacceptably long when processing large amounts of data (e.g., during web searches by search engines). In contrast, non-autoregressive (NAR) models generate all word-element in parallel, resulting in higher word-element generation efficiency. However, NAR models are not as accurate as AR models in predicting word-element. In fact, depending on system requirements or capabilities, deployment costs, latency requirements, accuracy requirements, etc., there is a trade-off between deploying AR models or NAR models. Therefore, it would be desirable to develop a technology that better meets these requirements while minimizing trade-offs.

[0002] The aspects disclosed herein are made with respect to these and other general considerations.In addition, although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving specific problems identified elsewhere in the background or this disclosure. Summary of the invention

[0003] According to the present disclosure, the above and other problems are solved by providing a pre-training flow that combines AR models and NAR models for sequence-to-sequence generation.

[0004] The present disclosure relates to generating a language generation model based on a pre-training and fine-tuning process. In particular, the pre-training process generates a mainstream and one or more prediction streams for each word of a target sentence. The result set of the prediction stream for the target sentence represents a superset of the data used for fine-tuning to generate various combinations of a series of AR and NAR models for deployment in different scenarios. Then, the fine-tuning process generates a trained model by selecting streams from the mainstream and prediction streams to build a structure that combines the features of the AR and NAR models. The streams selected for building the trained model are based on balancing the efficiency and accuracy requirements for a specific language processing scenario. A specific trained model can be defined based on a predetermined number of word-grams to be processed using an AR model and the remaining word-grams processed by a NAR model. A specific combination of AR and NAR models can be selected based on balancing real-time efficiency and accuracy requirements. The disclosed technology enables the use of an AR model to partially generate a target sequence (e.g., the first two words of a sentence predicted based on an AR model) and the use of a NAR model to generate the remaining word-grams or words of the target sequence. In this way, predicting tokens or words that require higher accuracy (e.g., initial tokens) uses more resources than processing the remaining words of the target sentence (e.g., subsequent words). The accuracy of predicting the entire sentence is improved by correctly generating the first few tokens, while the efficiency is improved by generating the remaining tokens in parallel based on the NAR model.

[0005] The disclosed technology includes an encoding process and a decoding process for generating a trained model. The decoding process will be the main focus of the present disclosure. The decoding process includes a pre-training sub-process and a fine-tuning process. The pre-training sub-process includes a self-attention mechanism based on an n-stream structure to model different iterative combinations of AR and NAR models. In the fine-tuning sub-process, the present technology is able to generate a trained model based on the best combination of AR and NAR models, for example, by selecting from a continuum of combinations of AR and NAR models to strike a balance between word unit accuracy and processing efficiency.

[0006] Some examples include a method for generating a data model (e.g., a language generation model). The method may include receiving training data, generating a mainstream based on the training data, wherein the mainstream represents a word-gram stream, and wherein each word-gram represents a value associated with the training data, generating a series of prediction streams based on the mainstream, wherein each prediction stream includes one or more masked word-grams corresponding to one or more word-gram positions in the word-gram stream, selecting a first set of streams for predicting word-grams according to a first processing model based on a target data model, wherein the first set of streams is selected from the mainstream and the series of prediction streams, selecting a second set of streams for predicting word-grams according to a second processing model based on the target data model, wherein the second set of streams is selected from the series of prediction streams, and generating a trained data model corresponding to the target data model using the selected first set of streams and second set of streams for predicting word-grams according to accuracy constraints and efficiency constraints during deployment. By generating a trained data model corresponding to the target data model for predicting word-grams, a trained data model that satisfies requirements associated with efficiency constraints and accuracy constraints can be generated in a deployed system environment. The first processing model can be associated with an autoregressive (AR) model, and wherein the second processing model is associated with a non-autoregressive (NAR) model. The target data model may specify a certain number of word-grams to be processed based on the first processing model and the remaining word-grams to be processed based on the second processing model. The second stream in the second stream set may refer to the predicted value of the word-gram in the first stream in the second stream set. The second stream set may be processed in parallel according to the second processing model. The second stream in the first stream set may refer to the value of the word-gram in the first stream in the first stream set. The first stream may be the mainstream. The method may also include determining the target data model based on balancing the accuracy constraints and efficiency constraints for predicting word-grams. The method may also include receiving a query, generating a query set based on the received query using a trained data model, retrieving one or more keywords based on the query set, generating a keyword set by matching the retrieved one or more keywords with at least one known keyword in a keyword corpus, and providing the generated keyword set.

[0007] Some examples include a system for training a data model. The system includes a processor and a memory storing computer executable instructions that, when executed by the processor, cause the system to: receive training data, generate a main stream based on the training data, wherein the main stream represents a word-gram stream, and wherein each word-gram represents a value associated with the training data, generate a series of prediction streams based on the main stream, wherein each prediction stream includes one or more masked word-grams corresponding to one or more word-gram positions in the word-gram stream, select a first set of streams for predicting word-grams according to a first processing model based on a target data model, wherein the first set of streams is selected from the main stream and the series of prediction streams, select a second set of streams for predicting word-grams according to a second processing model based on the target data model, wherein the second set of streams is selected from the series of prediction streams, and generate a trained data model corresponding to the target data model using the selected first set of streams and second set of streams for predicting word-grams according to accuracy constraints and efficiency constraints during deployment. By generating a trained data model corresponding to the target data model for predicting word-grams, a trained data model that satisfies requirements associated with efficiency constraints and accuracy constraints can be generated in a deployed system environment. The first processing model can be associated with an autoregressive (AR) model, and wherein the second processing model is associated with a non-autoregressive (NAR) model. The target data model can specify a number of word-grams to be processed based on the first processing model and the remaining word-grams to be processed based on the second processing model. The second stream in the second stream set can reference predicted values ​​of word-grams in the first stream in the second stream set. The second stream set can be processed in parallel according to the second processing model. The second stream in the first stream set can reference the value of word-grams in the first stream in the first stream set. The first stream can be the mainstream. The computer executable instructions, when executed, also cause the system to determine the target data model based on balancing accuracy constraints and efficiency constraints for predicting word-grams.

[0008] Some examples include a computer-readable non-transitory recording medium storing computer-executable instructions that, when executed by a processor, cause a computer system to: receive training data, generate a main stream based on the training data, wherein the main stream represents a stream of word-grams and wherein each word-gram represents a value associated with the training data, generate a series of prediction streams based on the main stream, wherein each prediction stream includes one or more masked word-grams corresponding to one or more word-gram positions in the stream of word-grams, select a first set of streams for predicting word-grams according to a first processing model based on a target data model, wherein the first set of streams is selected from the main stream and the series of prediction streams, select a second set of streams for predicting word-grams according to a second processing model based on the target data model, wherein the second set of streams is selected from the series of prediction streams, and generate a trained data model that is consistent with the target data model for predicting word-grams using the selected first set of streams and second set of streams for predicting word-grams according to accuracy constraints and efficiency constraints during deployment.

[0009] The present invention summary is provided to introduce a selection of concepts in a simplified form, which is further described in the following detailed description. The present invention summary is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features and / or advantages of the examples will be set forth in part in the following description, and in part will be apparent from the description, or may be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0011] Figure 1 An overview of an example system for generating trained models and deploying them into application services according to aspects of the present disclosure is shown.

[0012] Figure 2 An example of a system architecture according to aspects of the present disclosure is shown.

[0013] Figure 3 An example of a pre-trained stream according to aspects of the present disclosure is shown.

[0014] Figure 4 An example of a data structure for predicting a flow during pre-training according to aspects of the present disclosure is shown.

[0015] Figure 5A An example of determining matching keywords by generating a query based on a received query using a trained model according to aspects of the present disclosure is shown.

[0016] Figure 5B An example of determining matching keywords based on generative retrieval using a trained model according to aspects of the present disclosure is shown.

[0017] Figure 6 Examples of methods for generating and using trained models according to aspects of the present disclosure are shown.

[0018] Figure 7 A block diagram illustrating example physical components of a computing device with which aspects of the present disclosure may be practiced.

[0019] Fig. 8A is a simplified diagram of a mobile computing device with which aspects of the present disclosure may be practiced.

[0020] Figure 8B is another simplified block diagram of a mobile computing device with which aspects of the present disclosure may be practiced. DETAILED DESCRIPTION

[0021] Various aspects of the present disclosure are described more fully below with reference to the accompanying drawings, which are a part of the present disclosure and illustrate specific example aspects. However, different aspects of the present disclosure can be implemented in many different ways and should not be construed as limited to the aspects set forth herein; on the contrary, these aspects are provided so that the present disclosure will be thorough and complete, and the scope of these aspects will be fully conveyed to those skilled in the art. Aspects can be practiced as methods, systems, or devices. Therefore, aspects can take the form of hardware implementations, full software implementations, or implementations combining software and hardware aspects. Therefore, the following detailed description is not limiting.

[0022] Natural language processing relies on predicting missing data in a given dataset for a variety of purposes, including language translation, summarizing text, information retrieval, image recognition, etc. Currently, there are two different models for automatically generating natural language data. The autoregressive (AR) generative model predicts output (e.g., word-unit) based on previously generated output (e.g., prior word-unit), and repeats the process multiple times depending on the length of the target sentence being predicted or generated. The AR generative model produces high-quality (e.g., highly accurate) prediction results. However, since the AR generative model is intensively processed and suffers from high latency, problems arise in efficiently predicting target sentences. When processing large training corpora in a real-time processing environment, latency may become unacceptably long. In contrast, the non-autoregressive (NAR) model processes all word-units in parallel. The NAR generative model uses one or more masks in the target sentence and the connections between the masks to predict the corresponding word-unit or word of the target sentence. Since the NAR model processes all word-units in parallel, it cannot benefit from previously predicted data. Therefore, predicting a target sentence based on the NAR generative model may be more efficient but less accurate than predictions based on the AR generative model. The present application solves this trade-off by providing a system that is efficiently tunable to train a language model based on combined aspects of both AR generation models and NAR generation models. The present application enables balancing the competing demands of accuracy and efficiency to generate a trained model customized for a specific processing scenario.

[0023] As discussed in more detail below, the present disclosure relates to a language generation model for producing a trained model that improves efficiency without substantially compromising accuracy. Specifically, the language generation model selectively uses a combination of AR and NAR generation models to balance efficiency and accuracy of natural language processing based on specific system resources, accuracy requirements, latency requirements, etc. The disclosed technology solves this problem by generating a set of prediction data streams through a pre-training process that can be used to generate a continuum of combinations of AR and NAR generation models for fine-tuning the trained model based on real-time requirements.

[0024] Figure 1An overview of an example system 100 for generating and using a language model according to aspects of the present disclosure is shown. System 100 represents a system for using a language generation model (or models) to generate a trained model using a trainer and a production server. System 100 includes a client device 102, an application server 110, a data server 120, a model generator 140, and a network 130. Client device 102 communicates with application server 110, which includes one or more sets of instructions to be executed as an application on client device 102. Application server 110 includes data retriever 112 and data viewer 114. One or more sets of instructions in application server 110 can provide an interactive user interface through interactive browser 104. Search server 120 includes query receiver 122, generative keyword retriever 124, trained model 126, and answer provider 128. Network 130 provides network connectivity between client device 102, application server 110, and search server 120. The model generator 140 includes an encoder 142, a decoder 143, training data 148, and a deployer 150. The decoder 143 includes a pre-trainer 144 and a fine-tuner 146. The data server 160 stores data and provides data to the search server 120 and the application server 110 through the network 130.

[0025] The client device 102 is connected to the application server 110 via the network 130 to execute the application including user interaction through the interactive browser 104. The application server 110 interacts with the client device 102 and the search server 120 via the network 130 to perform information search and retrieval. According to the present disclosure, the model generator 140 generates a trained language model (or other trained models for predicting other types of missing data (e.g., pixel data)). In various aspects, the model generator 140 can generate a trained language model based on a combination of AR and NAR models.

[0026] The client device 102 is a general-purpose computer device that provides user input capabilities (e.g., via an interactive browser 104 for searching information on a network 130). In some aspects, the client device 102 optionally receives user input from a system developer to achieve a desired balance between efficiency and accuracy based on the number of tokens to be processed by the AR model and the NAR model in the resulting trained language model. For example, the interactive browser 104 can present a graphical user interface associated with a web browser. In various aspects, the client device 102 can communicate with the application server 110 via the network 130.

[0027] The application server 110 is a server that enables users (who can search for information based on deployed training models) and developers (who can use the model generator 140 to generate trained models for deployment) to interactively use the system 100 on the client device 102. The application server 110 may include applications, including a data retriever 112 and a data viewer 114. The data viewer 112 may provide a presentation of data for viewing by a user. For example, the data retriever 114 may be used to query and receive data by communicating with the search server 120.

[0028] In various aspects, the data retriever 112 may be connected to a query receiver 122 of a search server 120 to query for information. The generative keyword retriever 124 may perform generative retrieval of keywords based on a received query by expanding the query and retrieving data from a data server 160. The trained model 126 (e.g., a trained language generation model) may perform expansion of the received query to a query set by predicting terms of the query based on the received query. The answer provider 128 may provide answers (e.g., results) to the query to the data retriever 112 on an interactive browser 104 in a client device 102. The data viewer 114 may receive the results to the query for viewing by a user using the interactive browser 104 on the client device 102.

[0029] The model generator 140 represents an application / system for generating a language generation model and training the language generation model to be deployed as a trained model 126 in the search server 120. In an embodiment, the model generator 140 includes an encoder 142, a decoder 143, training data 148, and a deployer 150. The decoder 143 may further include a pre-trainer 144 and a fine tuner 146. The encoder 142 encodes the training data 148. The pre-trainer 144 uses the encoded training data to pre-train and generate a set of mainstream and prediction streams as a superset of the data for fine-tuning. The fine tuner 146 selects the mainstream and one or more prediction streams to generate a trained model, which is a combination of AR and NAR models as specified by the developer. When hybrid AR-NAR is selected to process all tokens (further described below), the fine tuner 146 does not select the mainstream but selects all prediction streams in the prediction stream to generate a trained model. The training data 148 includes a sentence corpus for training the model. In various aspects, the training data 148 includes at least one text sentence including a collection of words (or tokens). In some other aspects, the training data 148 includes an image corpus for training the model. The training data corpus for images may include pixel data for the images. The deployer 150 deploys the fine-tuned (i.e., trained) model to the search server 120 as the trained model 126.

[0030] In an embodiment, the model generator 140 generates a trained model for deployment (wherein the trained model may include a trained language model, a trained image model, etc.). The generation process includes an encoding process and a decoding process. The encoder 142 receives training data (e.g., at least one text sentence) and encodes the text sentence into an encoded word element. A multidimensional vector may represent each encoded word element. The encoded word element set of the target sentence for training may be in the form of a mainstream. The pre-trainer 144 generates a set of prediction streams based on a self-attention model using the mainstream. In various aspects, the self-attention model according to the present disclosure allows not only "attention" to the current word element itself but also allows "attention" to other words before the current word element in the target sentence. That is, the result of the prediction for the previous word element is used to predict the current word element by analyzing how the current word element is related to each word element in the previous word element in the target sentence.

[0031] The decoder includes a pre-trainer for the pre-training process and a fine tuner for the fine tuning process accordingly. The pre-training process (e.g., performed by the pre-trainer 144) receives the encoded word-grams of the training sentences and generates a mainstream and a prediction stream set for predicting the word-grams of the target sentence. The prediction stream set represents a superset of data for generating different combinations of AR and NAR models customized for different accuracy and efficiency requirements. In traditional decoders for AR language generation models, "teacher-forcing" is a common term used to describe the use of previously predicted word-grams as context inputs to predict the current word-gram. In various aspects, teacher-forcing may include comparing the predicted answer with the correct answer, correcting the predicted answer with the correct answer if the predicted answer is incorrect, and using the correct answer to predict subsequent word-grams or words in the sentence. The mainstream includes a word-gram stream. In some aspects, the pre-trainer 144 generates a mainstream based on "teacher-forcing". In traditional decoders for NAR language generation models, several different initialization methods are popular, such as encoder replication, [MASK](M) initialization, and posterior distribution approximation. For NAR language generation models, length prediction can determine the number of masked tokens to feed.

[0032] The fine-tuning process (e.g., performed by the fine-tuner 146) uses the mainstream and the complete set of prediction streams to generate a trained model customized for a specific processing scenario. This is accomplished by selecting a subset of streams that combine aspects of the AR and NAR models to meet the efficiency / accuracy requirements of a specific scenario. In order to balance the accuracy and efficiency requirements, the combination may involve selecting the number of tokens in the target sentence to be predicted based on the AR model and the remaining tokens to be predicted based on the NAR model. Then, the stream subset is selected from the superset for processing the multiple initial tokens based on AR and processing the remaining tokens based on NAR. Predictions based on the AR model provide higher accuracy than predictions based on the NAR model. However, the prediction processing efficiency based on the AR model is lower than that based on the NAR model. However, this combination enables the initial token to be predicted more accurately using AR, and based on using the initial prediction as context, subsequent tokens can be predicted more accurately and efficiently using the NAR model. The pre-trainer generates a series of prediction streams based on the mainstream. Each prediction stream includes one or more masked tokens corresponding to one or more token positions in the token stream in the prediction stream. In some examples, each prediction stream in the superset of prediction streams references one or more token predictions of a previous stream. In this case, unlike a traditional NAR model, the disclosed model can be referred to as a "hybrid AR-NAR model" because it incorporates retrospective context (e.g., predictions of previous tokens) in the prediction streams used during parallel processing to predict the remaining tokens of the target sentence.

[0033] In various aspects, the trained model contains two components: an encoder part and a decoder part. The pre-training and fine-tuning process updates the parameters in both the encoder part and the decoder part. The structure of the encoder part is not changed during the pre-training and fine-tuning process, but the parameters are updated. The structure of the decoder part generated during the pre-training and fine-tuning process depends on the target data model of the combination of the specified processing model, ranging from AR to hybrid AR-NAR to NAR models.

[0034] In one example, for a target sentence of three words (i.e., three tokens), a trained language model is generated for predicting a first token based on an AR model and predicting the remaining two tokens based on a hybrid AR-NAR model, and the fine tuner 146 selects a mainstream, a first prediction stream, and a second prediction stream. In this case, the first token is based on the first token in the mainstream. The second token in the first prediction stream refers to the first token in the mainstream. The third token in the second prediction stream refers to the first token in the mainstream and the second token in the first prediction stream.

[0035] In another example, for a target sentence of four words (i.e., four tokens), a trained language model is generated for predicting the first two tokens based on the AR model and the remaining two tokens based on the hybrid AR-NAR model, and the fine tuner 146 selects a mainstream, a first prediction stream, and a second prediction stream. The first token is based on the first token in the mainstream. The second token in the first prediction stream refers to the first token in the mainstream. The third token in the first prediction stream refers to the first and second tokens in the mainstream. The fourth token in the second prediction stream refers to the first and second tokens in the mainstream and the third token in the first prediction stream.

[0036] In yet another example, all tokens of the target sentence may be processed based on the hybrid AR-NAR model described above. That is, the fine tuner 146 may select all prediction streams in the prediction stream instead of the mainstream to predict a given number of tokens in the target sentence. However, since the generated prediction stream references the prior token prediction, even if the streams are processed in parallel, the accuracy is improved based on the previous context (i.e., the prior token prediction), but the efficiency is maintained based on the parallel processing.

[0037] In various aspects, the decoder 143 according to the present embodiment treats one or more (including all) combinations of different combinations of previously predicted word units and masks (M) as decoder inputs. As described above, the pre-trainer 144 generates a prediction stream that refers to (or pays attention to) the prediction of the word unit before the current word unit. By considering all different combinations, the decoder 143 combines the AR language generation model and the NAR language generation model for unified large-scale pre-training. In various aspects, the decoder 143 provides pre-training in an efficient n-stream manner (e.g., processing multiple prediction streams), and uses a self-attention mechanism (e.g., mainstream) to model the continuum of all structures of the prediction model of the AR and NAR combination. The decoder 143 then provides fine-tuning to generate a trained language model, which is optimized for the balance between efficiency and accuracy, as required by a specific deployment environment. Therefore, the decoder 143 can support prediction processing based on a portion of the AR language generation model and prediction processing based on the remainder of the hybrid AR-NAR language generation model to balance efficiency and accuracy.

[0038] Thus, as described above, the pre-trainer 144 uses the mainstream to generate a set of prediction streams. The mainstream and the set of prediction streams are supersets of streams that are used for fine-tuning to generate various trained models based on the type of target data model used to predict word units, from models that are more based on AR models (more accurate) to models that are more based on hybrid AR-NAR models (more efficient). The fine-tuner 146 performs fine-tuning by selecting a subset of streams from the superset of pre-trained streams generated by the pre-trainer 144. The subset of streams is selected to generate a trained model based on the need to evaluate the accuracy / efficiency of a specific processing scenario, and the trained model is customized to meet those needs. That is, for a target sentence of a given length, it can be determined that a certain number of word units should be processed by AR, and the remaining word units are processed by hybrid AR-NAR. Based on this determination, the fine-tuner 146 can select a subset of streams required to perform word unit processing accordingly.

[0039] As will be understood, with respect to Figure 1 The various methods, devices, applications, features, etc. described are not intended to limit the system 100 to be performed by the specific applications and features described. Therefore, additional controller configurations may be used to practice the methods and systems herein and / or the features and applications described may be excluded without departing from the methods and systems disclosed herein.

[0040] Figure 2 Examples of three different types of system architectures for language generation models according to various aspects of the present disclosure are shown. First, system 200A shows a trained model based on an AR model. System 200A receives an input 202A including a set of word-grams. There are five word-grams in the example, the first word-gram is a special "start of a sentence" symbol, the second word-gram is y0, the third word-gram is y1, the fourth word-gram is y2, and the fifth word-gram is y3. Decoder 204A decodes the input word-gram to generate an output word-gram as output 206A. In the AR language generation model, each word-gram pays attention to its previous word-gram by feeding the output word-gram as input to the next word-gram. In system 200A, the first output y0 is fed into the second input word-gram to predict y1 as the second output. Then the second output y1 is fed into the third input word-gram to predict the third output y2.

[0041] In a trained model based on a combined AR / NAR language generation model according to system 200B, the trained model predicts multiple leading word-grams, such as the first two word-grams, based on the AR model. The combined model uses the NAR model to predict the remaining word-grams. In various aspects, accurately predicting the first few (e.g., the first two) word-grams can improve the accuracy of predicting the remaining word-grams in the sentence. Using the AR model for the first few words can improve the accuracy of predicting word-grams (words), while efficiency can still be improved by using the NAR model to process the remaining word-grams. In system 200B, the first predicted word-gram "y0" is fed into the second input word-gram to predict the second word-gram "y1". The prediction of the remaining word-grams uses the NAR language generation model, using parallel processing by placing masks on the last three inputs.

[0042] System 200C shows a non-autoregressive (NAR) language generation model. All five input word-grams are masked to predict each word-gram as y0 for the first output word-gram, y1 for the second output word-gram, y2 for the third output word-gram, y3 for the fourth output word-gram, and y4 for the fifth output word-gram. Decoder 204C processes all five input word-grams independently and in parallel to generate predicted word-grams. Although the language model trained according to system 200C based on the NAR language generation model provides high efficiency in predicting word-grams in parallel, the model trained based on the NAR language generation model may suffer from poor accuracy because the processing of the corresponding word-grams is independent and the prediction of other word-grams is not taken into account when predicting the current word-gram.

[0043] Figure 3 Examples of pre-trained streams and trained models based on pre-trained streams according to aspects of the present disclosure are shown. Example 300 shows three different trained language models for predicting the word unit "y3". In various aspects, the pre-trainer generates a mainstream 310A and a set of prediction streams 312 to 318 based on the length of the target sentence. As shown, the target sentence includes at least five words or terms. For example, the pre-training process generates a mainstream 310A, a first prediction stream 312A to C, a second prediction stream 314B to C, a third prediction stream 316C, and a fourth prediction stream 318C.

[0044] As shown, decoder 302A represents decoding input tokens (y0, y1, y2 and mask symbol "M") based on an input training sentence trained with at least five words to predict token y3 as an output token. Mask ("M") represents a special token that is used as a mask to indicate the token to be predicted. In this case, the first prediction stream 312A refers to the output of the first three tokens (e.g., y0, y1, and y2) in the main stream 310A to predict the fourth token y3.

[0045] In various aspects, the pre-training process can include copying the decoder layers into a mainstream and a set of prediction streams with shared parameters. When generating the mainstream, the word-grams are fed into the decoder by correcting the predicted word-grams with the correct answers for the corresponding word-grams. Therefore, if the prediction is incorrect, each word-gram in the mainstream is corrected. In the prediction stream, the masked word-grams (i.e., masks with "M") pay attention to one or more word-grams in the mainstream and previous generations of prediction streams to predict the value of the masked word-grams. The masked word-grams in the first prediction stream 312A to C constitute a trained model based on the AR language generation model.

[0046] For example, the pre-trainer can generate a prediction stream based on a hybrid AR-NAR model, in which masked tokens refer to previous tokens in an AR manner. Unlike a typical NAR model that uses all possible combinations and connections with other tokens in the target sentence but does not refer to the predicted values ​​of prior tokens, the technology according to the present disclosure may include a structure in which tokens are processed to pay attention to (or refer to) the token prediction for the previous token in the sentence without paying attention to the subsequent tokens.

[0047] In a first example, the first prediction stream 312A shows a predicted word-gram y3 (e.g., the fourth word). All five words are masked, indicating that the prediction operations for all words are performed in parallel in a NAR manner. However, each word-gram pays attention to the word-gram before it in the mainstream 310A, which is similar to the AR approach. For example, the fourth word-gram from the left in the first prediction stream 312A corresponds to y3, as shown by the arrow 336 initiated from the masked fourth word-gram to the y3 output word-gram. To predict y3, the fourth word-gram pays attention to the first word-gram (y0) in the mainstream 310A, as shown by the first reference or participation link 330. The fourth word-gram also pays attention to the second word-gram (y1) in the mainstream 310A, as shown by the second reference or participation link 332, and the third word-gram (y2) in the mainstream 310A, as shown by the third reference or participation link 334. In this case, the fourth token (y3) is the token output by noting (e.g., referencing) the three previous tokens y0 to y2 (or words of the training sentence). As described above, referring to or noting previous tokens to predict the token in the current position is similar to the AR language generation model. Even so, during pre-training, the pre-trainer generates all tokens in each generation of prediction stream in parallel according to the NAR language generation model.

[0048] In various aspects, each prediction stream is associated with a position (i.e., generation or order) in a sequence of prediction streams. In the third example, for example, the first prediction stream 312C precedes the second prediction stream 314C, which precedes the third prediction stream 316C, which precedes the fourth prediction stream 318C. Four generations of prediction streams can be generated using a single processing path without repeating operations. Therefore, prediction streams (e.g., first prediction streams 312A to C) having the same position in a sequence of corresponding masked word elements can be generated using concurrent processing. Therefore, the pre-training process includes a concurrent process for generating all prediction streams.

[0049] In the first prediction stream 312A, the masked word-gram y3 pays attention to the previously predicted words (i.e., golden words) y0, y1, and y2 from the mainstream 310A to predict the value for the word-gram. For word-gram y3, the reference or participation to these words provides the lowest prediction difficulty and improves the accuracy associated with the AR model by including all previous words as the basis for prediction. Each of the other masked words in the first prediction stream 312 is processed similarly to the case for word-gram y3 and is pre-trained in an AR manner (by reviewing previous word-gram outputs) and also in a NAR manner (by parallel processing). Here, the first prediction streams 312A to C are similar to the structure according to the AR model by referring to or paying attention to the previous word-gram outputs (e.g., predicted values) in the mainstream, but the first prediction streams 312A to C are processed in parallel with all words masked according to the NAR model.

[0050] In the second example, the masked token y3 in the second prediction stream 314B refers to or attends to previously predicted tokens y0 and y1 from the main stream 310A and token y2 from the first prediction stream 312B. Token y2 in the first prediction stream 312B and token y3 in the second prediction stream 312B use the conditional probability P(y2, y3|y0, y1). Tokens y2 and y3 generated by the decoder 302B are used as Figure 3 The output in indicates the combined conditional probabilities based on the use of the NAR approach. Due to the transition from using the AR approach for word-grams y0 and y1 followed by the NAR approach to predict word-grams y2 and y3, the difficulty of accurately predicting word-grams in the second prediction stream 314B is increased.

[0051] In a third example, word-gram y0 in the first prediction stream 312C is predicted in a NAR manner without paying attention to word-grams in the mainstream. Here, masked word-gram y0 is predicted through self-attention. Therefore, word-gram y1 in the second prediction stream 314C is predicted without paying attention to word-grams in the mainstream, but based on the conditional probabilities associated with word-gram y0 in a NAR manner. Word-gram y2 in the third prediction stream 316C is predicted without paying attention to word-grams in the mainstream but in a NAR manner, but based on the conditional probabilities associated with word-grams y0 and y1.

[0052] In contrast, in the third example, word y3 is predicted using the fourth prediction stream 318C with reference to word y0 in the first prediction stream 312C, word yl in the second prediction stream 314C, and word y2 in the third prediction stream 316C. The predicted result may not be as accurate as when processed in a pure AR manner, but some prior context (prior prediction output) and higher efficiency in NAR-based processing are provided. In order to predict word y3 based on the fourth prediction stream 318C, only the word that is masked and previously predicted in a NAR manner from the previous prediction stream is visible. That is, the first word in each prediction stream constitutes a NAR prediction when predicting word y3 using the fourth prediction stream 318C. Each masked word in the prediction stream is predicted simultaneously for efficient processing. Each word in the target word sequence is predicted based on the AR approach by referring to (or noting) the previous word in the sentence.

[0053] In various aspects, the output of the pre-training utilizes the pre-training of a unified AR-NAR combination. By paying attention to the preceding word-gram, the pre-training of the unified AR-NAR combination is consistent with more accurate training based on the AR model. In addition, the design of attention between word-grams without paying attention to subsequent word-grams prevents bidirectional attention. Adapting bidirectional attention (usually including paying attention to the preceding word-gram and the subsequent word-gram) requires a word-gram number or length predictor in the sentence before decoding can occur. The requirement for length information will make the decoding process unnecessarily intensive. In various aspects, only the masked word-gram in front is visible to the word-gram. In other aspects, by eliminating the need for estimating length information, the decoding in the present disclosure enables the first generated "[eos]" word-gram to be processed as a signal of the end-of-sentence word-gram, which is consistent with the AR model processing of the "[eos]" word-gram.

[0054] In every aspect, Figure 3The example of fine-tuning is shown in the example of . That is, the pre-trainer generates a set of streams including mainstream and predicted streams as a common basis for adapting fine-tuning to generate trained models based on various combinations of language generation models. The processing models range from AR language generation models for accuracy (such as the first example), NAR language generation models for efficiency (such as the third example), and partially using a combination of AR language generation models and NAR language generation models (such as the second example). In fact, the decoder based on the present disclosure acts as a bridge between the AR model and the NAR model.

[0055] Once the pre-trainer generates a set of mainstream and predicted streams in pre-training, the fine-tuner selects a subset of the generated streams to generate a trained language generation model during the fine-tuning process. In various aspects, the fine-tuning can receive input (e.g., from a developer) that specifies one or more trained language models (e.g., a target model and a target data model) to generate a trained model, for example, using an AR model for processing a certain number of tokens and using a NAR model for processing the remaining tokens of the target sentence according to accuracy constraints and efficiency constraints during deployment. Determining the target data model can be based on balancing accuracy constraints and / or efficiency constraints for predicting tokens.

[0056] As described above, the fine tuner generates a trained model to address a specific processing scenario. For example, available resources (e.g., memory and processing availability), accuracy requirements, efficiency requirements, etc. can be evaluated to identify the best trained model for a specific situation. For example, if the processing availability is low, the best trained model may rely less on the AR model and more on the NAR model. Alternatively, if accuracy is important, the best trained model may rely more on the AR model and less on the NAR model. For example, the best trained model for generating accurate predictions without a large amount of latency may involve a combination of AR and NAR models, such as using an AR model to predict the first word and using NAR to predict the remaining word. After determining the structure for the best model (e.g., AR for the first word, NAR for the remaining word), a subset of the streams used to generate the trained model can be selected from a superset of the streams generated during preprocessing.

[0057] exist Figure 3In the second example, fine-tuning generates a trained model based on a combination between AR / NAR models, including selecting a mainstream and two or more prediction streams. In the case of predicting word y3 based on the first two word-grams in an AR manner and based on the remaining word-grams in a NAR manner, the fine-tuner selects the mainstream 310A, the first prediction stream 312B, and the second prediction stream 314B. The word-gram y3 in the second prediction stream 314B pays attention to the first word-gram y0 and the second word-gram y1 in the mainstream 310A, while paying attention to the third word-gram y2 in the first prediction stream 312B. Therefore, the combined language generation model uses the AR language generation model for the first two word-grams (words) to achieve high accuracy, while using the NAR language generation model for "y2" and "y3" to achieve high efficiency.

[0058] exist Figure 3 In the third example of , fine-tuning is used to generate a trained model based on the NAR language generation model 352. Here, the fine-tuner selects the first prediction stream 312A to C, the second prediction stream 314B to C, the third prediction stream 316C, and the fourth prediction stream 318C. That is, the fine-tuner selects all prediction streams instead of the mainstream to generate the structure of the trained model based on the NAR model. For example, in order to predict the word "y3", the word for "y3" pays attention to the first word of each previous prediction stream: "y0" in the first prediction stream 312C, "y1" in the second prediction stream 314C, and "y2" in the third prediction stream 316C. In various aspects, the fine-tuner for the NAR language generation model performs predictions of all masked word-grams in the prediction streams in a concurrent manner. Concurrent processing makes fine-tuning for generating the NAR language generation model very efficient.

[0059] Figure 4 An illustrative example of generating a prediction stream during a pre-training process according to aspects of the present disclosure is shown. Figure 4 Four prediction streams are shown: a first prediction stream 312, a second prediction stream 314, a third prediction stream 316, and a fourth prediction stream 318. The illustration of the cells above the corresponding labels of the prediction streams describes which positions of the word-grams are attended to (referenced) for predicting the corresponding word-grams. The horizontal numbers indicate the word-gram positions to be trained (predicted). The vertical numbers indicate the prior word-gram positions to be attended to (referenced) when predicting the word-gram at the current position.

[0060] In the corresponding cell, "SA" indicates self-attention. "MS" indicates attention to (reference to) the word-meta at that position in the mainstream. "1st" indicates attention to (reference to) the word-meta at that position in the first prediction stream. "2nd" indicates attention to (reference to) the word-meta at that position in the second prediction stream. "3rd" indicates attention to (reference to) the word-meta at that position in the third prediction stream. An empty cell indicates that there is no attention from the word-meta to the corresponding word-meta in the mainstream or prediction stream.

[0061] like Figure 3 As shown, using the first prediction stream 312, the first word at word position (0) is predicted based on self-attention without reference to any other word to output the word value "y0", as given by Figure 3 The second word at word position (1) is predicted based on self-attention and attention to (i.e., reference to) the first word (0) in the mainstream to output the word value "y1". The third word at word position (2) is predicted based on self-attention and attention to the first word (0) and the second word (1) in the mainstream, as shown by Figure 3 The fourth word at word position (3) is predicted based on self-attention and attention to the first, second and third words in the main stream, as shown by Figure 3 As shown in the first prediction flow process 312A.

[0062] Using the second prediction stream 314, the first word-gram at word-gram position (0) is predicted based on self-attention. The second word-gram at word-gram position (1) is predicted based on self-attention and attention to the first word-gram (0) in the first prediction stream, as given by Figure 3 The third word at word position (2) is predicted based on self-attention and attention to the first word (0) in the mainstream and the second word (1) in the first prediction stream. The fourth word at word position (3) is predicted based on self-attention and attention to the first word (0) and the second word (1) in the mainstream and attention to the third word (2) in the first prediction stream, as shown by Figure 3 As shown in the second prediction flow process 314B.

[0063] Using the third prediction stream 316, the first word-gram at word-gram position (0) is predicted based on self-attention. The second word-gram at word-gram position (1) is predicted based on self-attention and attention to the first word-gram (0) in the first prediction stream. The third word-gram at word-gram position (2) is predicted based on self-attention and attention to the first word-gram (0) in the first prediction stream and the second word-gram (1) in the second prediction stream, as shown in FIG. Figure 3 The fourth word-gram at word-gram position (3) is predicted based on self-attention and attention to the first word-gram (0) in the main stream, the second word-gram (1) in the first prediction stream, and the third word-gram (2) in the second prediction stream.

[0064] In the fourth prediction stream 318, the first word-gram at word-gram position (0) is predicted based on self-attention. The second word-gram at word-gram position (1) is predicted based on self-attention and paying attention to the first word-gram (0) in the first prediction stream. The third word-gram at word-gram position (2) is predicted based on self-attention and paying attention to the first word-gram (0) in the first prediction stream and the second word-gram (1) in the second prediction stream. The fourth word-gram at word-gram position (3) is predicted based on self-attention and paying attention to the first word-gram (0) in the first prediction stream, the second word-gram (1) in the second prediction stream, and the third word-gram (2) in the third prediction stream, as shown in FIG. Figure 3 As shown in the fourth prediction flow process 318C.

[0065] In various aspects, pre-training includes generating a set of mainstream and prediction streams. In the i-th prediction stream, a word-gram is predicted as the i-th subsequent word-gram from a previously predicted word-gram. Each word-gram in the first prediction stream 312A-C is predicted in an autoregressive (AR) mode. That is, the masked word-gram of each word-gram in the first prediction stream 312A-C pays attention to its previously predicted word-gram from the mainstream.

[0066] Figure 5A An example keyword matching process is shown by deploying a trained language generation model in a content search engine. The process receives a query 502 for searching for information with keywords. A generation step 504 generates a query set based on the received query by using the trained language generation model. The query set is indicated by query rewrite-1 506A, query rewrite-2 506B, query rewrite-3 506C, etc. A retrieval operation 510 retrieves results based on the query set. The results are then matched with a set of known keywords 508. In various aspects, the trained language generation model is based on a hybrid AR-NAR model, similar to Figure 3 The third example in FIG. The trained language generation model uses the received query 502 to generate new queries, which is similar in the context of using a set of prediction streams that are fine-tuned to predict in a NAR manner. The trained language model generates queries within an acceptable range of accuracy in an efficient manner for real-time processing by a content search engine.

[0067] Figure 5BAn example of a component system for generating matching keywords according to aspects of the present disclosure is shown. Rectangular boxes represent data. Elliptical boxes represent operations. A query 502 is received and a generative keyword retrieval operation 526 step uses a keyword Trie 524 (i.e., a search tree graph for retrieving keywords) to generate a set of matching keywords. A Trie, also known as a prefix tree, is a tree-like data structure having nodes that store each keyword in a library. The keyword Trie 524 can be generated by a Trie construction step 522, which uses a keyword corpus 520.

[0068] Figure 6 is an example of a method for generating and using a trained language generation model according to aspects of the present disclosure. Figure 6 6. Generally, the method 600 begins with a start operation 602 and ends with an end operation 624. The method 600 may include more or fewer steps or may be different from the method 600 in some embodiments. Figure 6 The method 600 may be executed as a set of computer executable instructions that are executed by a computer system and encoded or stored on a computer readable medium. In addition, the method 600 may be executed by gates or circuits associated with a processor, ASIC, FPGA, SOC, or other hardware device. Hereinafter, the method 600 will be referred to in conjunction with Figure 1 , 2 , 3, 4, 5A, 5B, 7 and 8A to B to explain the systems, components, devices, modules, software, data structures, data characteristic representations, signaling diagrams, methods, etc. described in FIG.

[0069] After start operation 602, method 600 begins at receiving operation 604, which receives training data. The developer can specify the type of training data, which can include a set of sentences for training a language generation model. Encoding operation 608 encodes the training data into a set of vectors. For example, in natural language processing, each word in a sentence can correspond to a word represented by one or more vectors.

[0070] A generating operation 610 generates a mainstream. The mainstream can be a series of word-grams, each word-gram representing a word in a sentence in an encoded form. The generated mainstream can be trained using the following steps: comparing the predicted answer with the correct answer, if the predicted answer is incorrect, correcting the predicted answer using the correct answer, and using the correct answer to predict subsequent word-grams or words in the sentence (e.g., teacher-forcing). In various aspects, the generating operation 610 is part of a pre-training process.

[0071] The generating operation 612 generates a set of prediction streams. In various aspects, the generating operation 612 can generate a set of prediction streams based on concurrent processing, such as Figure 4 As shown. The generation operation 612 can be part of a pre-training process. For each prediction stream, predicting one or more word-grams can pay attention to (or refer to) one or more previous word-gram values ​​(or outputs) in a previous prediction stream or mainstream. In various aspects, unlike the generation model based on the traditional NAR language model, the generation operation 612 can exclude attention to the connection with subsequent word-grams but can pay attention to the word-grams before the current word-gram. That is, although the generated prediction stream with masked word-grams for parallel processing can be similar to the NAR model in efficiency, processing word-grams by paying attention to previous word-grams reflects the characteristics of the AR model for improving accuracy.

[0072] A receiving operation 614 receives a model type of a target language generation model. The type of the target or trained language generation model can be selected based on balancing resource availability, efficiency requirements, accuracy requirements, etc. The type of the trained model can rely more on the AR model for accuracy or rely more on the NAR model for efficiency. For example, the trained model can specify a different number of word units (e.g., the first word unit, the first two word units) to be predicted based on a more accurate AR model, while the remaining word units are predicted based on a more efficient NAR model. In some aspects, the type of data used for prediction can determine the model type. For example, the type of data can include text sentences and image pixel data. In some other aspects, the model type can also be based on efficiency requirements, such as the maximum time allowed for prediction data.

[0073] A generating operation 616 generates a trained language model by fine-tuning a pre-trained model (e.g., the prediction stream generated at generating operation 612) based on the type of trained language model received at receiving operation 614. The fine-tuning step selects a particular set of streams to generate a trained language generation model based on the type received. In this way, for example, a trained model or a trained data model can be customized for a particular processing environment based on balancing efficiency and accuracy. Upon completion of generating operation 616, the fine-tuned model is a trained model for deployment for use.

[0074] Deployment operation 618 deploys the trained (fine-tuned) language model to a business or production system environment for use. For example, in various aspects, the trained model can be deployed for data search and keyword search servers. The trained model can automatically generate queries similar to a given query. In various aspects, different fine-tuners can be used to fine-tune the same pre-trained model to generate a trained model set, wherein each trained model is fine-tuned for different types of target data models, using AR models, NAR models, and one or more combinations of AR models and NAR models as processing models. Each trained model in the trained model set can provide different balances or trade-offs between efficiency constraints and accuracy constraints in data processing. In some other aspects, each trained model can be deployed to a deployment environment that matches these trade-off requirements for efficiency and accuracy. A trained data model based on AR model fine-tuning can be deployed to an environment where the accuracy of the predicted data takes precedence over the efficiency of the predicted data. Another trained data model based on NAR model fine-tuning can be deployed to another environment where the efficiency or processing time of the predicted data is strictly required, such as a large web search engine. As such, the present disclosure provides for pre-training of a superset of generated data (eg, a generated prediction stream) that may be fine-tuned to generate a data model that is customized to meet requirements in a specific deployment environment.

[0075] Retrieval operation 620 retrieves data for processing. In various aspects, the trained model generated in generation operation 616 is used for information retrieval, e.g., generative keyword retrieval. For example, a trained model based on a combined AR / NAR model can generate a query set based on a given query to enhance search results by retrieving information based on the query set in addition to the given query. In a system environment where high efficiency rather than accuracy is required, a trained language generation model that is fine-tuned to predict more tokens in a hybrid AR-NAR model may be appropriate. Keyword matching can be performed on the retrieved data, and the matched keywords can be provided as results.

[0076] Generate operation 622 can generate an answer (or result) to a given query based on generative retrieval. In various aspects, the trained language generation model can be used to supplement existing query-keyword matching algorithms that use sequence-to-sequence models. In various aspects, method 600 can end with end operation 624.

[0077] It should be understood that operations 602 to 624 are described for the purpose of illustrating the present method and system and are not intended to limit the present disclosure to a specific order of steps. For example, the steps may be performed in a different order, additional steps may be performed, and disclosed steps may be excluded without departing from the present disclosure.

[0078] Figure 7 is a block diagram illustrating the physical components (e.g., hardware) of a computing device 700, by which aspects of the present disclosure may be practiced. The computing device components described below may be applicable to the computing devices described above. In a basic configuration, the computing device 700 may include at least one processing unit 702 and a system memory 704. Depending on the configuration and type of the computing device, the system memory 704 may include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 704 may include an operating system 705 and one or more program tools 706, which are suitable for performing various aspects disclosed herein. The operating system 705 may, for example, be suitable for controlling the operation of the computing device 700. In addition, aspects of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application and is not limited to any particular application or system. This basic configuration is Figure 7 708. The computing device 700 may have additional features or functionality. For example, the computing device 700 may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, or tapes. Such additional storage Figure 7 , which is illustrated by a removable storage device 709 and a non-removable storage device 710.

[0079] As described above, a plurality of program tools and data files may be stored in system memory 704. While executing on at least one processing unit 702, program tools 706 (e.g., application 720) may perform processes including, but not limited to, aspects as described herein. Application 720 includes encoder 722, pre-trainer 724, fine tuner 726, query receiver 728, and generative keyword retriever 730, as described with respect to Figure 1 Other program tools that may be used according to aspects of the present disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided applications, and the like.

[0080] In addition, aspects of the present disclosure may be practiced in circuits on a single chip including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing a microprocessor, or circuits containing electronic components or a microprocessor. For example, aspects of the present disclosure may be practiced via a system on a chip (SOC) wherein Figure 7Each or many of the components shown in can be integrated onto a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operated via a SOC, the functions described herein regarding the ability to switch protocols for a client can be operated via dedicated logic integrated with other components of the computing device 700 on a single integrated circuit (chip). Various aspects of the present disclosure may also be practiced using other technologies capable of performing logical operations such as AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, aspects of the present disclosure may be practiced in a general-purpose computer or any other circuit or system.

[0081] The computing device 700 may also have one or more input devices 712, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or sliding input device, etc. Output device(s) 714 (such as a display, a speaker, a printer, etc.) may also be included. The above devices are examples and other devices may be used. The computing device 700 may include one or more communication connections 716 that allow communication with other computing devices 750. Examples of suitable communication connections 716 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports.

[0082] The term computer-readable media as used herein may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information (such as computer-readable instructions, data structures or program tools). System memory 704, removable storage device 709 and non-removable storage device 710 are all examples of computer storage media (e.g., memory storage devices). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, cassette, magnetic tape, disk storage or other magnetic storage device, or any other product that can be used to store information and can be accessed by computing device 700. Any such computer storage media may be part of computing device 700. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0083] Communication media may be embodied by computer readable instructions, data structures, program tools or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or modified in a manner that encodes information in the signal. By way of example and not limitation, communication media may include wired media such as a wired network or direct wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0084] Fig. 8A and 8B A computing device or mobile computing device 800 is shown, such as a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc., with which aspects of the present disclosure can be practiced. In some aspects, a computer system (e.g., a computer system) is used to perform a computer program executed by a user (e.g., Figure 1 The client used by the user of client device A 102A and the user of client device B 102B) may be a mobile computing device. Fig. 8A , one aspect of a mobile computing device 800 for implementing these aspects is shown. In a basic configuration, the mobile computing device 800 is a handheld computer having both input elements and output elements. The mobile computing device 800 typically includes a display 805 and one or more input buttons 810 that allow a user to input information into the mobile computing device 800. The display 805 of the mobile computing device 800 can also be used as an input device (e.g., a touch screen display). If included as an optional input element, the side input element 815 allows additional user input. The side input element 815 can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 800 can incorporate more or fewer input elements. For example, the display 805 may not be a touch screen in some aspects. In yet another alternative aspect, the mobile computing device 800 is a portable telephone system, such as a cellular phone. The mobile computing device 800 can also include an optional keypad 835. The optional keypad 835 can be a physical keypad or a "soft" keypad generated on the touch screen display. In various aspects, the output elements include a display 805 for displaying a graphical user interface (GUI), a visual indicator 820 (e.g., a light emitting diode), and / or an audio transducer 825 (e.g., a speaker). In some aspects, the mobile computing device 800 incorporates a vibration transducer for providing tactile feedback to the user. In yet another aspect, the mobile computing device 800 incorporates input and / or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port), for sending signals to or receiving signals from external devices.

[0085] Figure 8B is a diagram showing a computing device, a server (e.g., Figure 1 800 ), a mobile computing device, etc. That is, a mobile computing device 800 can be incorporated into a system 802 (e.g., a system architecture) to implement some aspects. The system 802 can be implemented as a "smart phone" capable of running one or more applications (e.g., a browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 802 is integrated into a computing device, such as an integrated digital assistant (PDA) and a wireless phone.

[0086] One or more applications 866 may be loaded into the memory 862 and run on or in association with the operating system 864. Examples of application programs include a phone dialer, an email program, a information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 802 also includes a non-volatile storage area 868 within the memory 862. The non-volatile storage area 868 may be used to store persistent information that should not be lost when the system 802 is powered off. The application 866 may use and store information in the non-volatile storage area 868, such as emails or other messages used by the email application, and the like. A synchronization application (not shown) also resides on the system 802 and is programmed to interact with a corresponding synchronization application residing on a host computer to keep information stored in the non-volatile storage area 868 synchronized with corresponding information stored on the host computer. It should be understood that other applications may be loaded into the memory 862 and run on the mobile computing device 800 described herein.

[0087] System 802 has a power source 870, which can be implemented as one or more batteries. Power source 870 can also include an external power source, such as an AC adapter or a powered docking cradle to replenish or recharge the batteries.

[0088] The system 802 may also include a radio interface layer 872 that performs the functions of sending and receiving radio frequency communications. The radio interface layer 872 facilitates wireless connectivity between the system 802 and the "outside world" via a communications operator or service provider. Transmissions to and from the radio interface layer 872 are performed under the control of the operating system 864. In other words, communications received by the radio interface layer 872 may be propagated to the application 866 via the operating system 864, and vice versa.

[0089] A visual indicator 820 (e.g., an LED) may be used to provide a visual notification, and / or an audio interface 874 may be used to generate an audible notification via an audio transducer 825. In the illustrated configuration, the visual indicator 820 is a light emitting diode (LED) and the audio transducer 825 is a speaker. These devices may be directly coupled to a power supply 870 so that when activated, they remain on for a duration specified by the notification mechanism even if the processor 860 and other components may be turned off to save battery power. The LED may be programmed to remain on indefinitely until the user takes action to indicate the power-on state of the device. The audio interface 874 is used to provide an audible signal to the user and receive an audible signal from the user. For example, in addition to being coupled to the audio transducer 825, the audio interface 874 may also be coupled to a microphone to receive an audible input, such as to facilitate a telephone conversation. According to aspects of the present disclosure, the microphone may also be used as an audio sensor to facilitate the control of notifications, as will be described below. The system 802 may also include a video interface 876 that supports the operation of the onboard camera 830 to record still images, video streams, and the like.

[0090] The mobile computing device 800 implementing the system 802 may have additional features or functionality. For example, the mobile computing device 800 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 8B , shown by non-volatile storage area 868.

[0091] Data / information generated or captured by the mobile computing device 800 and stored via the system 802 may be stored locally on the mobile computing device 800, as described above, or the data may be stored on any number of storage media that may be accessed by the device via the radio interface layer 872 or via a wired connection between the mobile computing device 800 and a separate computing device associated with the mobile computing device 800 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information may be accessed via the mobile computing device 800 via the radio interface layer 872 or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use in accordance with well-known data / information transfer and storage methods, including electronic mail and collaborative data / information sharing systems.

[0092] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure claimed in any way. The aspects, examples and details provided in this application are considered to be sufficient to convey ownership and enable others to make and use the best mode of the disclosure claimed. The disclosure claimed should not be interpreted as being limited to any aspect, for example, or the details provided in this application. Whether it is displayed and described in combination or separately, various features (both structural and methodological) are intended to be selectively included or omitted to produce an embodiment with a specific feature set. The description and illustration of the application have been provided, and those skilled in the art can envision changes, modifications and alternative aspects that fall within the spirit of the broader aspects of the overall inventive concept embodied in this application, which do not depart from the broader scope of the disclosure claimed.

[0093] The present disclosure relates to systems and methods for generating a keyword sequence from an input query according to at least the examples provided in the following sections:

[0094] (A1) In one aspect, some embodiments include a method (e.g., 600) of generating a language generation model from receiving training data (e.g., 148); receiving the training data (e.g., 148); generating a main stream (e.g., 310A) based on the training data (e.g., 148), wherein the main stream represents a stream of word-grams, and wherein each word-gram represents a value associated with the training data (e.g., 148); generating a series of prediction streams (e.g., 312A-C, 314B-C, 316C, and 318C) based on the main stream (e.g., 310A), wherein each prediction stream includes one or more masked word-grams corresponding to one or more word-gram positions in the stream of word-grams; based on a target data model (e.g., 350 and 352), selecting a first set of streams (e.g., 310A); ) for predicting word-grams according to a first processing model (e.g., 350), wherein the first set of streams is selected from a main stream (e.g., 310A) and a series of prediction streams (e.g., 312A-C, 314B-C, 316C, and 318C); based on a target data model (e.g., 350 and 352), selecting a second set of streams (e.g., 312, 314, 316, and 318) for predicting word-grams according to a second processing model (e.g., 352), wherein the second set of streams is selected from the series of prediction streams; and generating a trained data model (e.g., a combination of 310A, 312B, and 314B) corresponding to the target data model using the selected first set of streams and the second set of streams for predicting word-grams according to accuracy constraints and efficiency constraints during deployment.

[0095] (A2) In some embodiments of A1, the first processing model (e.g., 350) is associated with an autoregressive (AR) model (e.g., 200A), and wherein the second processing model (e.g., 352) is associated with a non-autoregressive (NAR) model (e.g., 200C).

[0096] (A3) In some embodiments of A1, the target data model (e.g., 350 and 352) specifies a certain number of words to be processed based on the first processing model (e.g., 350) and the remaining words to be processed based on the second processing model (e.g., 352).

[0097] (A4) In some embodiments of A1, a second stream in the second stream set refers to a predicted value of a word-gram in a first stream in the second stream set.

[0098] (A5) In some embodiments of A4, the second set of streams are processed in parallel according to the second processing model.

[0099] (A6) In some embodiments of A1, the second stream of the first stream set refers to the value of the word-gram in the first stream of the first stream set.

[0100] (A7) Some embodiments of A1 further include: determining a target data model based on balancing accuracy constraints and efficiency constraints for predicting word-grams.

[0101] (A8) In some embodiments of A1, the first flow is the mainstream.

[0102] (A9) Some embodiments of A1 also include: receiving a query, generating a query set based on the received query using a trained data model, retrieving one or more keywords based on the query set, generating a keyword set by matching the one or more keywords with at least one known keyword in a keyword corpus, and sending the generated keyword set.

[0103] Any one of the one or more aspects described above may be combined with any other of the one or more aspects. Any one of the one or more aspects as described herein.

Claims

1. A computer-implemented method for training a data model, the method comprising: receiving training data; generating a main stream based on the training data, wherein the main stream represents a stream of word-grams, and wherein each word-gram represents a value associated with the training data; generating a series of prediction streams based on the main stream, wherein each prediction stream includes one or more masked word-grams, the one or more masked word-grams corresponding to one or more word-gram positions in the word-gram stream; Based on the target data model, selecting a first set of streams for predicting word-grams according to a first processing model, wherein the first set of streams is selected from the main stream and the series of prediction streams and the first processing model is associated with an autoregressive (AR) model; selecting a second set of streams for predicting word-grams based on the target data model according to a second processing model, wherein the second set of streams is selected from the series of prediction streams and the second processing model is associated with a non-autoregressive (NAR) model; and A trained data model corresponding to the target data model is generated using the selected first and second sets of streams for predicting word-grams according to accuracy constraints and efficiency constraints during deployment.

2. The method of claim 1, wherein the target data model specifies a certain number of word-grams to be processed based on the first processing model and a remaining number of word-grams to be processed based on the second processing model. 3 . The method of claim 1 , wherein a second stream in the second set of streams refers to a predicted value of a word-gram in a first stream in the second set of streams. The method of claim 3 , wherein the second set of streams are processed in parallel according to the second processing model. 5 . The method of claim 1 , wherein a second stream in the first set of streams references values ​​of word-grams in a first stream in the first set of streams, and wherein the first stream is the primary stream.

6. The method according to claim 1, further comprising: Receive inquiries; Based on the received query, generate a query set using the training data model; Retrieving one or more keywords based on the query set; generating a keyword set by matching the one or more keywords with at least one known keyword in a keyword corpus; as well as The generated keyword set is sent.

7. The method according to claim 1, further comprising: The target data model is determined based on balancing accuracy constraints and efficiency constraints for predicting word-grams.

8. A system for training a data model, the system comprising: processor; as well as a memory storing computer executable instructions that, when executed by the processor, cause the system to: receiving training data; generating a mainstream based on the training data, wherein the mainstream represents a flow of word units, and wherein each word-gram represents a value associated with the training data; generating a series of prediction streams based on the main stream, wherein each prediction stream includes one or more masked word-grams, the one or more masked word-grams corresponding to one or more word-gram positions in the word-gram stream; Based on the target data model, selecting a first set of streams for predicting word-grams according to a first processing model, wherein the first set of streams is selected from the main stream and the series of prediction streams and the first processing model is associated with an autoregressive (AR) model; selecting a second set of streams for predicting word-grams based on the target data model according to a second processing model, wherein the second set of streams is selected from the series of prediction streams and the second processing model is associated with a non-autoregressive (NAR) model; and A trained data model corresponding to the target data model is generated using the selected first and second sets of streams for predicting word-grams according to accuracy constraints and efficiency constraints during deployment.

9. The system of claim 8, wherein the target data model specifies a number of word-grams to be processed based on the first processing model and the remaining word-grams to be processed based on the second processing model.

10. The system of claim 8, wherein a second stream in the second set of streams references predicted values ​​of word-grams in a first stream in the second set of streams, and wherein the second set of streams is processed in parallel according to the second processing model.

11. The system of claim 8, wherein a second stream in the first set of streams references values ​​of word-grams in a first stream in the first set of streams, and wherein the first stream is the primary stream.

12. The system of claim 8, wherein the computer executable instructions, when executed, further cause the system to: The target data model is determined based on balancing accuracy constraints and efficiency constraints for predicting word-grams.

13. A computer-readable non-transitory recording medium storing computer-executable instructions, which when executed by a processor causes a computer system to: receiving training data; generating a main stream based on the training data, wherein the main stream represents a stream of word-grams, and wherein each word-gram represents a value associated with the training data; generating a series of prediction streams based on the main stream, wherein each prediction stream includes one or more masked word-grams, the one or more masked word-grams corresponding to one or more word-gram positions in the word-gram stream; Based on the target data model, selecting a first set of streams for predicting word-grams according to a first processing model, wherein the first set of streams is selected from the main stream and the series of prediction streams and the first processing model is associated with an autoregressive (AR) model; selecting a second set of streams for predicting word-grams based on the target data model according to a second processing model, wherein the second set of streams is selected from the series of prediction streams and the second processing model is associated with a non-autoregressive (NAR) model; and A trained data model corresponding to the target data model is generated using the selected first and second sets of streams for predicting word-grams according to accuracy constraints and efficiency constraints during deployment.

Images